Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claims 1-22 are pending. Claims 1, 11 and 16 are independent and have different scopes and have been amended. New Claim 22 is added which is also independent.
This Application was published as U.S. 20230395065.
Apparent priority: 1 June 2022.
To the degree that it is intended that the “Moderator” in the instant Claims to refers to a “Human Moderator,” the Claims have been amended to specify a “Human Moderator.”
Applicant’s amendments and arguments are considered but are either unpersuasive or moot in view of the new grounds of rejection that, if presented, were necessitated by the amendments to the Claims.
This action is Final.
Response to Amendments
Applicant has amended limitation(s) of various “stages” in Claims 11-21 to include structure and these limitations are no longer subject to 112(f).
35 U.S.C. 101 Rejection is withdrawn in view of the amendments.
Independent claims have been amended as follows:
1. A method for online voice content moderation, the method comprising:
providing a multi-stage voice content analysis system including a communication interface for receiving toxic speech content associated with an online platform, the online platform having a platform content policy, the system including a processor operatively coupled with and in communication with an input, the processor including a pre-moderator stage circuitry having a toxicity scorer configured to provide a toxicity score for a given toxic speech content from a first user received by the processor via the input, the toxicity score being a function of a platform content policy;
generating, by the toxicity scorer, the toxicity score for the toxic speech content as a function of the platform content policy;
determining, by the toxicity scorer, that the toxicity score for the toxic speech content, from the first user, meets or exceeds a toxicity score threshold;
automatically banning or muting the first user having the toxicity score for the toxic speech content that meets or exceeds the toxicity score threshold; and
providing the toxic speech content to a human moderator as a function of the toxicity score.
11. A multi-stage content analysis system comprising:
a first stage circuitry trained using a database operatively coupled with the first stage circuitry having training data with positive or negative examples of training content for the first stage, the first stage circuitry configured to: receive speech content, analyze the speech content to categorize the speech content as having first-stage positive speech content or first-stage negative speech content;
a pre-moderator stage circuitry in communication with the first stage engine configured to analyze at least a portion, but less than all, of the first-stage negative speech content, the pre-moderator stage circuitry further configured to analyze at least a portion of the first-stage positive speech content to categorize the first-stage positive speech content as having pre-moderator-stage positive speech content or pre-moderator-stage negative speech content, the pre-moderator stage further configured to update the database using the pre-moderator-stage positive speech content or the pre-moderator-stage negative speech content, the pre-moderator stage including a toxicity scorer configured to provide a toxicity score for the pre-moderator-stage positive speech content;
determine, for a first user, if the toxicity score the toxic speech content meets or exceeds a toxicity score threshold, and automatically reduce an audio exposure of a second user to the first user if the toxicity score for the toxic speech content of the first user meets or exceeds the toxicity score threshold; and
a user interface operatively coupled with the pre-moderator stage circuitry, the user interface configured to display the toxicity score for the pre-moderator-stage positive speech content as a function of the toxicity score.
16. A method for online voice content moderation, the method comprising:
providing a multi-stage voice content analysis system, the system including a pre-moderator stage circuitry having a toxicity scorer configured to provide a raw toxicity score for a plurality of toxicity categories for a given toxic speech content from a first user,
generating, by the toxicity scorer, a weighted toxicity score for the plurality of toxicity categories for the given toxic speech content as a function of the raw toxicity score and weighting factors from a platform content policy;
determining, by the toxicity scorer, the maximum weighted toxicity score and the associated toxicity category;
determining, by the toxicity scorer, that the maximum weighted toxicity score meets or exceeds a toxicity score threshold;
automatically banning or muting the first user having the maximum weighted toxicity score that meets or exceeds the toxicity score threshold; and
providing the toxic speech content to a human moderator with an indication of the maximum weighted toxicity score and the associated toxicity category.
22. (New) A multi-stage content analysis system comprising:
means for providing a toxicity score for a given toxic speech content from a first user, the toxicity score being a function of a platform content policy;
means for generating the toxicity score for the toxic speech content as a function of the platform content policy;
means for determining that the toxicity score for the toxic speech content, from the first user, meets or exceeds a toxicity score threshold;
means for automatically banning or muting the first user having the toxicity score for the toxic speech content that meets or exceeds the toxicity score threshold; and
means for providing the toxic speech content to a human moderator as a function of the toxicity score.
Response to Arguments
Regarding 35 U.S.C. 101 Rejections
Rejection is withdrawn in view of the amendments.
Applicant relies on the most recent guidelines. Response 9-17. Arguments are not persuasive as discussed below. In particular, Applicant argues that operations that cannot be done practically in the human mind should not be rejected as human mental activity:
PNG
media_image1.png
312
732
media_image1.png
Greyscale
Response 9.
In Reply: Sometimes the claims are so broad that the problem is not with “cannot be practically performed in the human mind.” Rather, the problem is that the Claims do not set any concrete and workable criteria for a Machine to perform these operations such that nothing other than a human mind is capable of performing them. A machine requires more detail as to what it is supposed to do whereas a human can make his own decisions based on broad instructions. The Claims are considered to include the level of detail suitable for a machine considering the level of skill in the art.
Regarding 35 U.S.C. 103 Rejections
Most of the arguments are moot in view of the new grounds of rejection that if presented were necessitated by the amendments to the Claims.
Regarding platform policy please see:
Epstein provides its “platform content policy” in the form of labeled training samples to the “offensive word classifier.” Figure 2, 208. “[0046] At stage 208, an offensive words classifier is trained using the labeled first set of text samples. Stage 208 can be the first of multiple training iterations in training the classifier. In this first iteration, initial rules and signals may be determined so as to configure the classifier to be able to recognize one or more signals (or features) associated with a text sample and to generate an offensiveness label for the text sample….” Instant Application provides: “[0016] … The content policy may be received by a manual user entry and/or by answering a questionnaire. The content policy provides a set of statistical weights that are applied to the raw scores for each toxicity category.” This conforms to the labeled set of Epstein.
Instant Application appears to include an elaborate discussion of variations in the Content Policy which are NOT currently in the Claims.
Tongya is directed to moderating content on a “Platform for Online Multi-User Chat Service 20” and teaches that the platform administrator or some of the users may define the content policy and feedback which also is a type of policy determination is provided manually by moderators. “[0034] … The offensive language mapping data 80 may be manually generated by an administrator of the platform 20, generated using crowd-source techniques, or generated using another type of heuristic method….” “[0054]… The heuristic offensive language checking module 72 may be manually generated by moderators to flag specific words or phrases that are offensive.” “[0021] …The platform 20 for the online multi-user chat service may apply platform-wide filtering on user chat data 22 according to rules and protocols of the platform 20 in addition to the user filter preferences 36 associated with each individual user.”
35 U.S.C. 112(f) Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.
Such claim limitation(s) is/are: various “means for” limitations in Claim 22.
This Claim was added by amendment and indicates an express intent to include a claim with functional language.
Additionally, the “toxicity scorer” which has been added to Claim 1 and its dependents and Claim 16 and its dependents and “first stage engine” of Claims 16 and its dependents are interpreted under 35 U.S.C. 112(f) as these are not a term of art and lack any structural definition in the Claim. (Engine can be software or hardware.)
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims of the instant Application are rejected on the ground of nonstatutory double patenting as being unpatentable over claims of U.S. Patent No. 11,996,117, and provisionally rejected over the claims of U.S. 18/204,873 (Allowed but not issued) and U.S. 18/660,835 (Not yet examined) as shown below alone or in combination with the 103 references cited below when minor limitations (such as display to a moderator) are missing from the claim of the reference. A table of mapping was provided in the previous Office action. A modified table is not presented at this time and until the Claims reach a final allowable format. New Claim 22 is broad and maps to the previously mapped claims of the references above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-9, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Epstein (U.S. 20150309987) in view of Tongya (U.S. 20220284884).
Regarding Claim 1, Epstein teaches:
1. A method for online voice content moderation, the method comprising:
providing a multi-stage voice content analysis system [Epstein, Figure 1, “automatic speech recognizer 106” followed by “offensive word classifier and redactor 108” which teach two or three stages of analysis.] (Instant Application: “[0042] …For example, one stage may be a keyword detector running on the speaker's computer. Another stage may be a transcription engine running on a GPU, followed by some transcription interpretation logic running on a CPU in the same computer….” And “[0043] One or more of the stages 112-118 may include a toxicity scorer 236….”) including a communication interface for receiving toxic speech content associated with an online platform, [The user devices of Epstein are used as portals to social media and online discussion groups which are “online platforms”: “[0002] … With great ease, a single device can be used as a portal to social media content, personal messaging systems, online discussion groups, web sites, games, productivity software, and more. …”] the online platform having a platform content policy, [Epstein, Figure 1, the “offensive word classifier and redactor 108” of Figure 1 is trained to reflect the platform content policy. Figure 2, 208 and [0046]. “[0058] After the classifier is initially trained at stage 208 to use one or more content-based signals and/or non-content context-based signals associated with a text sample, the classifier can then be re-trained in one or more subsequent training iterations….However, while the classifier may have been initially trained on a relatively small number of text samples in the first set that were hand-labeled by one or more users, subsequent re-training stages may use increasingly larger and diverse sets of text samples that have been labeled by a classifier that was trained in a prior iteration….”]
the system including a processor operatively coupled with and in communication with an input, [Epstein, Figure 5, showing the hardware components including processor 502 and “[0083] The computing device 500 includes a processor 502, a memory 504, a storage device 506, a high-speed interface 508 connecting to the memory 504 and multiple high-speed expansion ports 510, and a low-speed interface 512 connecting to a low-speed expansion port 514 and the storage device 506….” “[0086] … The low-speed expansion port 514, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.” “[0099] … The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.”] the processor including a pre-moderator stage circuitry having a toxicity scorer configured to provide a toxicity score for a given toxic speech content from a first user received by the processor via the input, the toxicity score being a function of a platform content policy; [Epstein, Figure 1, “offensive words classifier and redactor 108” / “pre-moderator stage” is receiving speech input of the user 102a, b, c. Figure 3 begins with “receive utterance 302” which is transcribed 304 before being provided to the classifier 306 which assigns “offensiveness scores” to that terms at 314. The system is trained by a “pre-labeled training set 402” shown in Figure 4 which is labeled according to the platform content policy.]
generating, by the toxicity scorer, the toxicity score for the toxic speech content as a function of the platform content policy; [Epstein, “[0011] The method can further include obtaining, in response to providing the first text sample to the classifier, a label confidence score that indicates a confidence that the label correctly indicates whether the particular potentially offensive term is used in an offensive manner in the first text sample.” “[0033] … Accordingly, after the initial training iteration of the classifier, additional unlabeled text samples can be provided to the classifier and labeled (and/or scored) by the classifier to indicate whether respective potentially offensive terms in the text samples are used in an offensive manner in the text samples….” “[0045] In some implementations, labels may be associated with an offensiveness score that indicates a degree of offensiveness of a term in a particular text sample.…a label may be represented by a score that more finely indicates how offensive or non-offensive a term is being used in a text sample. …”]
determining, by the toxicity scorer, that the toxicity score for the toxic speech content, from the first user, meets or exceeds a toxicity score threshold; [Epstein has a variable threshold/range for determining toxicity which depends on the confidence in the speech recognition stage. “[0073] … For example, a classifier may normally be configured to label text samples, including high confidence transcriptions, “non-offensive” for offensiveness scores in the range 0-5, and “offensive” for offensiveness scores in the range 6-10. However, if the transcription confidence score for a transcription is below a threshold score, the classifier may adjust the ranges that apply to each label. For example, the classifier may be configured to label low confidence transcriptions as “non-offensive” for offensiveness scores in the range 0-3, and “offensive” for offensiveness scores in the range 4-10….” This teaches a Toxicity Threshold of 6 for good transcription results and Toxicity Threshold of 4 for poor transcription results. See also: “[0074] … In some implementations, offensive terms having an offensiveness score that satisfies a threshold score may be redacted. …”]
automatically banning or muting the first user having the toxicity score for the toxic speech content that meets or exceeds the toxicity score threshold; and [Epstein’s classifier 108 is called “Offensive words classifier and redactor 108” and as an act of “moderation” it automatically redacts the words that are considered offensive: “[0074] At stage 318, one or more offensive terms are redacted from a text sample if the classifier has indicated that the terms are offensive. In some implementations, a term that has been labeled “offensive” can be redacted based on the label, and “non-offensive” labeled terms may not be redacted. In some implementations, offensive terms having an offensiveness score that satisfies a threshold score may be redacted. Redaction includes taking action to block the display of offensive portions of a text sample. Redaction may include one or more of deleting an offensive term, obscuring an offensive term with different characters, or otherwise modifying a text sample so that offensive terms are not displayed in their original form. For example, the word “shag” may be deleted or may be obscured.”]
providing the toxic speech content to a human moderator as a function of the toxicity score.
Epstein provides its “platform content policy” in the form of labeled training samples to the “offensive word classifier.” Figure 2, 208. “[0046] At stage 208, an offensive words classifier is trained using the labeled first set of text samples. Stage 208 can be the first of multiple training iterations in training the classifier. In this first iteration, initial rules and signals may be determined so as to configure the classifier to be able to recognize one or more signals (or features) associated with a text sample and to generate an offensiveness label for the text sample….” Instant Application provides: “[0016] … The content policy may be received by a manual user entry and/or by answering a questionnaire. The content policy provides a set of statistical weights that are applied to the raw scores for each toxicity category.” This conforms to the labeled set of Epstein. However, Instant Application appears to include an elaborate discussion of variations in the Content Policy which are NOT in the Claim.
Epstein redacts the offensive words and does not ban or mute the user.
Epstein, Figure 3, ends with “redact offensive terms 318.” Does not teach providing the content that is scored as toxic to a human moderator although the initial training data is labeled by humans. Later Epstein teaches: “[0044] … In some examples, reports of offensive content can be manually reviewed to determine the trustworthiness of the report before labeling the text sample as offensive, or reports may be trusted if a threshold number of reports are received that indicate a text sample is offensive.” “[0072] At stage 316, the process 300 can generate a label for the transcription of the utterance. The label can be selected based on the offensiveness score that the classifier determined for the transcription. In some implementations, respective labels can correspond to different ranges of offensiveness scores. For example, assuming that the classifier is configured to generate offensiveness scores in the range 0 through 10, transcriptions that have an offensiveness score in the range 0-5 may be labeled “non-offensive,” whereas transcriptions having an offensiveness score in the range 6-10 may be labeled “offensive.”…”
Tongya teaches:
providing a multi-stage voice content analysis system including a communication interface for receiving toxic speech content associated with an online platform, the online platform having a platform content policy, [Tongya is directed to moderating content on a “Platform for Online Multi-User Chat Service 20” and teaches that the platform administrator or some of the users may define the content policy and feedback which also is a type of policy determination is provided manually by moderators. “[0034] … The offensive language mapping data 80 may be manually generated by an administrator of the platform 20, generated using crowd-source techniques, or generated using another type of heuristic method….” “[0054]… The heuristic offensive language checking module 72 may be manually generated by moderators to flag specific words or phrases that are offensive.” “[0021] …The platform 20 for the online multi-user chat service may apply platform-wide filtering on user chat data 22 according to rules and protocols of the platform 20 in addition to the user filter preferences 36 associated with each individual user.”]
…
automatically banning or muting the first user having the toxicity score for the toxic speech content that meets or exceeds the toxicity score threshold; and [Tongya, Figure 4 shows the details of “filter decision service 48” which leads to the “filter action 64” including filtering user chat or banning the user. “[0030] … The filter decision service 48 may then determine a filter action 64 to be performed for the first user chat data 54. In the illustrated example, the filter action 64 includes filtering out the first user chat data 54, and sending the filtered user chat data 66 that does not include the first user chat data 54 to the plurality of client devices 14 in the chat session 52. It should be appreciated that other filter actions 64 may be performed optionally or in addition to the illustrated filter action. For example, the filter action 64 may include a chat restriction and/or a ban for the user profile 34 associated with the user that send the first user chat data 54. As another example, the filter action 64 may include filtering out specific words or phrases within the first user chat data 54 rather than the entire statement of the first user chat data 54.”]
providing the toxic speech content to a human moderator as a function of the toxicity score. [Tongya, Figures 3 and 4 shows that after “Filter Decision Service 48” the labels are provided to a “Moderator Computer Device 84.” See [0036]-[0037] where an “aggregated predicted label 80” is generated as a result of a “weighting technique” that weights the outputs of the different “toxicity deep learning models 46.” “[0038] As illustrated in FIG. 3, the platform 20 may include a feedback mechanism that elicits feedback from moderators for predicted labels determined by the toxicity deep learning model 44 and the filter decision service 48. In this example, the user chat filtering program 42 may be further configured to generate a feedback package 82 that includes a target portion of the user chat data 22 that was processed by the toxicity deep learning model 44. The feedback package 82 further includes the output of the plurality of trained machine learning models 46 for the target portion of user chat data 22….” “[0021] … These moderation decisions 38 may be applied automatically based on detected offensive language usage and/or applied manually by moderators….”]
Epstein and Tongya pertain to content moderation and it would have been obvious to combine the moderator review of Tongya with the automatic moderation of Epstein to provide a final more reliable human check on the results generated by the machine. This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Regarding Claim 2, Epstein does not mention a policy.
Tongya teaches:
2. The method of claim 1, further comprising:
receiving, by the toxicity scorer, feedback from the human moderator regarding whether the toxic speech content is considered to be toxic based on a content policy. [Tongya is directed to moderating content on a “Platform for Online Multi-User Chat Service 20” and teaches that the platform administrator or some of the users may define the content policy and feedback which also is a type of policy determination is provided manually by moderators. “[0034] … The offensive language mapping data 80 may be manually generated by an administrator of the platform 20, generated using crowd-source techniques, or generated using another type of heuristic method….” “[0054]… The heuristic offensive language checking module 72 may be manually generated by moderators to flag specific words or phrases that are offensive.” “[0021] …The platform 20 for the online multi-user chat service may apply platform-wide filtering on user chat data 22 according to rules and protocols of the platform 20 in addition to the user filter preferences 36 associated with each individual user.”]
Epstein and Tongya pertain to content moderation and it would have been obvious to combine the policy of Tongya with the system of Epstein which must inherently include a policy but is silent on that point for completeness. This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Regarding Claim 3, Epstein teaches:
3. The method of claim 1, further comprising:
setting, by the toxicity scorer, the toxicity score threshold for automatic moderator action. [Epstein, Figure 3 shows the grading of an input utterance by the classifier 108 of Figure 1. Ranges of scores/ thresholds are used to determine the degree of offensiveness: “[0072] At stage 316, the process 300 can generate a label for the transcription of the utterance. The label can be selected based on the offensiveness score that the classifier determined for the transcription. In some implementations, respective labels can correspond to different ranges of offensiveness scores. For example, assuming that the classifier is configured to generate offensiveness scores in the range 0 through 10, transcriptions that have an offensiveness score in the range 0-5 may be labeled “non-offensive,” whereas transcriptions having an offensiveness score in the range 6-10 may be labeled “offensive.”…”]
Regarding Claim 4, Epstein teaches:
4. The method of claim 3, further comprising:
providing, by the processor, a subset of the toxic speech content to the human moderator as a function of the toxicity score, wherein the subset of the toxic speech content has an associated toxicity score below the threshold toxicity score from a second user. [Epstein’s classifier 108 receives the words above and below the threshold score. See [0072] provided above. The training samples in the “pre-labeled training set 402” of Figure 2 include samples with toxicity/offensiveness scores above and below a threshold both. While, Epstein does not teach a human moderator as the final arbiter, the re-training of the model which is the goal of this Claim is taught by Epstein: “[0007] The classifier can be iteratively trained by performing multiple training iterations, each training iteration including providing a particular set of text samples to the classifier, obtaining labels for the particular set of text samples that were generated by the classifier in response, and re-training the classifier based at least on the particular set of text samples and the labels for the particular set of text samples that were generated by the classifier.”]
Epstein preform training by a human-labeled set and then re-trains with the labels generated by the classifier. It does not provide the output of the classifier to the same human labelers again for their input.
Tongya teaches:
providing, by the processor, a subset of the toxic speech content to the human moderator as a function of the toxicity score, wherein the subset of the toxic speech content has an associated toxicity score below the threshold toxicity score from a second user. [Tongya, Figure 4, “feedback response 88” from the human moderator (84) is going back to train the “toxicity deep learning model 44.” “[0040] … In another example, the input received from the moderator user may indicate whether the aggregated predicted label 80, or the predicted labels in the outputs from the plurality of trained machine learning models 46 are correct or incorrect.” “[0041] The server system 12 may be configured to receive a feedback response 88 from the moderator user computer device 84 that includes the moderator label 86 and/or other types of moderator inputs. The server system 12 may be configured to perform batched feedback training on the plurality of trained machine learning models 46 based on the received feedback response 82 from the moderator user computer device 84.”]
Rationale as provided for Claim 1. Either the machine or a human can be used as the arbiter of the output and human is always more accurate but more costly such that the method of Tongya improves the dataset used for re-training of the model.
Regarding Claim 6, Epstein teaches:
6. The method of claim 1, wherein a machine learning system is trained to generate a toxicity score using a toxicity score dataset. [Epstein, Figure 4, “[0004] This document further describes that the classifier can be trained using semi-supervised machine learning techniques….”]
Regarding Claim 7, Epstein teaches:
7. The method of claim 6, wherein the toxicity score dataset includes an adult language component, an audio assault component, a violent speech component, a racial hate speech component, and a gender hate speech component. [Epstein includes examples of adult, racial, and audio assault speech: “[0003] … For example, the word “shag” may be offensive in certain contexts, but not in others. Thus, “I hope we can shag tonight” may be offensive, whereas “This great wool shag has a beautiful pattern” likely is not….” “[0045] …For example, a particular potentially offensive term that is used near a racial term or other highly offensive term in a text sample may be assigned a highly offensive score….” “[0048] For example, the first set of text samples may include the following three text samples: (i) “Get away from me, you bloody old man,” (ii) “That bloodied man had better get some help fast,” and (iii) “That bloodied man was quickly heading toward unconsciousness.” …”]
Epstein does not include examples of violent or gender hate speech.
Tongya teaches:
7. The method of claim 6, wherein the toxicity score dataset includes an adult language component, an audio assault component, a violent speech component, a racial hate speech component, and a gender hate speech component. [Tongy is directed to filtering offensive speech and the types of filter can be selected and the types that are listed cover all of the options of this Claim: “[0122] In this aspect, additionally or alternatively, the user selected filter preference may be selected from a list of types of offensive language that includes a profanity type, a bullying type, an adult language type, and a hate speech type. The hate speech type may include at least one of racist hate speech type and/or sexist hate speech type.”]
Epstein and Tongya pertain to content moderation and it would have been obvious to combine the additional criteria of Tongya with those enumerated by Epstein for a more comprehensive set of checks. This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Regarding Claim 8, Epstein suggests having different components for different offensive terms, if not types, by teaching at [0040] that that classifier can be trained for one potentially offensive term at a time. But, does not expressly teach that the types of offensive speech are separately identified for components of a clip.
Tongya teaches:
8. The method of claim 7, wherein the toxicity score is provided for each individual component of a speech clip. [Tongya teaches that the filtering is performed for portions of user chat which teach the components of a speech clip. “… The user chat filtering program includes a plurality of trained machine learning models and a filter decision service that determines a filter action to be performed for target portions of user chat data based on output of the plurality of trained machine learning models for those target portions of user chat data.” Abstract. Figure 4, input of “target portion of user chat data 22.” “[0024] … After the toxicity deep learning model 44 has processed a target portion of user chat data, such as, for example, one sentence or chat entry from a user, the filter decision service 48 may determine a filter action to be performed for target portions of user chat data from based on output of the plurality of trained machine learning models 46 for those target portions of user chat data.”]
Epstein and Tongya pertain to content moderation and it would have been obvious to combine the part by part review of Tongya with the system of Epstein which is silent on this point in order to have a more fine grained measure. This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Regarding Claim 9, Epstein teaches:
9. The method of claim 7, wherein the toxicity score is provided as an overall toxicity score for a speech clip. [Epstein uses the different types of offensive speech as training samples and as a result generates an overall score that has taken into consideration different types of offensive language. “[0040] … The first set of text samples may include only text samples that include a particular potentially offensive term, or may include text samples that include at least one of multiple different potentially offensive terms. Thus, the process 200 may train a classifier for a particular potentially offensive term using only text samples that include the particular potentially offensive term at one time, or may train the classifier on multiple different potentially offensive terms at a time. In some implementations, potentially offensive terms can be identified from a pre-determined list of offensive terms (e.g., in a repository of offensive terms)….”]
Claim 22 is a M+F system claim with limitations similar to the limitations of Claim 1 and is rejected under similar rationale.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Epstein and Tongya and further in view of Hartung (U.S. 20180277113).
Regarding Claim 5, Epstein and Tongya both teach adjustment of the score and Tongya teaches adjustment of the offensiveness score according to moderator feedback. Epstein discusses adjustment of the offensiveness threshold according to the accuracy of the speech recognition (see mapping to Claim 1.) but Epstein does not send to a human and the threshold for sending to the human moderator is not changed in Tongya.
Hartung teaches:
5. The method of claim 4, further comprising:
updating, by the toxicity scorer, the toxicity score for the subset of the toxic speech content as a function of the received feedback from the human moderator; [Hartung, Figure 11 shows a feedback loop which is not for toxicity determination but is nevertheless a feedback loop: “request for additional voice input 1116.” “[0161] At block 1118, the process 1100 receives feedback from the voice service server indicating whether the closest-matched activation word was recognized by the voice service. If no, the process 1100 proceeds to block 1116 and requests additional user voice input.”]
determining, by the processor, an accuracy of the subset of the toxic speech content; and [Hartung, Figure 11, 1106, determines the confidence value /accuracy of speech recognition. “[0163] In some embodiments, the process 1100 stores different threshold values N based on voice service feedback. For example, a first threshold value N1 corresponding to a first voice service (e.g., Alexa) may be different than a second threshold value N2 corresponding to a second voice service (e.g., Google). The differences between N1 and N2 may be based on feedback received from the corresponding first and second voice services after one or more successful and/or unsuccessful recognition attempts. For instance, after several successful recognition attempts with the first voice service, the first threshold value may decrease from, for example, 50% to 45%. Conversely, after several unsuccessful recognition attempts with the second voice service, the first threshold value may increase from, for example, 50% to 60%.” The threshold of Hartung applies to speech recognition confidence: “[0149] …The implementation 1000 can determine that a confidence score greater or equal to a predetermined threshold (e.g., 40%, 50%, 60%) is a close enough match to transmit the received voice input data to the voice service corresponding to the closest match (e.g., ALEXA®).”]
adjusting, by the toxicity scorer, a toxicity score threshold for automatic moderator action, wherein adjusting reduces the toxicity score threshold. [Hartung, Figure 11, “additional processing 1112” and “updated value >= N?” and “update recognition model 1120.” “[0162] At block 1120, the process 1100 proceeds to block 1120 in response to data received from the voice service indicating a recognized activation word, in which the process 1100 performs and updates the recognition model accordingly to help improve future recognition attempts. In some embodiments, the process 1100 updating the recognition model comprises an adjustment the threshold value N based on the feedback received from the voice service at block 1118. For example, the process 1100 can increase the threshold value N in response to one or more unrecognized closest-matched activation words transmitted from the device. Conversely, the process 1100 can decrease the threshold value N in response to one or more recognized closest-matched activation words.” ]
Epstein/Tongya and Hartung are directed to speech recognition that is looking for certain specific content and it would have been obvious to combine the threshold adjustment of Hartung, which is expressed in the context of speech recognition confidence values, with the system of combination to apply threshold adjustment to the offensiveness/toxicity scoring of the combination. Both directions of threshold adjustment are common depending on policy: if the system is working well, you can lower the threshold or conversely: if you are getting too few results through, lower the threshold. This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Epstein and Tongya and further in view of Shoemake (U.S. 20150070516).
Regarding Claim 10, Epstein teaches:
10. The method of claim 6, wherein the toxicity score dataset includes an emotion component, a user context component, and an age component. [Epstein teaches that the context is considered strongly in training the classifier as shown in Figure 4, “context signals engine 422.”]
Epstein does not mention an emotion or age component.
Tongya teaches:
10. The method of claim 6, wherein the toxicity score dataset includes an emotion component, a user context component, and an age component. [Tongya looks for the overall sentiment. “[0028] … Thus, even though both statements include profanity, the first user chat data 56 may be considered to include more severe offensive language than the second user chat data due to the overall intent and sentiment of the statements.”]
Rationale for combination as provided for Claim 1.
Neither mention age as components of the score/label.
Shoemake teaches:
10. The method of claim 6, wherein the toxicity score dataset includes an emotion component, a user context component, and an age component. [Shoemake is directed to content filtering and sets up the filters according to age and emotion which requires the training data to be according to age and emotion as well. “[0158] Method 200, at block 225, might comprise estimating, with the presence detection device, characteristics of the user (including, without limitation, age, gender, demographics, and/or the like), based at least in part on information derived from at least a portion of the presence information.….” “[0173] … In some instances, the audience-based information might be monitored using one or more of … mood recognition techniques, emotion recognition techniques, voice recognition techniques, vocal tone recognition techniques, speech recognition techniques … and/or the like.”]
Epstein/Tongya and Shoemake are directed to content filtering and it would have been obvious to add the sentiment and age factors from Shoemake to the factors set forth by the combination because as Shoemake indicates these also impact the intent and