CTNF 18/503,283 CTNF 98338 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Rejections - 35 USC § 112: Indefiniteness 07-30-02 AIA The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. 07-34-01 Claims 2, 4, 6, 9, 12, 16, and 18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Regarding claim 2 , the claim recites the limitation wherein said fairness weight score with one or more bias are a fraction of fairness weight scores with no bias. The claim is indefinite as it is unclear how a fairness weight score can have no bias when the fairness weight scores are based on biases in claim 1. For the purposes of examination, the claim is interpreted as having multiple fairness weight scores. Regarding claims 4 and 6 , the claims are rejected for at least their dependence on claim 2. Regarding claim 9 , the claim recites the limitation wherein said fairness weight score with one or more bias are a fraction of fairness weight scores with no bias. The claim is indefinite as it is unclear how a fairness weight score can have no bias when the fairness weight scores are based on biases in claim 8. For the purposes of examination, the claim is interpreted as having multiple fairness weight scores. Regarding claim 12 , the claim is rejected for at least its dependence on claim 9. Regarding claim 16 , the claim recites the limitation wherein said fairness weight score with one or more bias are a fraction of fairness weight scores with no bias. The claim is indefinite as it is unclear how a fairness weight score can have no bias when the fairness weight scores are based on biases in claim 15. For the purposes of examination, the claim is interpreted as having multiple fairness weight scores. Regarding claim 18 , the claim is rejected for at least its dependence on claim 16. Claim Rejections - 35 USC § 101 07-04-01 AIA 07-04 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1-20 are rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. Regarding claim 1 , in step 1 of the 101 analysis set forth in MPEP 2106, the claim recites A method for evaluating content moderation by training an Artificial Intelligence (AI) engine, comprising: . The claim recites a method. A method is one of the four statutory categories of invention. In Step 2A, Prong 1 of the 101 analysis set forth in MPEP 2106, the examiner has determined that the following limitations recite a process that, under broadest reasonable interpretation, covers a mental process or mathematical concept but for the recitation of generic computer components: generating a plurality of current labels for said plurality of data using an associated category and a topic; (i.e., the broadest reasonable interpretation includes a step of observation, evaluation, and judgement and could be performed mentally or with pen and paper like creating labels of data, which is either a mental process of observation/evaluation/judgement (MPEP 2106) ). determining if said current labels or said past labeled data have one or more associated biases, wherein said one or more associated biases are based on a bias criteria ; (i.e., the broadest reasonable interpretation includes a step of observation, evaluation, and judgement and could be performed mentally or with pen and paper like determining bias of labels, which is either a mental process of observation/evaluation/judgement (MPEP 2106) ). calculating a fairness weight score for each current label based on a number of biases determined to be associated with said current label and whether said current label is associated with any past label data determined to have one or more biases; (i.e., the broadest reasonable interpretation includes mathematical calculation like calculating a fairness weight score, a mathematical calculation is considered a mathematical concept (MPEP 2106) ). generating a final label for each plurality of current labels based on said fairness weight score; (i.e., the broadest reasonable interpretation includes a step of observation, evaluation, and judgement and could be performed mentally or with pen and paper like creating a label based on a fairness score, which is either a mental process of observation/evaluation/judgement (MPEP 2106) ). If the claim limitations, under their broadest reasonable interpretation, covers activities classified under Mental processes: concepts performed in the human mind (including observation, evaluation, judgement, or opinion) (see MPEP 2106.04(a)(2), subsection (III)) or Mathematical concepts: mathematical relationships, mathematical formulas or equations, or mathematical calculations (see MPEP 2106.04(a)(2), subsection (I)). Accordingly, the claim recites an abstract idea. In Step 2A, Prong 2 of the 101 analysis, set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application : obtaining a plurality of data to be used for training said AI engine; (i.e., the broadest reasonable interpretation of receiving a data instance is mere data gathering, which is an insignificant extra solution activity (MPEP 2106.05(g)) ). collecting any past labeled data that is similar to said plurality of current labels; (i.e., the broadest reasonable interpretation of receiving a data instance is mere data gathering, which is an insignificant extra solution activity (MPEP 2106.05(g)) ). and using said final label for training said AI engine. (i.e., the generic computer components recited in this limitation merely add the words “apply it”, or an equivalent, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)) ). Since the claim does not contain any other additional elements, that amount to integration into a practical application, the claim is directed to an abstract idea. In Step 2B of the 101 analysis set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception: Regarding limitation(s) (V and VI), under the broadest reasonable interpretation, recite steps of mere data gathering, which has been recognized by the courts as being well-understood, routine, and conventional functions. Specifically, the courts have recognized computer functions directed to mere data gathering as well-understood, routine, and conventional functions when they are claimed in a merely generic manner or as insignificant extra-solution activity when considering evidence in view of Berkheimer v. HP, Inc., 881 F.3d 1360, 1368, 125 USPQ2d 1649, 1654 (Fed. Cir. 2018), see USPTO Berkheimer Memorandum (April 2018)). Examiner uses Berkheimer: Option 2, a citation to one or more of the court decisions discussed in MPEP 2106.05(d)(II) as noting well-understood, routine, and conventional nature of the additional elements: Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network). See MPEP 2106.05(d)(II). Further, limitation (VII), under the broadest reasonable interpretation, merely recite steps that apply generic training using data, which represents merely adding the words “apply it”, or an equivalent, which are not indicative of an inventive concept (MPEP 2106.05(f)). Considering additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible. Regarding claim 2 , it is dependent upon claim 1 and fails to resolve the deficiencies identified above by integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. For example, claim 2 recites wherein said fairness weight score with one or more bias are a fraction of fairness weight scores with no bias. Under the broadest reasonable interpretation, the limitations recite calculating the fairness weight score based on fractions which is interpreted as using a mathematical calculation. A mathematical calculation is interpreted as a mathematical concept. Therefore, claim 2 does not solve the deficiencies of claim 1. Regarding claim 3 , it is dependent upon claim 1 and fails to resolve the deficiencies identified above by integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. For example, claim 3 recites wherein each current label has a plurality of data elements. Under the broadest reasonable interpretation, the limitations recite determining labels with multiple elements which is a step of observation, evaluation, and judgement which can be performed mentally or with pen and paper. The steps of observation, evaluation, and judgement are mental processes. Therefore, claim 3 does not solve the deficiencies of claim 1. Regarding claim 4 , it is dependent upon claim 2 and fails to resolve the deficiencies identified above by integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. For example, claim 4 recites wherein said updated final label is iteratively analyzed and said weight scores are updated and provided to said AI engine as new data is obtained. Under the broadest reasonable interpretation, the limitations recite determining updated labels multiple times which is a step of observation, evaluation, and judgement which can be performed mentally or with pen and paper. The steps of observation, evaluation, and judgement are mental processes. Therefore, claim 4 does not solve the deficiencies of claim 2. Regarding claim 5 , it is dependent upon claim 1 and fails to resolve the deficiencies identified above by integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. For example, claim 5 recites wherein fairness weight scores are adjusted over time based on analysis of data. Under the broadest reasonable interpretation, the limitations recite calculating the fairness weight score multiple times which is interpreted as using a mathematical calculation. A mathematical calculation is interpreted as a mathematical concept. Therefore, claim 5 does not solve the deficiencies of claim 1. Regarding claim 6 , it is dependent upon claim 2 and fails to resolve the deficiencies identified above by integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. For example, claim 6 recites wherein said AI engine has one or more machine language models. Under the broadest reasonable interpretation, the limitations merely recite steps that apply a generic machine learning model as a tool to perform a judicial exception, which represents merely adding the words “apply it”, or an equivalent, which are not indicative of an inventive concept (MPEP 2106.05(f)). Therefore, claim 6 does not solve the deficiencies of claim 2. Regarding claim 7 , it is dependent upon claim 1 and fails to resolve the deficiencies identified above by integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. For example, claim 7 recites wherein said AI engine generates a plurality of predicted outcome and updaters and modifies said predicted outcome based on said final label. Under the broadest reasonable interpretation, the limitations recite outputting outcomes and updates which merely recite steps of mere data outputting, which has been recognized by the courts as being well-understood, routine, and conventional functions. Specifically, the courts have recognized computer functions directed to mere data outputting as well-understood, routine, and conventional functions when they are claimed in a merely generic manner or as insignificant extra-solution activity (MPEP 2106.05(g)). Therefore, claim 7 does not solve the deficiencies of claim 1. Regarding claim 8 , in step 1 of the 101 analysis set forth in MPEP 2106, the claim recites A computer system for evaluating content moderation by training an Artificial Intelligence (AI) engine, comprising: one or more processors, one or more computer-readable memories and one or more computer-readable storage media; . The claim recites a system with hardware elements which is interpreted as a machine. A machine is one of the four statutory categories of invention. For the Step 2A/2B analyses, since claim 8 is similar to claim 1 it is rejected under the same rationales as claim 1. The additional limitation below fails to resolve the deficiencies identified above by integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. A computer system for evaluating content moderation by training an Artificial Intelligence (AI) engine, comprising: one or more processors, one or more computer-readable memories and one or more computer-readable storage media; (i.e., the generic computer components recited in this limitation merely add the words “apply it”, or an equivalent, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)) ). program instructions, stored on at least one of the one or more storage media for execution by at least one of the one or more processors via at least one of the one or more memories, (i.e., the generic computer components recited in this limitation merely add the words “apply it”, or an equivalent, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)) ). Considering additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible. Regarding claims 9-14 , the claims are similar to claims 2-7 and are rejected under the same rationales. Regarding claim 15 , in step 1 of the 101 analysis set forth in MPEP 2106, the claim recites A computer program product for evaluating content moderation by training an Artificial Intelligence (AI) engine comprising: one or more computer readable storage media; . The claim recites a non-transitory computer readable medium, see Spec. ⁋15, which is interpreted as an article of manufacture. An article of manufacture is one of the four statutory categories of invention. For the Step 2A/2B analyses, since claim 15 is similar to claim 1 it is rejected under the same rationales as claim 1. The additional limitation below fails to resolve the deficiencies identified above by integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. A computer program product for evaluating content moderation by training an Artificial Intelligence (AI) engine comprising: one or more computer readable storage media; (i.e., the generic computer components recited in this limitation merely add the words “apply it”, or an equivalent, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)) ). program instructions, stored on at least one of the one or more storage media for execution by at least one of the one or more processors via at least one of the one or more memories, (i.e., the generic computer components recited in this limitation merely add the words “apply it”, or an equivalent, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)) ). Considering additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible. Regarding claims 16-20 , the claims are similar to claims 2-7 and are rejected under the same rationales. Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim s 1, 8, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Ghai, et al., Non-Patent Literature “Measuring Social Biases of Crowd Workers using Counterfactual Queries” (“Ghai”) in view of Yaghi, et al., US Pre-Grant Publication US20240202571A1 (“Yaghi”) . Regarding claim 1 , Ghai discloses: A method for evaluating content moderation by training an Artificial Intelligence (AI) engine, (Ghai, abstract, “Crowdsourcing, a popular cost effective measure to gather labeled training datasets, is not immune to the inherent social biases of crowd workers…In this work, we propose a new method based on counterfactual fairness to quantify the degree of inherent social bias in each crowd worker. This extra information can be leveraged together with individual worker responses to curate a less biased dataset [ by training an Artificial Intelligence (AI) engine, ].”, and Ghai, pg. 2 col. 1, “Given a labeling task which consists of a sensitive attribute like gender, race, etc., a crowd worker will be considered fair if she provides the same label for a query and its counterfactual. For instance a crowd worker is tasked to label a text statement for toxicity [ A method for evaluating content moderation ].”). comprising: obtaining a plurality of data to be used for training said AI engine; (Ghai, pg. 2 col. 1, “Crowdsourcing is widely used to curate training datasets for ML models [ comprising: obtaining a plurality of data to be used for training said AI engine; ].”). generating a plurality of current labels for said plurality of data using an associated category and a topic; (Siegel, pg. 3 col. 2, “Let’s consider a toy problem where crowd workers are asked to predict recidivism. As shown in fig.1 (a), a crowd worker is presented with a set of features representing a convict. The task is to predict the likelihood of re-offending within 2 years on a 1-5 scale. Let’s say each crowd worker is asked to label x such queries [ generating a plurality of current labels for said plurality of data ]. We will choose a subset of size n out of x queries and generate counterfactuals for only those queries Q; Figure 1 shows that a query has multiple fields and topics (i.e. using an associated category and a topic; ).”). determining if said current labels…have one or more associated biases, wherein said one or more associated biases are based on a bias criteria; (Siegel, pg. 4 col. 1, “The mean absolute difference in labels provided for all n pairs of Q and CQ will represent the bias score for a crowd worker as shown eq.1 [ determining if said current labels…have one or more associated biases, wherein said one or more associated biases are based on a bias criteria; ].”). calculating a fairness weight score for each current label based on a number of biases determined to be associated with said current label…; (Siegel, pg. 4 col. 1, “The mean absolute difference in labels provided for all n pairs of Q and CQ will represent the bias score for a crowd worker as shown eq.1. WorkerBias = 1 n n i=1 |Label(Qi) − Label(CQi)| (1) A zero score characterizes perfect unbiased behavior and higher values symbolize more biased behavior [ based on a number of biases determined to be associated with said current label…; ]. This extra information will be used in conjunction with the crowd worker responses to yield fairer labels and hence fairer datasets. A simple way to incorporate this information might be to filter out biased workers whose bias score is beyond a threshold [ calculating a fairness weight score for each current label ].”). generating a final label for each plurality of current labels based on said fairness weight score; and using said final label for training said AI engine. (Siegel, pg. 4 col. 1, “A zero score characterizes perfect unbiased behavior and higher values symbolize more biased behavior. This extra information will be used in conjunction with the crowd worker responses to yield fairer labels and hence fairer datasets. A simple way to incorporate this information might be to filter out biased workers whose bias score is beyond a threshold [ generating a final label for each plurality of current labels based on said fairness weight score; and using said final label for training said AI engine. ].”). While Ghai teaches a system that determines the mitigates bias between multiple annotators for labeling training data, Ghai does not explicitly teach: a collecting any past labeled data that is similar to said plurality of current labels; …or said past labeled data… has bias …and whether said current label is associated with any past label data determined to have one or more biases… Ghai teaches: collecting any past labeled data that is similar to said plurality of current labels; (Yaghi, ⁋52, “In some embodiments, the system may determine similarity metrics between two different labels, such as those determined at different times for the same training datum. For example, the system may determine a label similarity metric by comparing the text within labels, and use this similarity metric to determine whether to generate a warning based on a label similarity threshold. By generating similarity metrics, the system may objectively determine whether datasets or labels are related to each other, and, as such, the system may make improved recommendations for labels for both pre-existing datasets and new data received [ collecting any past labeled data that is similar to said plurality of current labels; ].”). …or said past labeled data… has bias (Yaghi, ⁋52, “By generating similarity metrics, the system may objectively determine whether datasets or labels are related to each other, and, as such, the system may make improved recommendations for labels for both pre-existing datasets and new data received. By leveraging an objective metric, such as similarity metrics, the system may reduce the effect of human bias, error or inconsistencies in labeling tasks, leading to improved machine learning outcomes (e.g., lower model error indicators) [ …or said past labeled data… has bias ].”). …and whether said current label is associated with any past label data determined to have one or more biases… (Yaghi, ⁋52, “By generating similarity metrics, the system may objectively determine whether datasets or labels are related to each other, and, as such, the system may make improved recommendations for labels for both pre-existing datasets and new data received. By leveraging an objective metric, such as similarity metrics, the system may reduce the effect of human bias, error or inconsistencies in labeling tasks, leading to improved machine learning outcomes (e.g., lower model error indicators) [ …and whether said current label is associated with any past label data determined to have one or more biases… ].”). Ghai and Yaghi are both in the same field of endeavor (i.e. training data labeling). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ghai and Yaghi to teach the above limitation(s). The motivation for doing so is that finding similarities between a label and previous labels can improve model performance by leveraging prior knowledge (cf. Yaghi, ⁋52, “By leveraging an objective metric, such as similarity metrics, the system may reduce the effect of human bias, error or inconsistencies in labeling tasks, leading to improved machine learning outcomes (e.g., lower model error indicators).”). Regarding claim 8 , the claim is similar to claim 1 and rejected under the same rationales. Yaghi teaches the additional limitations: A computer system…comprising: one or more processors, one or more computer-readable memories and one or more computer-readable storage media; (Yaghi, ⁋160, “A16. A tangible, non-transitory, machine-readable medium storing instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform operations comprising those of any of embodiments A1-A15. A17. A system comprising one or more processors; and memory storing instructions that, when executed by the processors, cause the processors to effectuate operations comprising those of any of embodiments A1-A15 [ A computer system…comprising: one or more processors, one or more computer-readable memories and one or more computer-readable storage media; ].”). program instructions, stored on at least one of the one or more storage media for execution by at least one of the one or more processors via at least one of the one or more memories, (Yaghi, ⁋160, “and memory storing instructions that, when executed by the processors, cause the processors to effectuate operations”). Ghai and Yaghi are both in the same field of endeavor (i.e. training data labeling). Ghai teaches a base method for labeling training data. Yaghi teaches a known technique of using a computer to perform training data labeling. It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ghai and Yaghi to teach the above limitation(s). The motivation for doing so is that applying Yaghi’s known technique of using a computer to perform training data labeling to Ghai’s base system of labeling training data would yield predictable results. Regarding claim 15 , the claim is similar to claim 1 and rejected under the same rationales. Yaghi teaches the additional limitations: A computer program product…comprising: one or more computer readable storage media; (Yaghi, ⁋160, “A16. A tangible, non-transitory, machine-readable medium storing instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform operations comprising those of any of embodiments A1-A15. A17. [ A computer program product…comprising: one or more computer readable storage media; ].”). program instructions, stored on at least one of the one or more storage media for execution by at least one of the one or more processors via at least one of the one or more memories, (Yaghi, ⁋160, “and memory storing instructions that, when executed by the processors, cause the processors to effectuate operations”). Ghai and Yaghi are both in the same field of endeavor (i.e. training data labeling). Ghai teaches a base method for labeling training data. Yaghi teaches a known technique of using a computer to perform training data labeling. It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ghai and Yaghi to teach the above limitation(s). The motivation for doing so is that applying Yaghi’s known technique of using a computer to perform training data labeling to Ghai’s base system of labeling training data would yield predictable results . 07-21-aia AIA Claim s 2-3, 6, 9-11, 16-17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Ghai, et al., Non-Patent Literature “Measuring Social Biases of Crowd Workers using Counterfactual Queries” (“Ghai”) in view of Yaghi, et al., US Pre-Grant Publication US20240202571A1 (“Yaghi”) and further in view of Marchal, et al., Non-Patent Literature “Establishing Annotation Quality in Multi-Label Annotations” (“Marchal”) . Regarding claim 2 , Ghai in view of Yaghi teaches the method of claim 1 . While the combination teaches a system that determines the mitigates bias between multiple annotators for labeling training data, the combination does not explicitly teach wherein said fairness weight score with one or more bias are a fraction of fairness weight scores with no bias. Marchal teaches wherein said fairness weight score with one or more bias are a fraction of fairness weight scores with no bias. (Marchal, pg. 3661 col. 2, “The observed agreement of an item i is then defined by: Ai o = Wk′ c1Wk′ k′ ∈ k′ i c2 (3) where k′ i is the set of intersecting labels of item i and Wk′ c1 and Wk′ c2 are the weights of the label annotated by each coder. The overall observed agreement (Ao) is the mean value of the observed agreement per item (Ai o) [ wherein said fairness weight score with one or more bias are a fraction of fairness weight scores with no bias. ].”). Ghai, in view of Yaghi, and Marchal are both in the same field of endeavor (i.e. data annotation). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ghai, in view of Yaghi, and Marchal to teach the above limitation(s). The motivation for doing so is that considering agreement between annotators reduces the bias involved with a label (cf. Marchal, pg. 3666 col. 2, “the goal of agreement measures is not to establish how strongly the coders agree, but rather how reliable the label is. The label on which annotators agree is more likely to be a true label”). Regarding claim 3 , Ghai in view of Yaghi teaches the method of claim 1 . While the combination teaches a system that determines the mitigates bias between multiple annotators for labeling training data, the combination does not explicitly teach wherein each current label has a plurality of data elements. Marchal teaches wherein each current label has a plurality of data elements. (Marchal, abstract, “In many linguistic fields requiring annotated data, multiple interpretations of a single item are possible. Multi-label annotations more accurately reflect this possibility [ wherein each current label has a plurality of data elements. ].”). Ghai, in view of Yaghi, and Marchal are both in the same field of endeavor (i.e. data annotation). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ghai, in view of Yaghi, and Marchal to teach the above limitation(s). The motivation for doing so is that multi-labeling improves the accuracy of a label by leveraging multiple sources (cf. Marchal, pg. 3659 col. 2, “Allowing for multi-label coding can result in more accurate annotations as well as higher agreement, and with that most likely higher reliability of a final label.”). Regarding claim 6 , Ghai in view of Yaghi and Marchal teaches the method of claim 2 . Ghai further teaches wherein said AI engine has one or more machine language models. (Ghai, pg. 2 col. 1, “Crowdsourcing is widely used to curate training datasets for ML models [ wherein said AI engine has one or more machine language models. ].”). Regarding claims 9 and 16 , the claims are similar to claim 2 and are rejected under the same rationales. Regarding claims 10-11 and 17 , the claims are similar to claim 3 and are rejected under the same rationales. Regarding claim 20 , the claim is similar to claim 6 and is rejected under the same rationales . 07-21-aia AIA Claim s 4, 12, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Ghai, et al., Non-Patent Literature “Measuring Social Biases of Crowd Workers using Counterfactual Queries” (“Ghai”) in view of Yaghi, et al., US Pre-Grant Publication US20240202571A1 (“Yaghi”) and further in view of Marchal, et al., Non-Patent Literature “Establishing Annotation Quality in Multi-Label Annotations” (“Marchal”) and Duesterwald, et al., US Pre-Grant Publication US20190042953A1 (“Duesterwald”) . Regarding claim 4 , Ghai in view of Yaghi and Marchal teaches the method of claim 2 . While the combination teaches a system that determines the mitigates bias between multiple annotators for labeling training data by using weighted values, the combination does not explicitly teach wherein said updated final label is iteratively analyzed and said weight scores are updated and provided to said AI engine as new data is obtained. Duesterwald teaches wherein said updated final label is iteratively analyzed and said weight scores are updated and provided to said AI engine as new data is obtained. (Duesterwald, ⁋54, “In other words, the trust score of an annotator is updated and adjusted in response to probing said annotator [ wherein said updated final label is iteratively analyzed ]. Probe mod 325 may also adjust the trust score based on the determined risk and the action taken for the sample (i.e., whether the sample was accepted, rejected, or delayed). In some embodiments of the present invention, the trust score may be adjusted using a trust score aging scheme where the trust score is automatically adjusted based on usage and elapsed time. In other embodiments, the trust score may be adjusted based on the results of the probes [ and said weight scores are updated and provided to said AI engine as new data is obtained. ].”). Ghai, in view of Yaghi and Marchal, and Duesterwald are both in the same field of endeavor (i.e. data annotation). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ghai, in view of Yaghi and Marchal, and Duesterwald to teach the above limitation(s). The motivation for doing so is to ensure that annotators’ reliabilities are up to date which improves model performance (cf. Duesterwald, ⁋2 and ⁋20). Regarding claims 12 and 18 , the claims are similar to claim 4 and are rejected under the same rationales . 07-21-aia AIA Claim s 5, 7, 13-14, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Ghai, et al., Non-Patent Literature “Measuring Social Biases of Crowd Workers using Counterfactual Queries” (“Ghai”) in view of Yaghi, et al., US Pre-Grant Publication US20240202571A1 (“Yaghi”) and further in view of Duesterwald, et al., US Pre-Grant Publication US20190042953A1 (“Duesterwald”) . Regarding claim 5 , Ghai in view of Yaghi teaches the method of claim 1 . While the combination teaches a system that determines the mitigates bias between multiple annotators for labeling training data, the combination does not explicitly teach wherein fairness weight scores are adjusted over time based on analysis of data. Duesterwald teaches wherein fairness weight scores are adjusted over time based on analysis of data. (Duesterwald, ⁋54, “In some embodiments of the present invention, the trust score may be adjusted using a trust score aging scheme where the trust score is automatically adjusted based on usage and elapsed time. In other embodiments, the trust score may be adjusted based on the results of the probes [ wherein fairness weight scores are adjusted over time based on analysis of data. ].”). Ghai, in view of Yaghi, and Duesterwald are both in the same field of endeavor (i.e. data annotation). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ghai, in view of Yaghi, and Duesterwald to teach the above limitation(s). The motivation for doing so is to ensure that annotators’ reliabilities are up to date which improves model performance (cf. Duesterwald, ⁋2 and ⁋20). Regarding claim 7 , Ghai in view of Yaghi teaches the method of claim 1 . While the combination teaches a system that determines the mitigates bias between multiple annotators for labeling training data, the combination does not explicitly teach wherein said AI engine generates a plurality of predicted outcome and updaters and modifies said predicted outcome based on said final label. Duesterwald teaches wherein said AI engine generates a plurality of predicted outcome and updaters and modifies said predicted outcome based on said final label. (Duesterwald, ⁋64, “Active Learning refers to a semi-supervised machine learning approach where a user (annotator) is iteratively queried to label selected training samples [ wherein said AI engine generates a plurality of predicted outcome ]. Different active learning tools may have different ways to select and present unlabeled samples to annotators. A goal of active learning is to minimize the number of samples necessary while also optimizing the performance of the model trained from a resulting ground truth; supervised learning is interpreted as constantly updating a model’s parameters based on comparing a prediction to a label (i.e. and updaters and modifies said predicted outcome based on said final label. ).”). Ghai, in view of Yaghi, and Duesterwald are both in the same field of endeavor (i.e. data annotation). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ghai, in view of Yaghi, and Duesterwald to teach the above limitation(s). The motivation for doing so is that training a model optimizes the performance of the model (cf. Duesterwald, ⁋64). Regarding claims 13 and 19 , the claims are similar to claim 5 and are rejected under the same rationales. Regarding claim 14 , the claim is similar to claim 7 and is rejected under the same rationales. Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS S WU whose telephone number is (571)270-0939. The examiner can normally be reached Monday - Friday 8:00 am - 4:00 pm EST. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached at 571-431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /N.S.W./Examiner, Art Unit 2148 /SHERROD L KEATON/Primary Examiner, Art Unit 2148 Application/Control Number: 18/503,283 Page 2 Art Unit: 2148 Application/Control Number: 18/503,283 Page 3 Art Unit: 2148 Application/Control Number: 18/503,283 Page 4 Art Unit: 2148 Application/Control Number: 18/503,283 Page 5 Art Unit: 2148 Application/Control Number: 18/503,283 Page 6 Art Unit: 2148 Application/Control Number: 18/503,283 Page 7 Art Unit: 2148 Application/Control Number: 18/503,283 Page 8 Art Unit: 2148 Application/Control Number: 18/503,283 Page 9 Art Unit: 2148 Application/Control Number: 18/503,283 Page 10 Art Unit: 2148 Application/Control Number: 18/503,283 Page 11 Art Unit: 2148 Application/Control Number: 18/503,283 Page 12 Art Unit: 2148 Application/Control Number: 18/503,283 Page 13 Art Unit: 2148 Application/Control Number: 18/503,283 Page 14 Art Unit: 2148 Application/Control Number: 18/503,283 Page 15 Art Unit: 2148 Application/Control Number: 18/503,283 Page 16 Art Unit: 2148 Application/Control Number: 18/503,283 Page 17 Art Unit: 2148 Application/Control Number: 18/503,283 Page 18 Art Unit: 2148 Application/Control Number: 18/503,283 Page 19 Art Unit: 2148 Application/Control Number: 18/503,283 Page 20 Art Unit: 2148 Application/Control Number: 18/503,283 Page 21 Art Unit: 2148 Application/Control Number: 18/503,283 Page 22 Art Unit: 2148 Application/Control Number: 18/503,283 Page 23 Art Unit: 2148 Application/Control Number: 18/503,283 Page 24 Art Unit: 2148 Application/Control Number: 18/503,283 Page 25 Art Unit: 2148
Prosecution Insights