DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
1. Regarding the claim objections, Applicant has cancelled claims 6 and 11 and has amended claims 13 and 14 to address the minor informalities. Accordingly, the objections are withdrawn.
2. Regarding the rejection under 35 U.S.C. § 101, Applicant's arguments filed 03/30/2026 have been fully considered but they are not persuasive.
Applicant argues on pgs. 9-10 that the claims are eligible under 35 U.S.C. § 101.
Step 2A Prong 1
Applicant first argues that the claims do not recite abstract ideas under Step 2A Prong 1, stating that the claimed invention does not recite mental processes nor mathematical concepts. The Examiner respectfully disagrees. First, the claimed invention recites several steps which can be performed by a person with the aid of pen and paper. A person can analyze a data entry and a prompt/question to determine if it is a false-positive (e.g. a movie is incorrectly identified as being a comedy), and in response can write down a test and further write down predictions related to this test. The claimed machine learning models and checker are recited at a high level of generality, and do not preclude these steps from being performed mentally as they do not impose any meaningful limits on practicing these mental processes.
Furthermore, the claim recites mathematical concepts. The limitations broadly reciting computing of loss functions and updating of parameters based on minimizations of a loss function amounts to mathematical calculations. Therefore, the claims recite abstract ideas.
Step 2A Prong 2
Applicant further argues that the claims integrate the judicial exception into a practical application, as the application operation grounds the claim in a real-word technical output, and the claims recite a concrete pipeline which improves retrieval/recommendation predictions for a downstream content system. The Examiner respectfully disagrees. While the Examiner agrees that the application step does help show that this model is being used in some way, the claims as currently written do not reflect a technical improvement. While generic computer elements are recited (checkers, machine learning models, further machine learning models), no specific architectures/components/interconnections nor specific loss function formulations/training procedures beyond merely computing and minimizing a generic loss function for generic machine learning models are recited. Thus, these components amount to mere instructions to implement the judicial exception using a generic computer, which do not integrate the judicial exception into a practical application as they do not impose any meaningful limits on practicing the abstract ideas. Furthermore, there are no limitations present in the claims which reflect this improvement to retrieval/recommendation tasks for a content system, as argued by Applicant, as the application step only broadly recites to “apply” the model. Therefore, the claims are directed to abstract ideas.
Step 2B
Applicant further argues that the claims amount to significantly more than the judicial exception as they recite a specific, unconventional technical pipeline which address technical problems of false positives and incorrect predictions in machine learning models. The Examiner respectfully disagrees. As discussed above regarding Step 2A Prong 2, the additional elements recited in the claims (checkers, machine learning models) which are not mental processes or mathematical concepts amount to generic computer components. These generic computer components do not amount to significantly more than the judicial exception as they do not provide an inventive concept. Therefore, the claims are not patent eligible.
Hence, Applicant’s arguments are not persuasive.
3. Regarding the rejections under 35 U.S.C. § 102 and 103, Applicant’s arguments have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
4. Claims 1-2, 7, 13-16, 18-19, and 21-31 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1, “A method” is recited, which is directed to one of the four statutory categories of invention (process) (Step 1: YES). However, the claims limitations, under their broadest reasonable interpretation, recite mental processes or mathematical concepts which fall into the category of abstract idea (Step 2A Prong 1: YES).
The following limitations, under their broadest reasonable interpretation, recite mental processes or mathematical concepts:
receiving a positive labeled data entry comprising a query, an identifier for a media content item, and a positive match value corresponding to the query and the media content item: a person obtains labeled data with a query (e.g. particular command/request/category), an identifier (e.g. a title of a particular film), and a label (e.g. 0 or 1 if item matches or does not match with query)
translating the positive labeled data entry into a prompt, wherein the prompt includes a question or request to output a match value for the query and the media content item: a person writes down a prompt using the labeled data (e.g. “Does [content item] relate to [query]?”)
generating… one or more predictions in response to receiving the prompt: a person determines predictions based on the prompt (determines if yes or no)
determining …that the positive labeled data entry is a false-positive data entry, based on the one or more predictions being associated with a negative match value: a person makes a decision based on an output of the ML model (e.g. decides that the labeled data is a false positive if the output says “NO”)
in response to determining that the positive labeled data entry is the false-positive data entry, generating a test from the positive labeled data entry, wherein the test represents a negative training sample: a person writes down a test (e.g. a further written prompt) using the labeled data, where the test represents negative training sample
generating…a test prediction using the test: a person writes down a test prediction by analyzing the test
computing a loss function based on the test and the test prediction: computing of loss function amounts to mathematical calculations
updating one or more parameters of the further machine learning model to minimize the loss function: updating the model by minimizing a loss function amounts to a mathematical concept.
applying…to a further media content item and a further query to generate a further prediction: a person uses the previous test to inform their decision about a new query/media content item to write down another prediction
Claim 1 does not contain any additional elements which integrate the judicial exception into a practical application (Step 2A Prong 2: NO). The only additional limitations are “generating, by one or more machine learning models”, “determining, by a checker”, “generating, by a further machine learning model”, “updating one or more parameters of the further machine learning model”, and “applying the further machine learning model having one or more updated parameters”. These limitations are recited at a high level of generality and amount to mere instructions to implement the judicial exception using a generic computer. Even when viewed in combination, mere instructions to implement the judicial exception using a generic computer do not integrate the judicial exception into a practical application as they do not impose any meaningful limits on practicing the judicial exception. Accordingly, claim 1 is directed to an abstract idea.
Claim 1 does not contain any additional elements which amount to significantly more than the judicial exception (Step 2B: NO). As discussed above, the only additional limitations are mere instructions to implement the judicial exception using a generic computer, which even when viewed in combination do not amount to significantly more than the judicial exception as they do not provide an inventive concept. Therefore, claim 1 is not patent eligible.
Regarding dependent claims 2, 7 and 21-31, “The method” is recited, which is directed to one of the four statutory categories of invention (process) (Step 1: YES). However, the claims limitations, under their broadest reasonable interpretation, recite further mental processes or mathematical concepts which fall into the category of abstract idea (Step 2A Prong 1: YES).
The following limitations, under their broadest reasonable interpretation, recite further mental processes or mathematical concepts:
Claim 2:
wherein the prompt includes a title of the content item, the query, and metadata associated with the content item: a person writes down a prompt using the content item, query, and metadata (writes down “Does [content item] relate to [query] given the following: [metadata]?”)
Claim 7:
wherein the test comprises the query and the media content item: a person writes down a test including the query and content item
Claim 21:
computing the loss function comprises computing a triplet loss function using the positive labeled data entry as a negative ground truth label; and updating the one or more parameters of the further machine learning model comprises maximizing a distance of predictions from the negative ground truth label: computing a triplet loss by maximizing a distance of predictions from negative ground truth labels amounts to a mathematical calculation
Claim 22:
computing the loss function comprises computing the loss function further based on randomly selected negative training samples from a pool of negative training samples: computing a loss from random sampling amounts to a mathematical calculation
Claim 23:
wherein computing the loss function further based on selected negative training samples meeting a criterion involving one or more of distance, similarity, and loss: selecting samples based on a distance, similarity, or loss amounts to mathematical calculations
Claim 24:
wherein computing the loss function further based on an adjustable number of selected negative training samples: computing a loss based on a number of negative training samples amounts to a mathematical calculations
Claim 25:
in response to determining that the positive labeled data entry is the false-positive data entry, modifying the positive match value based on the one or more predictions: a person rewrites a match value to be correct based on the predictions (e.g. rewrites a data entry so the match value is ‘0’ instead of ‘1’)
Claim 26:
wherein generating the one or more predictions comprises generating a plurality of predictions … in response to receiving the prompt: a person writes down multiple predictions in response to receiving a prompt/question
Claim 26 contains the additional limitation “generating a plurality of predictions by an ensemble of expert large language models”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claim 27:
…searches a data source for a number of references to confirm whether the media content item matches the query: a person can search several data sources/references/documents to determine if a media content matches the query
Claim 27 contains the additional limitation “wherein the ensemble of the expert large language models comprises a fact checking model that searches…”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claim 28:
Claim 28 contains the additional limitation “wherein the ensemble of the expert large language models comprises a plurality of models being trained on different training data sets or having different machine learning architectures”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claim 29:
wherein determining that the positive labeled data entry is the false-positive data entry comprises: combining the plurality of predictions into a combined prediction; and evaluating whether the combined prediction is consistent with the positive match value of the positive labeled data entry: a person combines information from multiple predictions to determine if a false positive occurs (e.g. use majority rule to determine yes or no)
Claim 30:
wherein combining the plurality of predictions comprises combining the plurality of predictions into the combined prediction using a voting scheme: a person combines information from multiple predictions to determine if a false positive occurs (e.g. use majority rule to determine yes or no)
Claim 31:
wherein combining the plurality of predictions comprises calculating, based on the plurality of predictions, one or more of a mean value, a median value, a mode value, and a weighted sum: a person uses one of the above metrics to determine a prediction (e.g. mode, select either ‘1’ or ‘0’ based on how many predictions they count for each label)
Claims 2, 7, and 21-31 do not contain any additional elements which integrate the judicial exception into a practical application (Step 2A Prong 2: NO). As discussed above, the only additional limitations are mere instructions to implement the judicial exception using a generic computer, which do not integrate the judicial exception into a practical application as they do not impose any meaningful limits on practicing the abstract idea. Accordingly, claims 2-12 are directed to an abstract idea.
Claims 2-12 do not contain any additional elements which amount to significantly more than the judicial exception (Step 2B: NO). As discussed above, the only additional limitations are mere instructions to implement the judicial exception using a generic computer, which do not amount to significantly more than the judicial exception as they do not provide an inventive concept. Therefore, claims 2-12 are not patent eligible.
Regarding claim 13, “One or more non-transitory computer-readable media” is recited, which is directed to one of the four statutory categories of invention (article of manufacture) (Step 1: YES). However, the claims limitations, under their broadest reasonable interpretation, recite mental processes or mathematical concepts which fall into the category of abstract idea (Step 2A Prong 1: YES).
The following limitations, under their broadest reasonable interpretation, recite mental processes or mathematical concepts:
input a prompt to one or more machine learning models, the prompt being generated from a positive labeled data entry, and the positive labeled data entry comprising a query, and identifier for a media content item, and a positive match value corresponding to the query and the media content item: a person obtains labeled data with a query (e.g. particular command/request/category), an identifier (e.g. a title of a particular film), and a label (e.g. 0 or 1 if item matches or does not match with query), and then writes down a prompt using this information, and inputs it to a model
generate…one or more outputs in response to the prompt: a person writes down several outputs related to a prompt/question
determine that the labeled data entry is a false-positive data entry based on the one or more outputs generated…being associated with a negative match value: a person makes a decision based on an output that a false positive occurred (e.g. decides that the labeled data is a false positive if the output says “NO”)
in response to determining that the positive labeled data entry is the false-positive data entry, generate a test and a test prompt from the positive labeled data entry, wherein the test represents a negative training sample: a person writes down a test (e.g. a further written prompt) using the labeled data if a false positive is detected
…wherein the test prompt comprises the query, and the identifier for the media content item: a person writes down a test containing the query and the identifier
generate…a test prediction in response to the test prompt: a person writes down a prediction based on the test prompt/question
compute a loss function based on the test prediction and the test: computing a loss function amounts to a mathematical calculation
update one or more parameters of the further machine learning model to minimize the loss function: updating parameters according to a minimization of a loss function amounts to mathematical calculations
apply…to a further media content item and a further query to generate a further prediction: a person uses the previous test to inform their decision about a new query/media content item to write down another prediction
Claim 13 does not contain any additional elements which integrate the judicial exception into a practical application (Step 2A Prong 2: NO). The only additional limitations are “one or more non-transitory computer-readable media having instructions stored thereon, when the instructions are executed by one or more processors, cause the one or more processors to”, “input a prompt to one or more machine learning models”, “generate, by the one or more machine learning models”, “determine…based on the one or more outputs generated by the one or more machine learning models”, “input the test prompt to a further machine learning model”, “generate, by the further machine learning model”, and “apply the further machine learning model having one or more updated parameters…”. These limitations are recited at a high level of generality and amount to mere instructions to implement the judicial exception using a generic computer. Even when viewed in combination, mere instructions to implement the judicial exception using a generic computer do not integrate the judicial exception into a practical application as they do not impose any meaningful limits on practicing the judicial exception. Accordingly, claim 13 is directed to an abstract idea.
Claim 13 does not contain any additional elements which amount to significantly more than the judicial exception (Step 2B: NO). As discussed above, the only additional limitations are mere instructions to implement the judicial exception using a generic computer, which even when viewed in combination do not amount to significantly more than the judicial exception as they do not provide an inventive concept. Therefore, claim 13 is not patent eligible.
Regarding dependent claims 14-15, “The one or more non-transitory computer-readable media” is recited, which is directed to one of the four statutory categories of invention (article of manufacture) (Step 1: YES). However, the claims limitations, under their broadest reasonable interpretation, recite further mental processes or mathematical concepts which fall into the category of abstract idea (Step 2A Prong 1: YES).
The following limitations, under their broadest reasonable interpretation, recite further mental processes or mathematical concepts:
Claim 14:
wherein the prompt comprises a question and contextual information about the media content item with which the one or more machine learning models are used to generate an answer to the question: a person writes down the prompt as a question to see if the content item matches the query (e.g. writes down “Does [content item] match [query]?”), and includes contextual information about the content item to use for generating an answer.
Claim 14 contains no additional limitations.
Claim 15:
the one or more outputs generated by the plurality of different expert models comprise a plurality of outputs; and determining that the labeled data entry is a false-positive comprises: combining the plurality of outputs into a combined output; and comparing the combined output against the positive match value: a person combines outputs from a plurality of outputs and compares the output to the label to see if a false positive occurred
Claim 15 contains the additional limitation “the one or more machine learning models comprise a plurality of different expert models”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claims 14-15 do not contain any additional elements which integrate the judicial exception into a practical application (Step 2A Prong 2: NO). As discussed above, the only additional limitations are mere instructions to implement the judicial exception using a generic computer, which do not integrate the judicial exception into a practical application as they do not impose any meaningful limits on practicing the abstract idea. Accordingly, claims 14-15 are directed to an abstract idea.
Claims 14-15 do not contain any additional elements which amount to significantly more than the judicial exception (Step 2B: NO). As discussed above, the only additional limitations are mere instructions to implement the judicial exception using a generic computer, which do not amount to significantly more than the judicial exception as they do not provide an inventive concept. Therefore, claims 14-15 are not patent eligible.
Regarding claim 16, “A computer-implemented system” is recited, which is directed to one of the four statutory categories of invention (machine) (Step 1: YES). However, the claims limitations, under their broadest reasonable interpretation, recite mental processes or mathematical concepts which fall into the category of abstract idea (Step 2A Prong 1: YES).
The following limitations, under their broadest reasonable interpretation, recite mental processes or mathematical concepts:
…generating training data for training a machine learning model: a person can write down data for training purposes
…generate a plurality of responses about a query and a media content item in a positive labeled data entry from the one or more data sources: a person can write down a response about a query and content item in labeled data entry
…evaluate the plurality of responses, and output a test in response to determining, based on the plurality of responses, that the positive labeled data entry is a false-positive data entry, wherein the test represents a negative training sample: a person evaluates the response they wrote to see if a false-positive has occurred, and in response, writes down a test
the machine learning model to receive a test prompt generated from the test and generate a test prediction in response to the test prompt: a person inputs the test prompt to the machine learning model
…compute a loss function based on the test and the test prediction: computing a loss amounts to a mathematical calculation
…update one or more parameters of the machine learning model to minimize the loss function: updating parameters of the machine learning model amounts to a mathematical concept.
…generate a further prediction using one or more updated parameters based on a further media content item and a further query: a person uses the previous test to inform their decision about a new query/media content item to write down another prediction
Claim 16 does not contain any additional elements which integrate the judicial exception into a practical application (Step 2A Prong 2: NO). The only additional limitations are “one or more data sources”,“a checker, comprising: an ensemble of expert large language models…”, “”an evaluate part…”, “a machine learning model”, “a further evaluate part”, “an update part”, and “wherein the machine learning model is further to generate”. These limitations are recited at a high level of generality and amount to mere instructions to implement the judicial exception using a generic computer. Even when viewed in combination, mere instructions to implement the judicial exception using a generic computer do not integrate the judicial exception into a practical application as they do not impose any meaningful limits on practicing the judicial exception. Accordingly, claim 16 is directed to an abstract idea.
Claim 16 does not contain any additional elements which amount to significantly more than the judicial exception (Step 2B: NO). As discussed above, the only additional limitations are mere instructions to implement the judicial exception using a generic computer, which even when viewed in combination do not amount to significantly more than the judicial exception as they do not provide an inventive concept. Therefore, claim 16 is not patent eligible.
Regarding dependent claims 18-19, “The computer-implemented system” is recited, which is directed to one of the four statutory categories of invention (machine) (Step 1: YES). However, the claims limitations, under their broadest reasonable interpretation, recite further mental processes or mathematical concepts which fall into the category of abstract idea (Step 2A Prong 1: YES).
The following limitations, under their broadest reasonable interpretation, recite further mental processes or mathematical concepts:
Claim 18:
…correct a label in the positive labeled data entry based on the plurality of responses: a person writes down a corrected entry by correcting the label by using the responses (e.g. changing ‘1’ to ‘0’ if responses say it is false positive)
Claim 18 contains the additional limitation “wherein the checker further includes a modify part…”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claim 19:
…generate a prompt based on the query, the content item, and metadata about the media content item…. receives the prompt and generates the response based on the prompt: a person reads a prompt/question and writes down a response
Claim 19 contains the additional limitation “the checker further includes a translator part…the ensemble of the expert language models receives…and generates the plurality of responses…”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claims 18-19 do not contain any additional elements which integrate the judicial exception into a practical application (Step 2A Prong 2: NO). As discussed above, the only additional limitations are mere instructions to implement the judicial exception using a generic computer, which do not integrate the judicial exception into a practical application as they do not impose any meaningful limits on practicing the abstract idea. Accordingly, claims 18-19 are directed to an abstract idea.
Claims 18-19 do not contain any additional elements which amount to significantly more than the judicial exception (Step 2B: NO). As discussed above, the only additional limitations are mere instructions to implement the judicial exception using a generic computer, which do not amount to significantly more than the judicial exception as they do not provide an inventive concept. Therefore, claims 18-19 are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
5. Claims 1, 2, and 7, 25-26, and 29-31 are rejected under 35 U.S.C. 103 as being unpatentable over Nagaraju & Parisien (US 2024/0185001 A1, hereinafter Nagaraju) in view of Chong et al. (NPL Detecting Label Errors using Pre-Trained Language Models, hereinafter Chong) and further in view of Narayan et al. (NPL Can Foundation Models Wrangle Your Data?, hereinafter Narayan).
Regarding claim 1, Nagaraju discloses A method comprising: receiving a… labeled data entry comprising a query (para. 0037 “FIG. 4A illustrates an example template query 114, according to at least one embodiment. Template query 114 may include a static portion 410 and a dynamic portion 420.”), an identifier for a media content item (para. 0036 “Task data 112 may include one or more task-specific or domain-specific tokens. Tokens may be any words, phrases, abbreviations, sentences, exclamations, and/or the like, that are encountered in conversations, advertisements, and/or printing materials pertaining to relevant tasks/domains.”), and a …value corresponding to the query and the content item (para. 0037 “Individual inputs and outputs of a pair may include a label (e.g., input label, output label) and a value (e.g., input value, output value). In some embodiments, some input/output labels may be the same for multiple example input/output pairings, while the input/output values for those pairings may be different. For example, template query 114 for training a conversational navigation model may include multiple input/output labels that are the same, e.g., “Input label: Destinations” and “Output label: Directions requests,” while the input/output values are different.”); translating the …labeled data entry into a prompt… (para. 0035 “As illustrated in FIG. 3, query prompt generator 132 may receive task data 112 and template query 114 as input into query generator system 120. Query prompt generator 132 may combine task data 112 and template query 114 to create prompt 310.”); generating, by one or more machine learning models, one or more predictions in response to receiving the prompt (para. 0035 “Prompt 310 may then be provided to trained large language model 134 to generate an output value (e.g., NL query 320).”); …generating a test from the …labeled data entry (para. 0035 “In some embodiments, query prompt generator 132 may also generate target 330. NL query 320 and/or target 330 may then be stored in training data repository 170. Training engine 194 may then use generated NL query 172 (e.g., NL query 320) and target result 174 (e.g., target 330) to train parameters of conversational MLM 192 as described herein.”; test comprises a set of training data (NL query and target result pair)); generating, by a further machine learning model, a test prediction using the test (further machine learning model: Fig. 2, 192 “Conversational MLM”; para. 0030 “Generated NL query 172 may be used by training engine 194 to identify parameters (e.g., neural weights, biases, parameters of activation functions, etc.) of conversational MLM 192 that maximize success of task-oriented dialogue system 180. In some embodiments, training of conversational MLM 192 may be supervised, e.g., using human-annotations of generated NL query 172 as ground truth or using target result 174 as ground truth”; para. 0031 “For every training input (e.g., generated NL query 172), training engine 194 may cause conversational MLM 192 to generate training output(s).); computing a loss function based on the test and test prediction (para. 0031 Training engine 194 may then compare observed output(s) with the desired target output(s). The desired target output(s) may include target result 174 and/or human-annotated ground truths corresponding to the input (e.g., generated NL query 172). The resulting error or mismatch, e.g., the difference between the desired target output(s) and the actual output(s) of conversational MLM 192, may be back-propagated through conversational MLM 192); updating one or more parameters of the further machine learning model to minimize the loss function (para. 0031 “For every training input (e.g., generated NL query 172), training engine 194 may cause conversational MLM 192 to generate training output(s). Training engine 194 may then compare observed output(s) with the desired target output(s). The desired target output(s) may include target result 174 and/or human-annotated ground truths corresponding to the input (e.g., generated NL query 172). The resulting error or mismatch, e.g., the difference between the desired target output(s) and the actual output(s) of conversational MLM 192, may be back-propagated through conversational MLM 192, and the weights and biases in the conversational MLM 192 may be adjusted to make the actual or predicted output(s) closer to the target (ground truth) output(s).”); and applying the further machine learning model having one or more updated parameters to a further media content item and a further query to generate a further prediction (para. 0031 “This adjustment may be repeated until the output error for a given training input satisfies a predetermined condition (e.g., falls below a predetermined value). Subsequently, a different training input may be selected, a new output generated, and a new series of adjustments implemented, until conversational MLM 192 is trained to (e.g., converges to) a target degree of accuracy. Although training of conversational MLM 192 is described in the aforementioned example, similar operations may be performed in training of other MLMs (e.g., large language model 134).”).
Nagaraju does not specifically disclose [receiving a ] positive [labeled data entry comprising…a] positive [match value…], [translating the] positive [labeled data entry into a prompt]…; determining, by a checker, that the positive labeled data entry is a false-positive data entry, based on the one or more predictions being associated with a negative match value; in response to determining that the positive labeled data entry is the false-positive data entry, generating a test from the positive labeled data entry, wherein the test represents a negative training sample.
Chong teaches [receiving a ] positive [labeled data entry comprising…a] positive [match value…] (e.g. Table 1, 1st example label error, a false positive (label is ‘positive’, ‘sentiment’ is negative)), [translating the] positive [labeled data entry into a prompt] (Nagaraju teaches translation into a prompt of labeled data (see above mapping); Chong teaches labeled data entries containing a positive label: (e.g. Table 1, 1st example label error, a false positive (label is ‘positive’, ‘sentiment’ is negative))…; determining, by a checker, that the positive labeled data entry is a false-positive data entry, based on the one or more predictions being associated with a negative match value (pg. 4, 1st para. “Foundation Model Ensembling (FME) combines multiple foundation models on the same task…Rather than using a validation set to choose the signal model with the lowest loss on the task, FME uses the top three models trained in a hyperparameter sweep, and differing in both hyperparameters and random initialization, as fully described in Appendix D. FME creates a synthetic probability distribution over the task outputs by averaging the probabilities predicted using each individual model. FME then hypothesizes items in order of loss over the synthetic distribution… ”; checker (FM ensemble) is used to determine label errors such as in entry 1 in Table 1 for IMDB dataset (see also Table 3, showing experimental results for IMDB dataset using FME for label error detection)); in response to determining that the positive labeled data entry is the false-positive data entry, generating a test from the positive labeled data entry, wherein the test represents a negative training sample (Chong further teaches cleaning/correcting data entries which have errors in labels; this would include changing a false ‘positive’ label to a correct ‘negative’ label; these corrected labels are used as tests (validation/test data) for training a model…: pg. 7, section “End-to-end noising” “For each dataset, we prepare three versions of the validation and test splits, respectively: a clean version assumed to contain zero errors, a noisy version, with label noise deliberately introduced, and a corrected version generated from noisy splits using our main error detection method (ranking errors with FME and correcting the top Err% data points). We train 40 hyperparameter sweeps, with performance cross-evaluated on all prepared data splits…”).
Nagaraju and Chong are considered to be analogous to the claimed invention as
they both are in the same field of natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Nagaraju to incorporate the teachings of Chong in order to specifically receive a positive labeled data entry with a positive match value, translate the positive labeled data entry into a prompt, to determine by a checker that the positive labeled data entry is a false-positive data entry, based on the one or more predictions being associated with a negative match value, and in response to determining that the positive labeled data entry is the false-positive data entry, generating a test from the positive labeled data entry, wherein the test represents a negative training sample. Doing so would be beneficial, as the above process would result in identifying and cleaning validation data used in training, which would improve test performance for a model trained with this cleaned data (Chong, pg. 10 4th para.).
Nagaraju in view of Chong does not specifically disclose wherein the prompt includes a question or request to output a match value.
Narayan wherein the prompt includes a question or request to output a match value (pg. 2, Fig. 2, foundation model prompted to determine if there is a match (correct label) in the data (e.g. for ‘Zero-shot’, model prompted question “Is there an error in Country” for “Country: England, City: Kyoto?”)).
Nagaraju, Chong, and Narayan are considered to be analogous to the claimed invention as they are all in the same field of natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Nagaraju in view of Chong to incorporate the teachings of Narayan in order to specifically have the prompt be a question or request to output a match value. Doing so would be beneficial, as prompting foundation models in this way achieves state of the art performance on data cleaning tasks (Narayan, Abstract).
Regarding claim 2, Nagaraju in view of Chong and Narayan discloses wherein the prompt further includes a title of the media content item, the query, and metadata associated with the media content item (Nagaraju, Fig. 4B; para. 0048 “FIG. 4B illustrates an example data flow combining task data 112 and a template query 114 to generate a prompt 310…”; para. 0052 “In some embodiments, conditions may be added to input/output pairings of template query 114 or may be added to prompt 310 before prompt 310 is provided to trained large language model 134.”).
Regarding claim 7, Nagaraju in view of Chong and Narayan discloses wherein the test comprises the query and the media content item (Nagaraju, test data (172 and 174) comprises query (172) and content item (contained in 172. See Fig. 4B, 320)).
Regarding claim 25, Nagaraju in view of Chong and Narayan discloses in response to determining that the positive labeled data entry is the false-positive data entry, modifying the positive match value based on the one or more predictions (Chong, datasets (including IMDB dataset in Table 1, demonstrating false positives) are corrected utilizing predictions from foundation model ensembling: pg. 7 section “End-to-end noising” “For each dataset, we prepare three version of the validation and test splits, respectively: a clean version assumed to contain zero errors, a noisy version, with label noise deliberately introduced, and a corrected version generated from noisy splits using our main error detection method (ranking errors with FME and correcting the top Err% data points)…”).
Regarding claim 26, Nagaraju in view of Chong and Narayan discloses wherein generating the one or more predictions comprises generating a plurality of predictions by an ensemble of expert large language models in response to receiving the prompt (Chong, pg. 4, 1st para. “Foundation Model Ensembling (FME) combines multiple foundation models on the same task…Rather than using a validation set to choose the signal model with the lowest loss on the task, FME uses the top three models trained in a hyperparameter sweep, and differing in both hyperparameters and random initialization, as fully described in Appendix D. FME creates a synthetic probability distribution over the task outputs by averaging the probabilities predicted using each individual model. FME then hypothesizes items in order of loss over the synthetic distribution… ”; the foundation models are large language models (pg. 3, section “Foundation models” “…directly applying the foundation model paradigm: we use a large language model that was first pre-trained on a task-agnostic dataset, then fine-tune the model for a given task…”); large language models generate outputs based on prompts, thus the predictions are generated in response to receiving the prompt).
Nagaraju, Chong, and Narayan are considered to be analogous to the claimed invention as they are all in the same field of natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Chong in order to specifically generate a plurality of predictions by an ensemble of expert large language models in response to receiving the prompt. Doing so would be beneficial, as ensembling confers significantly more gains in error detection performance (Chong, pg. 8, section “Overall LLM performance”).
Regarding claim 29, Nagaraju in view of Chong and Narayan discloses wherein determining the positive labeled data entry is the false-positive data entry comprises: combining the plurality of predictions into a combined prediction (Chong, pg. 4, 1st para. “FME creates a synthetic probability distribution over the task outputs by averaging the probabilities predicted using each individual model. FME then hypothesizes items in order of loss over the synthetic distribution…”); and evaluating whether the combined prediction is consistent with the positive match value of the positive labeled data entry (Nagaraju teaches comparing predicted output with label: para. 0031 “For every training input (e.g., generated NL query 172), training engine 194 may cause conversational MLM 192 to generate training output(s). Training engine 194 may then compare observed output(s) with the desired target output(s). The desired target output(s) may include target result 174 and/or human-annotated ground truths corresponding to the input (e.g., generated NL query 172).”).
Nagaraju, Chong, and Narayan are considered to be analogous to the claimed invention as they are all in the same field of natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Chong in order to specifically combine the plurality of predictions into a combined prediction to evaluate whether a false positive has occurred. Doing so would be beneficial, as ensembling confers significantly more gains in error detection performance (Chong, pg. 8, section “Overall LLM performance”).
Regarding claim 30, Nagaraju in view of Chong and Narayan discloses wherein combining the plurality of predictions comprises combining the plurality of predictions into the combined prediction using a voting scheme (Chong, pg. 4, 1st para. “FME creates a synthetic probability distribution over the task outputs by averaging the probabilities predicted using each individual model. FME then hypothesizes items in order of loss over the synthetic distribution…”; taught ensembling reads on broadly recited ‘voting scheme’). Nagaraju, Chong, and Narayan are considered to be analogous to the claimed invention as they are all in the same field of natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Chong in order to specifically combine the plurality of predictions into the combined prediction using a voting scheme. Doing so would be beneficial, as ensembling confers significantly more gains in error detection performance (Chong, pg. 8, section “Overall LLM performance”).
Regarding claim 31, Nagaraju in view of Chong and Narayan discloses wherein combining the plurality of predictions comprises calculating, based on the plurality of predictions, one or more of a mean value, a median value, a mode value, and a weighted sum (Chong, pg. 4, 1st para. “FME creates a synthetic probability distribution over the task outputs by averaging the probabilities predicted using each individual model. FME then hypothesizes items in order of loss over the synthetic distribution…”).
Nagaraju, Chong, and Narayan are considered to be analogous to the claimed invention as they are all in the same field of natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Chong in order to specifically combine the plurality of predictions into the combined prediction by calculating a mean value. Doing so would be beneficial, as ensembling confers significantly more gains in error detection performance (Chong, pg. 8, section “Overall LLM performance”).
6. Claims 13, 15-16, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Nagaraju in view of Chong.
Regarding claim 13, Nagaraju discloses One or more non-transitory computer-readable media having instructions stored thereon, when the instructions are executed by one or more processors, cause the one or more processors to (para. 0053 “In at least one embodiment, processing units performing any of methods 500 and/or 600 may be executing instructions stored on a non-transient computer-readable storage media.”; para. 0118): input a prompt to one or more machine learning models (para. 0035 “Prompt 310 may then be provided to trained large language model 134 to generate an output value (e.g., NL query 320).”), the prompt being generated from a …labeled data entry (para. 0035 “As illustrated in FIG. 3, query prompt generator 132 may receive task data 112 and template query 114 as input into query generator system 120. Query prompt generator 132 may combine task data 112 and template query 114 to create prompt 310.”), and the …labeled data entry comprising a query (para. 0037 “FIG. 4A illustrates an example template query 114, according to at least one embodiment. Template query 114 may include a static portion 410 and a dynamic portion 420.”), an identifier for a media content item (para. 0036 “Task data 112 may include one or more task-specific or domain-specific tokens. Tokens may be any words, phrases, abbreviations, sentences, exclamations, and/or the like, that are encountered in conversations, advertisements, and/or printing materials pertaining to relevant tasks/domains.”), and a …match value corresponding to the query and the media content item (para. 0037 “Individual inputs and outputs of a pair may include a label (e.g., input label, output label) and a value (e.g., input value, output value). In some embodiments, some input/output labels may be the same for multiple example input/output pairings, while the input/output values for those pairings may be different. For example, template query 114 for training a conversational navigation model may include multiple input/output labels that are the same, e.g., “Input label: Destinations” and “Output label: Directions requests,” while the input/output values are different.”); generate, by the one or more machine learning models, one or more outputs in response to the prompt (para. 0035 “Prompt 310 may then be provided to trained large language model 134 to generate an output value (e.g., NL query 320).”);…generate a test and a test prompt (the test is a particular piece of training data, which is associated with a test prompt: para. 0035 “As illustrated in FIG. 3, query prompt generator 132 may receive task data 112 and template query 114 as input into query generator system 120. Query prompt generator 132 may combine task data 112 and template query 114 to create prompt 310. Prompt 310 may then be provided to trained large language model 134 to generate an output value (e.g., NL query 320). In some embodiments, query prompt generator 132 may also generate target 330. NL query 320 and/or target 330 may then be stored in training data repository 170. Training engine 194 may then use generated NL query 172 (e.g., NL query 320) and target result 174 (e.g., target 330) to train parameters of conversational MLM 192 as described herein. Output generated using conversational MLM 192 (e.g., one or more question-answer pairs) may be stored in a dialogue data store 340.”)…and input the test prompt to a further machine learning model, wherein the test prompt comprises the query, and the identifier for the media content item (Fig. 2, 192 “Conversational MLM”; para. 0030 “Generated NL query 172 may be used by training engine 194 to identify parameters (e.g., neural weights, biases, parameters of activation functions, etc.) of conversational MLM 192 that maximize success of task-oriented dialogue system 180. In some embodiments, training of conversational MLM 192 may be supervised, e.g., using human-annotations of generated NL query 172 as ground truth or using target result 174 as ground truth”; para. 0035 “In some embodiments, query prompt generator 132 may also generate target 330. NL query 320 and/or target 330 may then be stored in training data repository 170. Training engine 194 may then use generated NL query 172 (e.g., NL query 320) and target result 174 (e.g., target 330) to train parameters of conversational MLM 192 as described herein.”; tests (training data) includes query (114) and identified (112): para. 0035 “As illustrated in FIG. 3, query prompt generator 132 may receive task data 112 and template query 114 as input into query generator system 120. Query prompt generator 132 may combine task data 112 and template query 114 to create prompt 310. Prompt 310 may then be provided to trained large language model 134 to generate an output value (e.g., NL query 320). In some embodiments, query prompt generator 132 may also generate target 330. NL query 320 and/or target 330 may then be stored in training data repository 170. Training engine 194 may then use generated NL query 172 (e.g., NL query 320) and target result 174 (e.g., target 330) to train parameters of conversational MLM 192 as described herein. Output generated using conversational MLM 192 (e.g., one or more question-answer pairs) may be stored in a dialogue data store 340.”); generate, by the further machine learning model, a test prediction in response to the test prompt (para. 0031 “For every training input (e.g., generated NL query 172), training engine 194 may cause conversational MLM 192 to generate training output(s).”); compute a loss function based on the test prediction and the test (para. 0031 Training engine 194 may then compare observed output(s) with the desired target output(s). The desired target output(s) may include target result 174 and/or human-annotated ground truths corresponding to the input (e.g., generated NL query 172). The resulting error or mismatch, e.g., the difference between the desired target output(s) and the actual output(s) of conversational MLM 192, may be back-propagated through conversational MLM 192); update one or more parameters of the further machine learning model to minimize the loss function (para. 0031 “For every training input (e.g., generated NL query 172), training engine 194 may cause conversational MLM 192 to generate training output(s). Training engine 194 may then compare observed output(s) with the desired target output(s). The desired target output(s) may include target result 174 and/or human-annotated ground truths corresponding to the input (e.g., generated NL query 172). The resulting error or mismatch, e.g., the difference between the desired target output(s) and the actual output(s) of conversational MLM 192, may be back-propagated through conversational MLM 192, and the weights and biases in the conversational MLM 192 may be adjusted to make the actual or predicted output(s) closer to the target (ground truth) output(s).”); and apply the further machine learning model having one or more updated parameters to a further media content item and a further query to generate a further prediction (para. 0031 “This adjustment may be repeated until the output error for a given training input satisfies a predetermined condition (e.g., falls below a predetermined value). Subsequently, a different training input may be selected, a new output generated, and a new series of adjustments implemented, until conversational MLM 192 is trained to (e.g., converges to) a target degree of accuracy. Although training of conversational MLM 192 is described in the aforementioned example, similar operations may be performed in training of other MLMs (e.g., large language model 134).”).
Nagaraju does not specifically disclose the prompt [being generated from a ] positive [labeled data entry, and the] positive [labeled data entry comprising…a] positive [match value…]; determine that the positive labeled data entry is a false-positive data entry based on the one or more outputs generated by the one or more machine learning models being associated with a negative match value; in response to determining that the positive labeled data entry is the false-positive data entry, [generate a test and a test prompt from the positive labeled data entry], wherein the test represents a negative training sample.
Chong teaches the prompt [being generated from a ] positive [labeled data entry, and the] positive [labeled data entry comprising…a] positive [match value…] (e.g. Table 1, 1st example label error, a false positive (label is ‘positive’, ‘sentiment’ is negative); Nagaraju teaches translation into a prompt of labeled data (see above mapping); Chong teaches labeled data entries containing a positive label: (e.g. Table 1, 1st example label error, a false positive (label is ‘positive’, ‘sentiment’ is negative)); determine that the positive labeled data entry is a false-positive data entry based on the one or more outputs generated by the one or more machine learning models being associated with a negative match value (pg. 4, 1st para. “Foundation Model Ensembling (FME) combines multiple foundation models on the same task…Rather than using a validation set to choose the signal model with the lowest loss on the task, FME uses the top three models trained in a hyperparameter sweep, and differing in both hyperparameters and random initialization, as fully described in Appendix D. FME creates a synthetic probability distribution over the task outputs by averaging the probabilities predicted using each individual model. FME then hypothesizes items in order of loss over the synthetic distribution… ”; checker (FM ensemble) is used to determine label errors such as in entry 1 in Table 1 for IMDB dataset (see also Table 3, showing experimental results for IMDB dataset using FME for label error detection)); in response to determining that the positive labeled data entry is the false-positive data entry, [generate a test and a test prompt from the positive labeled data entry], wherein the test represents a negative training sample (Chong further teaches cleaning/correcting data entries which have errors in labels; this would include changing a false ‘positive’ label to a correct ‘negative’ label; these corrected labels are used as tests (validation/test data) for training a model…: pg. 7, section “End-to-end noising” “For each dataset, we prepare three versions of the validation and test splits, respectively: a clean version assumed to contain zero errors, a noisy version, with label noise deliberately introduced, and a corrected version generated from noisy splits using our main error detection method (ranking errors with FME and correcting the top Err% data points). We train 40 hyperparameter sweeps, with performance cross-evaluated on all prepared data splits…”).
Nagaraju and Chong are considered to be analogous to the claimed invention as
they both are in the same field of natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Nagaraju to incorporate the teachings of Chong in order to specifically receive a positive labeled data entry with a positive match value, translate the positive labeled data entry into a prompt, to determine that the positive labeled data entry is a false-positive data entry, based on the one or more outputs generated by the one or more machine learning models being associated with a negative match value, and in response to determining that the positive labeled data entry is the false-positive data entry, generating a test from the positive labeled data entry, wherein the test represents a negative training sample. Doing so would be beneficial, as the above process would result in identifying and cleaning validation data used in training, which would improve test performance for a model trained with this cleaned data (Chong, pg. 10 4th para.).
Regarding claim 15, Nagaraju in view of Chong discloses the one or more machine learning models comprise a plurality of different expert models (Chong, pg. 4 1st para. “Foundation Model Ensembling (FME) combines multiple foundation models on the same task. We hypothesize that ensembling may be disproportionately effective at detecting label errors, as training noise induces models to learn random spurious correlations (Watson et al., 2022). Rather than using a validation set to choose the single model with the lowest loss on the task, FME uses the top three models trained in a hyperparameter sweep, and differing in both hyperparameters and random initialization, as fully described in Appendix D. FME creates a synthetic probability distribution over the task outputs by averaging the probabilities predicted using each individual model. FME then hypothesizes items in order of loss over the synthetic distribution.”); the one or more outputs generated by the plurality of different expert models comprise a plurality of outputs (Chong, pg. 4 1st para. “Foundation Model Ensembling (FME) combines multiple foundation models on the same task. We hypothesize that ensembling may be disproportionately effective at detecting label errors, as training noise induces models to learn random spurious correlations (Watson et al., 2022). Rather than using a validation set to choose the single model with the lowest loss on the task, FME uses the top three models trained in a hyperparameter sweep, and differing in both hyperparameters and random initialization, as fully described in Appendix D. FME creates a synthetic probability distribution over the task outputs by averaging the probabilities predicted using each individual model. FME then hypothesizes items in order of loss over the synthetic distribution.”); and determining that the positive labeled data entry is the false-positive comprises: combining the plurality of outputs into a combined output (Chong, pg. 4 1st para. “Foundation Model Ensembling (FME) combines multiple foundation models on the same task. We hypothesize that ensembling may be disproportionately effective at detecting label errors, as training noise induces models to learn random spurious correlations (Watson et al., 2022). Rather than using a validation set to choose the single model with the lowest loss on the task, FME uses the top three models trained in a hyperparameter sweep, and differing in both hyperparameters and random initialization, as fully described in Appendix D. FME creates a synthetic probability distribution over the task outputs by averaging the probabilities predicted using each individual model. FME then hypothesizes items in order of loss over the synthetic distribution.”); comparing the combined output against the positive match value (Nagaraju teaches comparing predicted output with label: para. 0031 “For every training input (e.g., generated NL query 172), training engine 194 may cause conversational MLM 192 to generate training output(s). Training engine 194 may then compare observed output(s) with the desired target output(s). The desired target output(s) may include target result 174 and/or human-annotated ground truths corresponding to the input (e.g., generated NL query 172).”).
Nagaraju and Chong are considered to be analogous to the claimed invention as
they both are in the same field of natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Nagaraju to incorporate the teachings of Chong in order to specifically have the one or more machine learning models comprise a plurality of different expert models which generate a plurality of outputs, and determine that the labeled data entry is a false-positive by combining the plurality of outputs into a combined output and comparing the combined output against the label. Doing so would be beneficial, as ensembling multiple foundation models confers significantly more gains for error detection tasks (Chong, pg. 8 1st para.).
Regarding claim 16, Nagaraju discloses A computer-implemented system (Fig. 1) comprising: one or more data sources for generating training data for training a machine learning model (Fig. 1, 110); a checker, comprising: …large language models to generate a plurality of responses about a query and a media content item in a… labeled data entry from the one or more data sources (para. 0035 “As illustrated in FIG. 3, query prompt generator 132 may receive task data 112 and template query 114 as input into query generator system 120. Query prompt generator 132 may combine task data 112 and template query 114 to create prompt 310.”); para. 0035 “Prompt 310 may then be provided to trained large language model 134 to generate an output value (e.g., NL query 320).”); and an evaluate part to evaluate the plurality of responses (para. 0049 “The generated output value (e.g., NL query 320) may then be stored in training data repository 170.”; para. 0050 “The new prompt 310 may then be provided to trained large language model 134 to generate a new output value (e.g., NL query 320), which may subsequently be stored in training data repository 170 (e.g., as generated NL query 172). This process may continue until a target condition is satisfied, e.g., a certain number of NL queries have been generated, a certain number of task data and/or template queries have been selected, a certain amount of time has elapsed since starting the dataset generation process, or the like”), and output a test…( para. 0035 “In some embodiments, query prompt generator 132 may also generate target 330. NL query 320 and/or target 330 may then be stored in training data repository 170. Training engine 194 may then use generated NL query 172 (e.g., NL query 320) and target result 174 (e.g., target 330) to train parameters of conversational MLM 192 as described herein.”; test comprises a set of training data (NL query and target result pair)); the machine learning model to receive a test prompt generated from the test and generate a test prediction in response to the test prompt (Fig. 2, 192 “Conversational MLM”; para. 0030 “Generated NL query 172 may be used by training engine 194 to identify parameters (e.g., neural weights, biases, parameters of activation functions, etc.) of conversational MLM 192 that maximize success of task-oriented dialogue system 180. In some embodiments, training of conversational MLM 192 may be supervised, e.g., using human-annotations of generated NL query 172 as ground truth or using target result 174 as ground truth”); a further evaluate part to compute a loss function based on the test and the test prediction (para. 0031 Training engine 194 may then compare observed output(s) with the desired target output(s). The desired target output(s) may include target result 174 and/or human-annotated ground truths corresponding to the input (e.g., generated NL query 172). The resulting error or mismatch, e.g., the difference between the desired target output(s) and the actual output(s) of conversational MLM 192, may be back-propagated through conversational MLM 192); an update part to update one or more parameters of the machine learning model to minimize the loss function, (para. 0031 “For every training input (e.g., generated NL query 172), training engine 194 may cause conversational MLM 192 to generate training output(s). Training engine 194 may then compare observed output(s) with the desired target output(s). The desired target output(s) may include target result 174 and/or human-annotated ground truths corresponding to the input (e.g., generated NL query 172). The resulting error or mismatch, e.g., the difference between the desired target output(s) and the actual output(s) of conversational MLM 192, may be back-propagated through conversational MLM 192, and the weights and biases in the conversational MLM 192 may be adjusted to make the actual or predicted output(s) closer to the target (ground truth) output(s).”) wherein the machine learning model is further to generate a further prediction using one or more updated parameters based on a further media content item and a further query (para. 0031 “This adjustment may be repeated until the output error for a given training input satisfies a predetermined condition (e.g., falls below a predetermined value). Subsequently, a different training input may be selected, a new output generated, and a new series of adjustments implemented, until conversational MLM 192 is trained to (e.g., converges to) a target degree of accuracy. Although training of conversational MLM 192 is described in the aforementioned example, similar operations may be performed in training of other MLMs (e.g., large language model 134).”).
Nagaraju does not specifically disclose: [comprising:] an ensemble of expert large language models [to generate…about a query and a media content item in a] positive [labeled data entry]…[output a test] in response to determining, based on the plurality of responses, that the positive labeled data entry is a false-positive data entry, wherein the test represents a negative training sample.
Chong teaches [comprising:] an ensemble of expert large language models (ensemble of best three foundation models: pg. 4, 1st para. “Foundation Model Ensembling (FME) combines multiple foundation models on the same task…Rather than using a validation set to choose the signal model with the lowest loss on the task, FME uses the top three models trained in a hyperparameter sweep, and differing in both hyperparameters and random initialization, as fully described in Appendix D. FME creates a synthetic probability distribution over the task outputs by averaging the probabilities predicted using each individual model. FME then hypothesizes items in order of loss over the synthetic distribution… ”; FMs are large language models: pg. 3, section “Foundation Models” “…directly applying the foundation model paradigm: we use a large language model that was first pre-trained on a task-agnostic dataset, then fine-tune the model for a given task…”; checker (FM ensemble) is used to determine label errors such as in entry 1 in Table 1 for IMDB dataset (see also Table 3, showing experimental results for IMDB dataset using FME for label error detection);) [to generate…about a query and a media content item in a] positive [labeled data entry] (e.g. Table 1, 1st example label error, a false positive (label is ‘positive’, ‘sentiment’ is negative); Nagaraju teaches translation into a prompt of labeled data (see above mapping); Chong teaches labeled data entries containing a positive label: (e.g. Table 1, 1st example label error, a false positive (label is ‘positive’, ‘sentiment’ is negative)); [output a test] in response to determining, based on the plurality of responses, that the positive labeled data entry is a false-positive data entry, wherein the test represents a negative training sample. (pg. 4, 1st para. “Foundation Model Ensembling (FME) combines multiple foundation models on the same task…Rather than using a validation set to choose the signal model with the lowest loss on the task, FME uses the top three models trained in a hyperparameter sweep, and differing in both hyperparameters and random initialization, as fully described in Appendix D. FME creates a synthetic probability distribution over the task outputs by averaging the probabilities predicted using each individual model. FME then hypothesizes items in order of loss over the synthetic distribution… ”; checker (FM ensemble) is used to determine label errors such as in entry 1 in Table 1 for IMDB dataset (see also Table 3, showing experimental results for IMDB dataset using FME for label error detection); (Chong further teaches cleaning/correcting data entries which have errors in labels; this would include changing a false ‘positive’ label to a correct ‘negative’ label; these corrected labels are used as tests (validation/test data) for training a model…: pg. 7, section “End-to-end noising” “For each dataset, we prepare three versions of the validation and test splits, respectively: a clean version assumed to contain zero errors, a noisy version, with label noise deliberately introduced, and a corrected version generated from noisy splits using our main error detection method (ranking errors with FME and correcting the top Err% data points). We train 40 hyperparameter sweeps, with performance cross-evaluated on all prepared data splits…”)
Nagaraju and Chong are considered to be analogous to the claimed invention as
they both are in the same field of natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Nagaraju to incorporate the teachings of Chong in order to use an ensemble of expert large language models to generate responses about positive labeled data entries, to determine that the positive labeled data entry is a false-positive data entry and in response to determining that the positive labeled data entry is the false-positive data entry, generating a test from the positive labeled data entry, wherein the test represents a negative training sample. Doing so would be beneficial, as the above process would result in identifying and cleaning validation data used in training, which would improve test performance for a model trained with this cleaned data (Chong, pg. 10 4th para.).
Regarding claim 19, Nagaraju in view of Chong discloses the checker further includes a translator part to generate a prompt based on the query, the media content item, and metadata about the content item (Nagaraju, para. 0035 “As illustrated in FIG. 3, query prompt generator 132 may receive task data 112 and template query 114 as input into query generator system 120. Query prompt generator 132 may combine task data 112 and template query 114 to create prompt 310.”; para. 0052 “In some embodiments, conditions may be added to input/output pairings of template query 114 or may be added to prompt 310 before prompt 310 is provided to trained large language model 134.”); and the ensemble of the expert large language models receives the prompt and generates the plurality of responses based on the prompt (Nagaraju, para. 0035 “Prompt 310 may then be provided to trained large language model 134 to generate an output value (e.g., NL query 320).”; Chong teaches the ensemble of expert large language models (see above claim mapping)).
7. Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Nagaraju in view of Chong and in further view of Narayan.
Regarding claim 14, Nagaraju in view of Chong does not specifically disclose wherein the prompt comprises a question and contextual information about the media content item with which the one or more machine learning models are used to generate an answer to the question.
Narayan teaches wherein the prompt comprises a question and contextual information about the media content item with which the one or more machine learning models are used to generate an answer to the question (Fig. 2 shows prompting of question to see if content item (in this case ‘Kyoto’) matches the query (in this case ‘England’), and additionally includes contextual information (few-shot, task demonstrations); caption “For zero-shot (left), the prompt is the task description and the example to complete. For few-shot (right), the prompt adds demonstrations of how to compete the task.”).
Nagaraju, Narayan, and Chong are considered to be analogous to the claimed invention as they are all in the same field of natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Narayan in order to have the prompt include a question whether the content item matches the query. Doing so would be beneficial, as prompting foundation models in this way achieves state of the art performance on data cleaning tasks (Narayan, Abstract).
8. Claim 18 are rejected under 35 U.S.C. 103 as being unpatentable over Nagaraju in view of Chong and in further view of Kobayashi & Matsuzawa (US 2020/0034751 A1, hereinafter Kobayashi).
Regarding claim 18, Nagaraju in view of Chong discloses to correct a label in the positive labeled data entry based on the plurality of responses (Chong further teaches cleaning/correcting data entries which have errors in labels based on foundation model ensembling responses; this would include changing a false ‘positive’ label to a correct ‘negative’ label; these corrected labels are used as tests (validation/test data) for training a model…: pg. 7, section “End-to-end noising” “For each dataset, we prepare three versions of the validation and test splits, respectively: a clean version assumed to contain zero errors, a noisy version, with label noise deliberately introduced, and a corrected version generated from noisy splits using our main error detection method (ranking errors with FME and correcting the top Err% data points). We train 40 hyperparameter sweeps, with performance cross-evaluated on all prepared data splits…”). Nagaraju in view of Chong does not specifically disclose wherein the checker further includes a modify part [to correct a label…]
Kobayashi teaches wherein the checker further includes a modify part [to correct a label …] (para. 0023 “To realize this, a possible bad training data which tends to cause misclassification will be treated instead of being deleted. For example, the bad training data will be modified by adding new words and rewriting the statement of question, by changing the intention (class) of question”; para. 0053 “At step 650, the computer system or server modifies the training data with the class value X, based on the evaluation and the validation at step 640. Modifying the training data includes changing the class value. In response to determine that the precision ratio is lower than a predetermined threshold and the representative class is a single class, the computer system or server changes the current class to the representative class.”).
Nagaraju, Kobayashi, and Chong are considered to be analogous to the claimed invention as they are all in the same field of natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Kobayashi in order to specifically have the checker include a modify part for correcting the label. Doing so would be beneficial, as this would improve the training data and lead to increased accuracy and precision (Kobayashi, para. 0020).
9. Claims 21-24 are rejected under 35 U.S.C. 103 as being unpatentable over Nagaraju in view of Chong and Narayan, and further in view of Wei et al. (NPL Few-Shot Text Classification with Triplet Networks, Data Augmentation, and Curriculum Learning, hereinafter Wei).
Regarding claim 21, Nagaraju in view of Chong and Narayan discloses the positive labeled data entry (see above claim mapping in claim 1), but does not specifically discloses computing the loss function comprises computing a triplet loss function using the positive labeled data entry as a negative ground truth label; and updating the one or more parameters of the further machine learning model comprises maximizing a distance of predictions from the negative ground truth label.
Wei teaches computing the loss function comprises computing a triplet loss function using …a negative ground truth label (pg. 3, section 3.2 “…During training, given a triplet of (anchor a, positive example p, and negative example n), a triplet loss network minimizes…” see Eq. 1); and updating the one or more parameters of the further machine learning model comprises maximizing a distance of predictions from the negative ground truth label (pg. 3, section 3.2 “…Specifically addressing few-shot classification, a triplet loss network minimizes distance between examples with the same label and maximizes distance between examples with different labels…”).
Nagaraju, Chong, Narayan, and Wei are considered to be analogous to the claimed invention as they are all in the same field of natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Wei in order to specifically compute a triplet loss using a negative ground truth label and to update the one or more parameters of the machine learning model by maximizing a distance of predictions from the negative ground truth label. Doing so would be beneficial, as this would enable few-shot learning for classification tasks (Wei, section 3.2).
Regarding claim 22, Nagaraju in view of Chong and Narayan does not specifically disclose computing the loss function comprises computing the loss function further based on randomly selected negative training samples from a pool of negative training samples.
Wei teaches computing the loss function comprises computing the loss function further based on randomly selected negative training samples from a pool of negative training samples (pg. 3, section 3.2 “…To sample triplets, we will consider two strategies: random sampling, which selects triplets randomly…”).
Nagaraju, Chong, Narayan, and Wei are considered to be analogous to the claimed invention as they are all in the same field of natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Wei in order to specifically compute the loss based on randomly selected negative training samples from a pool of negative training samples. Doing so would be beneficial, as this triplet loss formulation would enable few-shot learning for classification tasks (Wei, section 3.2).
Regarding claim 23, Nagaraju in view of Chong and Narayan does not specifically disclose wherein computing the loss function further based on selected negative training samples meeting a criterion involving one or more of distance, similarity, and loss.
Wei teaches wherein computing the loss function further based on selected negative training samples meeting a criterion involving one or more of distance, similarity, and loss (selected based on distance: pg. 3, section 3.2 “To sample triplets we will consider two strategies…and hard negative mining (Schroff et al., 2015), where triplets are sampled such that d(a,p) + α > d(a,n)…”).
Nagaraju, Chong, Narayan, and Wei are considered to be analogous to the claimed invention as they are all in the same field of natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Wei in order to specifically compute the loss based on selected training samples meeting a criterion involving one or more of distance, similarity, and loss. Doing so would be beneficial, as this hard negative mining strategy would provide hard examples that are difficult for the model to make correct judgements, which are more valuable and help the model learn the boundary quickly while speeding up the model convergence rate (NPL Hard Negative Mining in Nature Language Processing (How to Select Negative Examples in Classification and Rank Task), pg. 2, section 1.3).
Regarding claim 24, Nagaraju in view of Chong and Narayan does not specifically disclose wherein computing the loss function further based on an adjustable number of selected negative training samples.
Wei teaches wherein computing the loss function further based on an adjustable number of selected negative training samples (loss function dependent on number of selected triplets (number of pairs ‘i’ in Eq. 1); this number of selected triplets determines number of selected negative training samples (e.g. n triplets have n selected negative training samples); this value is set as hyperparameter: pg. 3, section 3.2 “Our triplet loss network architecture contains a linear layer with 200 hidden units, tanh activation, a dropout layer with p = 0.4, and a final layer with 40 hidden units. We use cosine distance, a margin of α=0.4, a batch size of 64 triplets, and a learning rate of 2x10-5”).
Nagaraju, Chong, Narayan, and Wei are considered to be analogous to the claimed invention as they are all in the same field of natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Wei in order to specifically compute the loss based an adjustable number of negative training samples. Doing so would be beneficial, as this triplet loss formulation would enable few-shot learning for classification tasks (Wei, section 3.2).
10. Claim 27 is rejected under 35 U.S.C. 103 as being unpatentable over Nagaraju in view of Chong and Narayan, and further in view of White & Hamzeh (US 2024/0386253 A1, hereinafter White).
Regarding claim 27, Nagaraju in view of Chong and Narayan discloses the ensemble of the expert large language models (see claim mapping for claim 26). However, Nagaraju in view of Chong and Narayan does not specifically disclose [wherein the ensemble of the expert large language models] comprises a fact checking model that searches a data source for a number of references to confirm whether the media content item matches the query.
White teaches a fact checking model that searches a data source for a number of references to confirm whether the media content item matches the query (para. 0029 “A fact checking phase 330 is next performed using the facts 320 obtained from the topic creation phase. At step 332, for each fact, searches are made in a trusted public knowledge base at step 334 and/or a trusted private knowledge base at step 336. The results of the searches at steps 334 and 336 are aggregated in a context aggregation step 338 to obtain a consolidated ground truth context 340.”; para. 0039-0046, system prompts LLM to identify if an answer (query) matches the media content item (consolidated ground truth), and uses this to determine if statement is a hallucination (false) or not (true); number of references are searched: para. 0035 “As an example, if the question or topic is medical related, then the search would include trusted medical knowledge bases, like U.S. National Institute of Health, PubMed Central; Infermedica.com; webmd.com, etc…”).
Nagaraju, Chong, Narayan, and White are considered to be analogous to the claimed invention as they are all in the same field of natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of White in order to use a fact-checking model that searches a data source for a number of references to confirm whether the media content item matches the query. Doing so would be beneficial, as this would determine if a particular data entry is correct or not (White, para. 0057).
11. Claim 28 is rejected under 35 U.S.C. 103 as being unpatentable over Nagaraju in view of Chong and Narayan, and further in view of Islam et al. (NPL Analyzing Bagging Methods for Language Models, hereinafter Islam).
Regarding claim 28, Nagaraju in view of Chong and Narayan does not specifically disclose wherein the ensemble of the expert large language models comprises a plurality of models being trained on different training data sets or having different machine learning architectures.
Islam teaches wherein the ensemble of the expert large language models comprises a plurality of models being trained on different training data sets or having different machine learning architectures (different data sets: pg. 2, section 5.1 “…We then use those hyperparameters and train n models on n bootstrapped versions of the training dataset…Each model then produces a prediction of the probability of the input belonging to each class, and we conduct an equal-weighted soft majority vote for the final class choice…”; see Fig 2, full train data split into bootstrap samples 1 to train individual models; thus each model sees different training datasets).
Nagaraju, Chong, Narayan, and Islam are considered to be analogous to the claimed invention as they are all in the same field of natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Islam in order to have the plurality of models be trained on different training data sets. Doing so would be beneficial, as performing this ensembling operation (bootstrap aggregation) would improve performance for cases where overfitting is occurring (Islam, section 7, 2nd para.)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Jain et al. (US 2023/0267175 A1): label verification procedure performed before using training data to train machine learning model, switching positive labels to negative labels (Fig. 2; para. 0102)
Manjunatha et al. (US 2024/0135165 A1): identifying false labels, and modifying the training set to obtain corrected training set (Fig. 6)
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CODY DOUGLAS HUTCHESON whose telephone number is (703)756-1601. The examiner can normally be reached M-F 8:00AM-5:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre-Louis Desir can be reached at (571)-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CODY DOUGLAS HUTCHESON/ Examiner, Art Unit 2659
/PIERRE LOUIS DESIR/ Supervisory Patent Examiner, Art Unit 2659
Prosecution Insights