Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1 – 20 are pending and examined herein.
Claims 8, 18, 16 are rejected under 35 U.S.C. 112(b).
Claims 1 – 20 are rejected under 35 U.S.C. 101.
Claims 1 – 20 are rejected under 35 U.S.C. 103.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 8, 18, 16 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 8 and 18 recites “training the classification model,” including applying the classification model in forward propagation and comparing predicted labels to evaluation labels. However, the claims then recite adjusting parameters of the machine learned language model in backpropagation. Although the adjusted parameters expressly recited as belonging to the machine learned language model, it is unclear how this step is part of “training the classification model,” and it is unclear whether the claimed training process is directed to updating the classification model, updating the machine learned language model, or both. For examination purposes, the claims would be interpreted as requiring forward propagation using the classification model to generate predicted labels, comparing the predicted labels to evaluation labels generated by the machine learned language model, and adjusting one or more parameters of the machine learned language model in backpropagation based on the comparison.
Claim 16 recites “wherein the instruction to the classification model is a search engine for an item based on a search query.” The limitation is unclear because claim 11 does not clearly provide antecedent basis for “the instruction to the classification model.” Claim 11 recites an instruction prompt to the machine learned language model and an instruction for the machine learned language model, but does not clearly recite an instruction to the classification model. Further, it is unclear how an “instruction” is “a search engine.” Accordingly, it is unclear whether claim 16 is intended to recite that the classification model is a search engine, that an instruction prompt describes a search engine task, or some other relationship between the instruction and the search engine. For examination purposes, the limitation is treated as reciting that the classification model is used in a search engine context for identifying an item based on a search query.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 - 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
MPEP § 2109(III) sets out steps for evaluating whether a claim is drawn to patent-eligible subject matter. The analysis of claims 1 – 20, in accordance with these steps, follows.
Step 1 Analysis:
Step 1 is to determine whether the claim is directed to a statutory category (process, machine, manufacture, or composition of matter.
Claims 1 – 10 are directed to a method, meaning that it is directed to the statutory category of process. Claims 11 – 19 are directed to a computer program product comprising one or more non-transitory computer- readable storage media, which is the statutory category of manufacture. Claim 20 is directed to a system, which can be an article of machine.
Step 2A Prong One, Step 2A Prong Two, and Step 2B Analysis:
Step 2A Prong One asks if the claim recites a judicial exception (abstract idea, law of nature, or natural phenomenon). If the claim recites a judicial exception, analysis proceeds to Step 2A Prong Two, which asks if the claim recites additional elements that integrate the abstract idea into a practical application. If the claim does not integrate the judicial exception, analysis proceeds to Step 2B, which asks if the claim amounts to significantly more than the judicial exception. If the claim does not amount to significantly more than the judicial exception, the claim is not eligible subject matter under 35 U.S.C. 101.
Regarding claim 1, the following claim elements are abstract ideas:
an instruction for the machine-learned language model to generate an evaluation label of a training sample of a classification model, (Evaluating information and assigning label is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.)
… wherein the evaluation label is to be used as a training label for the training sample in a supervised training of the classification model; (Defining the label as evaluation result for the sample is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.)
each response comprising the evaluation label corresponding to each evaluation request prompt; (Getting response being the evaluation label as the generated output from evaluation request is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.)
The following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
at an online system comprising one or more processors and one or more computer-readable media: (This is mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
providing an instruction prompt to a machine-learned language model, (This is mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
(2) a textual format related to how data is arranged, (This is mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
providing a batch of evaluation request prompts to the machine-learned language model, (This is mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
each evaluation request prompt comprising data that is at least partially arranged in the textual format described in the instruction prompt; (This is mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
receiving a plurality of responses from the machine-learned language model, (This is mere data gathering, an insignificant extra solution activity, which does not integrate the judicial exception into a practical application. The broadest reasonable interpretation of this claim is storing information in memory, which is a well-understood, routine conventional activity. See MPEP § 2106.05(d)(II)(iv). Therefore, this does not amount to significantly more than the judicial exception.)
storing at least evaluation labels and the data in the evaluation request prompts as training samples for the supervised training of the classification model. (This is mere data gathering, an insignificant extra solution activity, which does not integrate the judicial exception into a practical application. The broadest reasonable interpretation of this claim is storing information in memory, which is a well-understood, routine conventional activity. See MPEP § 2106.05(d)(II)(iv). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 2, the rejection of claim 1 is incorporated herein. Claim 2 further recites the following abstract idea:
and including a chain-of-thoughts instruction for the machine-learned language model to follow in performing evaluation, (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
the chain-of-thoughts instruction including an explanation for each evaluation label on how the training sample should be classified to the evaluation label. (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
Claim 2 further recites the following additional elements:
specifying an input textual format of an input that the machine-learned language model is to receive; (This is mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
specifying an output textual format of an output that the machine-learned language model is to generate; (This is mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 3, the rejection of claim 1 is incorporated herein. Claim 3 further recites the following abstract idea:
and specifying an output format that includes the evaluation label and a reason for assigning the evaluation label. (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
Claim 3 further recites the following additional elements:
providing a multimodal input example that follows the textual format, the multimodal input example including an input to the classification model, an output of the classification model, attribute data retrieved from a database of the online system, and an image; (This is mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 4, the rejection of claim 1 is incorporated herein. Claim 4 further recites the following abstract idea:
specifying a set of candidate evaluation labels in the instruction for the machine- learned language model to generate the evaluation label; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
providing an example for each candidate evaluation label on one or more selection criteria for the candidate evaluation label; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
and providing a chain-of-thoughts instruction in applying the one or more selection criteria for each candidate evaluation label. (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
Claim 4 doesn’t recite additional elements:
Regarding claim 5, the rejection of claim 1 is incorporated herein. Claim 5 further recites the following additional elements:
comprises one or more of: a set of candidate evaluation labels; an example of input of the machine-learned language model, the input being in the textual format; an example of output of the machine-learned language model; a chain-of-thought instruction; or an explanation for selecting each candidate evaluation label. (This is mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 6, the rejection of claim 1 is incorporated herein. Claim 6 further recites the following abstract idea:
including a first instruction for the machine-learned language model to identify a search intent from the search query; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
including a second instruction for the machine-learned language model to identify one or more attributes of the item returned by the search engine; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
and providing one or more examples to the machine-learned language model on how the evaluation label is assigned based on comparing the one or more attributes of the item to the search intent. (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
Claim 6 further recites the following additional elements:
wherein the classification model is a search engine for an item based on a search query, and wherein providing the instruction prompt to the machine- learned language model comprises: (This is mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 7, the rejection of claim 1 is incorporated herein. Claim 7 further recites the following additional elements:
retrieving historical inputs and historical outputs of the classification model that are stored in a database; (This is mere data gathering and outputting, an insignificant extra solution activity, which does not integrate the judicial exception into a practical application. The broadest reasonable interpretation of this claim is storing information in memory, which is a well-understood, routine conventional activity. See MPEP § 2106.05(d)(II)(iv). Therefore, this does not amount to significantly more than the judicial exception.)
formatting the historical inputs and historical outputs according to the textual format that is provided in the instruction prompt; (This is mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
and generating the batch of evaluation request prompts, each evaluation request prompt includes the historical inputs and historical outputs. (This is mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 8, the rejection of claim 1 is incorporated herein. Claim 8 further recites the following abstract idea:
applying, in forward propagation, the classification model to the training samples to generate predicted labels; (Applying forward propagation is merely mathematical calculation, which is mathematical concept.)
comparing the predicted labels to the evaluation label generated by the machine- learned language model; (Comparing the predicted labels to the evaluation labels is merely mathematical calculation, which is mathematical concept.)
and adjusting, in backpropagation, one or more parameters of the machine-learned language model based on the comparing. (Adjusting parameters and using backpropagation is merely mathematical calculation, which is mathematical concept.)
Claim 8 further recites the following additional elements:
retrieving the training samples of the classification model, the training samples each comprising the evaluation label generated by the machine-learned language model; (This is mere data gathering, an insignificant extra solution activity, which is a well-understood, routine conventional activity. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(d). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 9, the rejection of claim 1 is incorporated herein. Claim 9 further recites the following abstract idea:
identifying, by the classification model, an item based on the query; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.)
Claim 9 further recites the following additional elements:
receiving a query for the classification model; (This is mere data gathering, an insignificant extra solution activity, which is a well-understood, routine conventional activity. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(d). Therefore, this does not amount to significantly more than the judicial exception.)
applying the machine-learned language model to generate the evaluation label for the item; (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
and providing the evaluation label as a classification in response to the query. (This is mere data gathering and outputting, an insignificant extra solution activity, which does not integrate the judicial exception into a practical application. The broadest reasonable interpretation of this claim is storing information in memory, which is a well-understood, routine conventional activity. See MPEP § 2106.05(d)(II)(iv). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 10, the rejection of claim 1 is incorporated herein. Claim 10 further recites the following abstract idea:
identifying, by the classification model, a plurality of items based on the query; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.)
and selecting, based on the evaluation labels, one of the plurality of items as a sponsored item to be displayed as a response to the query. (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.)
Claim 10 further recites the following additional elements:
receiving a query for the classification model; (This is mere data gathering, an insignificant extra solution activity, which is a well-understood, routine conventional activity. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(d). Therefore, this does not amount to significantly more than the judicial exception.)
applying the machine-learned language model to generate evaluation labels for the plurality of items; (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Claims 11 – 19 recite substantially similar subject matter to claims 1 – 9 respectively and are rejected with the same rationale, mutatis mutandis.
Claim 20 recites substantially similar subject matter to claim 1 respectively and is rejected with the same rationale, mutatis mutandis.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 8, 11, 18, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Smith et al. (U.S. Pub. 2024/0160900) in view of Bansal et al (NPL: “Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost”).
Regarding Claim 1, Smith teaches
providing an instruction prompt to a machine-learned language model, the instruction prompt comprising (1) an instruction for the machine-learned language model to generate an evaluation label of a training sample of a classification model, and (2) a textual format related to how data is arranged, wherein the evaluation label is to be used as a training label for the training sample in a supervised training of the classification model; ([0016] of Smith states “An alternative approach is to submit a query to a pre-trained foundation or large language model (LLM), such as one used for question answering1 for each document and use the high confidence predictions from the large language model as weakly supervised labels; This is a form of prompting a foundation or large language model to generate a label for individual data points; In some embodiments, this process may include using a provided query or templated query as a prompt for the large language model, wherein a system creates corresponding prompts submitted to the language model, and the output of the large language model is a prediction (suggested label) for one or more of the datapoints in the dataset; Based on a confidence level of the predictions output by the large language model and the associated datapoints, forming one or more groups or clusters of datapoints by interactively thresholding a confidence level of the predictions;” [0277] of Smith states “and using a plurality of new datapoints and the new datapoints' assigned labels to train a machine learning or other form of model” Smith teaches prompting the LLM to generate a label for individual datapoints. The label is used as a weakly supervised training label for the downstream classification model. Also, Smith’s provided query or templated query is a predefined text template that defines how each input datapoint’s data is structured within the prompt submitted to the LLM.)
providing a batch of evaluation request prompts to the machine-learned language model, each evaluation request prompt comprising data that is at least partially arranged in the textual format described in the instruction prompt; ([0016] of Smith states “The predictions provide a set of weakly supervised labels that may be used downstream to generate annotated training data for a model;” [0277] of Smith states “and using a plurality of new datapoints and the new datapoints' assigned labels to train a machine learning or other form of model” Smith teaches generating corresponding prompts for each data point in a dataset, where each prompts are arranged with templated format. POSITA would recognize Smith’s applying the template to each datapoint in a dataset as a batch or plurality of evaluation request prompts.)
receiving a plurality of responses from the machine-learned language model, each response comprising the evaluation label corresponding to each evaluation request prompt; ([0016] of Smith states “In some embodiments, this process may include using a provided query or templated query as a prompt for the large language model, wherein a system creates corresponding prompts submitted to the language model, and the output of the large language model is a prediction (suggested label) for one or more of the datapoints in the dataset;” [0277] of Smith states “10. A method of training a model, comprising: generating a prediction for one or more of datapoints in a dataset using a large language model (LLM), wherein each of the datapoints is input to the large language model with a prompt and the generated prediction is an output of the large language model representing a label for the datapoint;” LLM generates a label as the output for each submitted data point.)
However, Smith does not explicitly teach that
and storing at least evaluation labels and the data in the evaluation request prompts as training samples for the supervised training of the classification model.
Bansal teaches that
and storing at least evaluation labels and the data in the evaluation request prompts as training samples for the supervised training of the classification model. (Pg. 6 of Bansal states “Step 3: Annotating sampled inputs using LLM Given a sampled set of unlabelled input Usampled, we use LLM annotations for these inputs to get an annotated set as L′ sampled. We denote LLM annotations function by T ′ : X → {0, 1}, and hence the LLM annotation for input xi as t′ ij ∈ {0, 1}. The augmented dataset made from U is hence L′ L′ sampled = {(xi, t′ i) : xi ∈ U} (7) Step 4: Finetuning classifier on augmented labelled data Finally we finetune the base model on the augmented dataset L+L′ sampled using Eq 4.” Bansal’s eq 7 shows an explicitly named, stored dataset pairing each input data point with its LLM generated label. Then step 4 passes L’ sampled directly to classifier finetuning, which establishes the (data, label) pairs stored as training samples and used in supervised training.)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Smith and Bansal. Smith teaches using a large language model to generate labels for datapoints and using the generated labels as training labels for a downstream machine learning model. Bansal teaches forming an augmented dataset of input samples paired with LLM generated annotations and finetuning a base model on that augmented dataset. One with the ordinary skill in the art would have been motivated to incorporate the teachings of Bansal into the Smith to store the LLM generated labels with the corresponding input data and use the resulting labeled samples for supervised training. The combination would have been predictable use of known LLM based annotation and supervised training techniques to improve quality and robustness of the scalable training data generation.
Regarding claim 8, the rejection of claim 1 is incorporated herein. The combination of Smith and Bansal teaches
retrieving the training samples of the classification model, the training samples each comprising the evaluation label generated by the machine-learned language model; (Pg. 6 of Bansal states “Step 3: Annotating sampled inputs using LLM Given a sampled set of unlabelled input Usampled, we use LLM annotations for these inputs to get an annotated set as L′ sampled. We denote LLM annotations function by T ′ : X → {0, 1}, and hence the LLM annotation for input xi as t′ ij ∈ {0, 1}. The augmented dataset made from U is hence L′ L′ sampled = {(xi, t′ i) : xi ∈ U} (7) Step 4: Finetuning classifier on augmented labelled data Finally we finetune the base model on the augmented dataset L+L′ sampled using Eq 4.” Basal teaches retrieving the augmented dataset comprising data inputs each paired with the LLM generated annotation label.)
applying, in forward propagation, the classification model to the training samples to generate predicted labels; (Pg. 6 of Bansal states “As the first step, we finetune our base model on the labeled data L to get a finetuned model f i.e., f = argminfE(xi,ti)∈L[L(f(xi), ti)] (4) Using f, we compute the Conditional Informativeness score zi(f, f0) where f0 is the base model, for each unlabeled input xi ∈ U i.e.” Bansal’s finetuning requires applying the classifier f to each training sample to compute. A POSITA would understand this is the standard forward propagation step in neural network finetuning.)
comparing the predicted labels to the evaluation label generated by the machine- learned language model; (Pg. 4 of Bansal states “These are examples where the base model either obtains the lowest mean squared error w.r.t. the ground truth labels or obtains the highest mean squared error. That is, half of the examples (30%) are the easy examples the base model is (most) correct on and the remaining half are the hard examples where the base model is (most) incorrect on. Further, we remove labels from the target domain. Hence from the original data we have 40% “source” labeled examples and 60% “target” unlabeled examples. For accuracy evaluation on both source and target domains, we create analogous domains over the test set too. We consider an active learning setup where selected inputs from target domain can be annotated by an LLM and augmented in the training set. After augmentation, the model is trained on the source domain + augmented dataset. We consider two popular active-sampling approaches in literature: Random and Uncertainty-based sampling.” Bansal teaches that the augmented dataset includes input samples paired with LLM generated labels and the base model is finetuned on the augmented dataset using MSE loss. When the classification model is trained on these samples, the model output for each input is compared against the corresponding LLM generated label. )
and adjusting, in backpropagation, one or more parameters of the machine-learned language model based on the comparing. (Pg. 6 of Bansal states “Step 3: Annotating sampled inputs using LLM Given a sampled set of unlabelled input Usampled, we use LLM annotations for these inputs to get an annotated set as L′ sampled. We denote LLM annotations function by T ′ : X → {0, 1}, and hence the LLM annotation for input xi as t′ ij ∈ {0, 1}. The augmented dataset made from U is hence L′ L′ sampled = {(xi, t′ i) : xi ∈ U} (7) Step 4: Finetuning classifier on augmented labelled data Finally we finetune the base model on the augmented dataset L+L′ sampled using Eq 4.” Under BRI that ‘machine learned language model’ here refers to the classification model as the claim is regarding training the classification model, Bansal teaches adjusting that model’s parameters via gradient based finetuning to minimize the loss comparing predicted labels to LLM generated labels.)
Claims 11, 18 recite substantially similar subject matter to claims 1, 8 respectively and are rejected with the same rationale, mutatis mutandis.
Claim 20 recites substantially similar subject matter to claim 1 respectively and is rejected with the same rationale, mutatis mutandis.
Claims 2, 4, 5, 12, 14, 15 are rejected under 35 U.S.C. 103 as being unpatentable over Smith et al. (U.S. Pub. 2024/0160900) in view of Bansal et al (NPL: “Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost”), further in view of He et al. (NPL: “AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators”).
Regarding claim 2, the rejection of claim 1 is incorporated herein. The combination of Smith and Bansal does not explicitly teach
specifying an input textual format of an input that the machine-learned language model is to receive;
specifying an output textual format of an output that the machine-learned language model is to generate;
and including a chain-of-thoughts instruction for the machine-learned language model to follow in performing evaluation, the chain-of-thoughts instruction including an explanation for each evaluation label on how the training sample should be classified to the evaluation label.
He teaches that
specifying an input textual format of an input that the machine-learned language model is to receive; (Pg. 2 of He states “The instructions for each task mainly includes three parts: task description, category definition, and demonstrated examples” Pg. 4 of He states “Directions: Given a search engine query: "google data studio sharepoint", first, consider what the user could have in mind when they type in the query and allow for misspellings or other ambiguity, then classify the relevance of keyword: "sharepoint migration tool file share" to the query into one of the following categories: "Not bad", "Bad".” HE teaches specifying the input textual format of the LLM’s input through a prompt structure comprising a task description slot, category definition slot, and demonstrated example slots” He’s prompt specifies the textual arrangement of the LLM input by identifying the query field and the keyword field.)
specifying an output textual format of an output that the machine-learned language model is to generate; (Pg. 4 of He states “Given a search engine query: "google data studio sharepoint", first, consider what the user could have in mind when they type in the query and allow for misspellings or other ambiguity, then classify the relevance of keyword: "sharepoint migration tool file share" to the query into one of the following categories: "Not bad", "Bad"… Briefly explain why the relevance is "Bad", with a response length not exceeding 100 words.” Pg. 13 of He states “This includes but is not limited to: incorrect/unrelated product, unrelated topic, wrong location when location is important, cannot be used in place of query product nor are they commonly purchased together, etc. Please predict whether the keyword is relevant to the query or not. The answer should be exact "Not bad" or "Bad".” He specifies the textual form of the LLM output by requiring the answer to be exactly one of the candidate labels and by requiring a brief explanation to be generated.)
and including a chain-of-thoughts instruction for the machine-learned language model to follow in performing evaluation, the chain-of-thoughts instruction including an explanation for each evaluation label on how the training sample should be classified to the evaluation label. (Pg. 2 of He states “Naturally, we can guide GPT-3.5 to annotate data using the same approach as with human annotators by providing it with task definitions and example samples. Furthermore, we found that requesting a LLM to furnish the rationale behind the ground truth label or answer for a particular example can prompt the LLM to produce high-quality explanations. Based on this, we construct the few-shot chain-of-thought prompt (Wei et al., 2022) with the self-generated explanations to annotate data. We refer to this method as ‘explain-then-annotate’, which further improves the annotation quality… Recent work (Wei et al., 2022) has shown that adding human written rationales to the demonstrated examples, called as chain-of-thought (CoT), can elicit the LLMs’ reasoning ability, thus gaining improvements on reasoning tasks. In this paper, we find that GPT-3.5 2 is a good reasoner who can automatically generate reasonable explanations for demonstrated examples. In the following, we will introduce how to generate explanations with GPT-3.5, and then create few-shot CoT prompts with the generated explanations. Generating Explanations with GPT-3.5. In this step, we simulate the way humans explain problems to induce GPT-3.5 to explain the annotated examples. To be concrete, we present the task description, specific example, and the corresponding true labels to GPT-3.5, and then ask it to answer why the corresponding answer for that example is the given label. By doing so, GPT-3.5 will generate reasonable explanations… Creating Few-shot CoT Prompts. After getting the explanations generated by GPT-3.5, we can construct the few-shot CoT prompt. We show the few-shot CoT prompts for GPT-3.5 on the user query and keyword relevance assessment, WiC and BoolQ tasks in Tables 16, 17 and 18, respectively.” He teaches constructing a few shot chain of thought prompt using self-generated explanations and using that prompt to annotate unlabeled data. He further teaches providing labeled samples for each category. It would have been obvious to include explained demonstrated examples for each candidate label so the LLM can follow how a sample should be classified to that label. )
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Smith, Bansal, and He. Smith teaches using a large language model to generate labels for datapoints and using the generated labels as training labels for a downstream machine learning model. Bansal teaches forming an augmented dataset of input samples paired with LLM generated annotations and finetuning a base model on that augmented dataset. He teaches structured LLM prompting for annotation, including defined input/output formats and few shot chain of thought prompting with self-generated explanations. One with the ordinary skill in the art would have been motivated to incorporate the teachings of He into the combination of Smith and Bansal to make the LLM annotation process more consistent, interpretable, and accurate. The combination would have been predictable use of known prompt engineering and application of explanation-based annotation technique to the known LLM labeling framework in combination of Smith and Bansal.
Regarding claim 4, the rejection of claim 1 is incorporated herein. The combination of Smith, Bansal and He teaches
specifying a set of candidate evaluation labels in the instruction for the machine- learned language model to generate the evaluation label; (Pg. 13 of He states “then classify the relevance of keyword to the query into one of the following categories: "Not bad", or "Bad"… This includes but is not limited to: incorrect/unrelated product, unrelated topic, wrong location when location is important, cannot be used in place of query product nor are they commonly purchased together, etc. Please predict whether the keyword is relevant to the query or not. The answer should be exact "Not bad" or "Bad".” Example of prompts in He enumerates exactly two candidate evaluation labels, which are the set of candidate evaluation labels specified in the instruction.)
providing an example for each candidate evaluation label on one or more selection criteria for the candidate evaluation label; (Pg. 12 of He states “"Not bad": the keyword is relevant to the user’s search query. This can include: broader or narrower product selection, competitor or alternative products, accessories, products often purchased together and related topics as well as direct matches to the user’s search. "Bad": the keyword is not relevant to the user’s search query. There is no relationship between the query and keyword. This includes but is not limited to: incorrect/unrelated product, unrelated topic, wrong location when location is important, cannot be used in place of query product nor are they commonly purchased together, etc.” For each candidate evaluation label, He provides a definition comprising the selection criteria for that label.)
Regarding claim 5, the rejection of claim 1 is incorporated herein. The combination of Smith, Bansal and He teaches
the instruction prompt further comprises one or more of: a set of candidate evaluation labels; an example of input of the machine-learned language model, the input being in the textual format; an example of output of the machine-learned language model; a chain-of-thought instruction; or an explanation for selecting each candidate evaluation label. (Pg. 12 of He states “Directions: Given a search engine query: "google data studio sharepoint", first, consider what the user could have in mind when they type in the query and allow for misspellings or other ambiguity, then classify the relevance of keyword: "sharepoint migration tool file share" to the query into one of the following categories: "Not bad", "Bad". The definitions of the categories are "Not bad": the keyword is relevant to the user’s search query. This can include: broader or narrower product selection, competitor or alternative products, accessories, products often purchased together and related topics as well as direct matches to the user’s search. "Bad": the keyword is not relevant to the user’s search query. There is no relationship between the query and keyword. This includes but is not limited to: incorrect/unrelated product, unrelated topic, wrong location when location is important, cannot be used in place of query product nor are they commonly purchased together, etc. Briefly explain the relevance between the keyword and query, with a response length not exceeding 100 Words” He teaches an instruction prompt comprising candidate evaluation labels, demonstrated input/output examples, a chain of though instruction, and per label explanations.)
Claims 12, 14, 15 recite substantially similar subject matter to claims 2, 4, 5 respectively and are rejected with the same rationale, mutatis mutandis.
Claims 7, 9, 17, 19 are rejected under 35 U.S.C. 103 as being unpatentable over Smith et al. (U.S. Pub. 2024/0160900) in view of Bansal et al (NPL: “Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost”), further in view of Zhu et al. (NPL: “Query-LIFE: Query-aware Language Image Fusion Embedding for E-Commerce Relevance”).
Regarding claim 7, the rejection of claim 1 is incorporated herein. The combination of Smith and Bansal teaches
formatting the historical inputs and historical outputs according to the textual format that is provided in the instruction prompt; ([0016] of Smith states “An alternative approach is to submit a query to a pre-trained foundation or large language model (LLM), such as one used for question answering1 for each document and use the high confidence predictions from the large language model as weakly supervised labels; This is a form of prompting a foundation or large language model to generate a label for individual data points; In some embodiments, this process may include using a provided query or templated query as a prompt for the large language model, wherein a system creates corresponding prompts submitted to the language model, and the output of the large language model is a prediction (suggested label) for one or more of the datapoints in the dataset;” Pg. 7 of Bansal states “Consider a set of unlabeled examples U consisting of pairs (xi, yi) to be annotated by the LLM. We construct a prompt which consists of set of pairs (xi, yi) of sentences.” A POSITA would combine the teachings from each reference to format the historical inputs (queries) and historical outputs (product data) from click log in Zhu according to the instruction prompt’s textual format.)
and generating the batch of evaluation request prompts, each evaluation request prompt includes the historical inputs and historical outputs. ([0277] of Smith states “generating a prediction for one or more of datapoints in a dataset using a large language model (LLM), wherein each of the datapoints is input to the large language model with a prompt and the generated prediction is an output of the large language model representing a label for the datapoint;” Pg. 6 of Bansal states “Step 3: Annotating sampled inputs using LLM Given a sampled set of unlabelled input Usampled, we use LLM annotations for these inputs to get an annotated set as L′ sampled. We denote LLM annotations function by T ′ : X → {0, 1}, and hence the LLM annotation for input xi as t′ ij ∈ {0, 1}. The augmented dataset made from U is hence L′ L′ sampled = {(xi, t′ i) : xi ∈ U} (7) Step 4: Finetuning classifier on augmented labelled data Finally we finetune the base model on the augmented dataset L+L′ sampled using Eq 4.” Smith and Bansal teaches generating a prompt for each datapoint in a dataset. Combining with click log pairs in Zhu, it teaches generating a batch of evaluation request prompts each comprising a historical input (query) and historical output (product data).)
However, the combination does not explicitly teach that
Zhu teaches that
retrieving historical inputs and historical outputs of the classification model that are stored in a database; (Pg. 5 of Zhu states “There are three training datasets. The first dataset samples 5M product title-image pairs. The second dataset samples 1.3M <query,title,image> positive pairs from the online clicking log of Miravia Search. The third dataset has 200,000 <query,title,image> labeled data, and 30K samples are selected as the evaluation set, where the ratio of positive to negative is 1:1.” Pg. 1 of Zhu states “Relevance module plays a fundamental role in e-commerce search as they are responsible for selecting relevant products from thousands of items based on user queries, thereby enhancing users experience and efficiency” Pg. 4 of Zhu states “In ITM, we frame the task as a binary classification problem where the model predicts whether an image-text pair is positive or negative. To obtain the matching score, we pass the model’s output through a two-class linear classifier, yielding a logit.We employ a hard negative mining strategy [11] and leverage labeled data.” Zhu teaches retrieving historical pairs from the online clicking log and the query is the historical input to the search classification model. Product title, image, and attributes are historical output data of the search model when the model returned output.)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Smith, Bansal, and Zhu. Smith teaches using a large language model to generate labels for datapoints and using the generated labels as training labels for a downstream machine learning model. Bansal teaches forming an augmented dataset of input samples paired with LLM generated annotations and finetuning a base model on that augmented dataset. Zhu teaches an e-commerce search relevance system in which query item pairs, product attributes, and product images are used to evaluate and classify item relevance to a user-query. One with the ordinary skill in the art would have been motivated to incorporate the teachings of Zhu into the combination of Smith and Bansal to generate evaluation labels for search result items based on the query, item attributes, and image information. The combination would have been predictable use of known LLM annotation techniques to known e-commerce relevance classification data.
Regarding claim 9, the rejection of claim 1 is incorporated herein. The combination of Smith, Bansal, and Zhu teaches
receiving a query for the classification model; identifying, by the classification model, an item based on the query; (Pg. 1 of Zhu states “Relevance module plays a fundamental role in e-commerce search as they are responsible for selecting relevant products from thousands of items based on user queries, thereby enhancing users experience and efficiency… Query-LIFE has been deployed on Miravia Search1, resulting in improved both relevance and conversion efficiency… Consequently, the relevance of products exposed to users triggered by their queries plays a crucial role in users shopping experiences, and also the transaction efficiency. Therefore, it is essential to accurately judge whether the candidate products are relevant to the user intentions for an e-commerce search engine” Pg. 7 of Zhu states “Furthermore, we carry out online A/B experiments, and BERT is the current baseline in Miravia Search. Annotators are invited to evaluate whether the relevance is improved by the proposed method. 10K query-item pairs are sampled top-10 items both from the buckets of BERT and Query-LIFE respectively.” Zhu teaches that search platform receives a user query and the relevance module (classification model) selects relevant products from dataset based on that query.)
applying the machine-learned language model to generate the evaluation label for the item; ([0277] of Smith states “generating a prediction for one or more of datapoints in a dataset using a large language model (LLM), wherein each of the datapoints is input to the large language model with a prompt and the generated prediction is an output of the large language model representing a label for the datapoint; based on a confidence level of the predictions output by the large language model and the associated datapoints, selecting a subset of the datapoints to apply the label to by interactively thresholding a confidence level of the predictions;” Pg. 6 of Bansal states “Step 3: Annotating sampled inputs using LLM Given a sampled set of unlabelled input Usampled, we use LLM annotations for these inputs to get an annotated set as L′ sampled. We denote LLM annotations function by T ′ : X → {0, 1}, and hence the LLM annotation for input xi as t′ ij ∈ {0, 1}. The augmented dataset made from U is hence L′ L′ sampled = {(xi, t′ i) : xi ∈ U} (7)”)
and providing the evaluation label as a classification in response to the query. (Pg. 2 of Zhu states “Hence, we use inner product of <query, multi-modal representation> to measure the relevance between one query and the multi-modal representation above, as shown in Figure 1(c). Thirdly, we use supervised contrastive learning and utilize generating ability both from multi-modal large model and large language model to filter out the false negative samples.” Pg. 6 of Zhu states “The relevance of a query-item can be divided into three types: Excellent, Fair and Bad. Excellent means same as the original highly standard, the item’s core products, functional attributes and other attributes perfectly match the query requirements. Fair means the core product is the same as query, but the functional attributes are inconsistent. Bad means the core products are different or the core products are the same, but the retained attributes in the brand or other key industries are different. We count the proportions of the three indicators of different models.” Pg. 7 of Zhu states “10K query-item pairs are sampled top-10 items both from the buckets of BERT and Query-LIFE respectively. The results are shown in Table 4, compared with BERT, the main improvement is that the Excellent ratio increased by 4.42% and the Bad ratio decreased by 2.79%. Since the search relevance is one key aspect of user experience, the conversion efficiency is also improved as a consequence.” Zhu teaches providing the relevance evaluation label as a classification for the query-item pair, in response to the search query. A POSITA would have used the LLM generated evaluation label as the relevance classification for the query item pair in response to the query.)
Claims 17, 19 recite substantially similar subject matter to claims 7, 9 respectively and are rejected with the same rationale, mutatis mutandis.
Claims 3, 6, 13, 16 are rejected under 35 U.S.C. 103 as being unpatentable over Smith et al. (U.S. Pub. 2024/0160900) in view of Bansal et al (NPL: “Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost”), He et al. (NPL: “AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators”), further in view of Zhu et al. (NPL: “Query-LIFE: Query-aware Language Image Fusion Embedding for E-Commerce Relevance”).
Regarding claim 3, the rejection of claim 1 is incorporated herein. The combination of Smith, Bansal, and He teaches
and specifying an output format that includes the evaluation label and a reason for assigning the evaluation label. (Pg. 5 of He states “To answer this question, we induce LLM to generate explanations using prompts that contain ground truth labels, and prompts that do not contain labels (we only replace the last sentence of the original prompt “Briefly explain why the relevance is "Bad", with a response length not exceeding 100 words.” With "Briefly explain the relevance between the keyword and query, with a response length not exceeding 100 words.").”)
However, the combination does not explicitly teach
providing a multimodal input example that follows the textual format, the multimodal input example including an input to the classification model, an output of the classification model, attribute data retrieved from a database of the online system, and an image;
Zhu teaches that
providing a multimodal input example that follows the textual format, the multimodal input example including an input to the classification model, an output of the classification model, attribute data retrieved from a database of the online system, and an image; (Pg. 1 of Zhu states “Traditionally, relevance model [2, 9, 10, 22, 25] have primarily relied on textual information, such as query and product descriptions (title, attribute, etc) to judge the relevance between queries and products. However the product information also includes images which captures a significant portion of user attention during browsing products, thus it is becoming increasingly essential to incorporate image into relevance modeling. This integration of both image and text data has the potential to provide a more comprehensive understanding of the products and better capture user intent.” Pg. 2 of Zhu states “As shown in Figure 1(a), it integrates query, title, and image into relevance tasks. Firstly, we randomly sample <query,title,image> triplet data from the online user behavior logs as pre-training data.” Pg. 5 of Zhu states “The first dataset samples 5M product title-image pairs. The second dataset samples 1.3M <query,title,image> positive pairs from the online clicking log of Miravia Search. The third dataset has 200,000 <query,title,image> labeled data, and 30K samples are selected as the evaluation set, where the ratio of positive to negative is 1:1.”)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Smith, Bansal, He, and Zhu. Smith teaches using a large language model to generate labels for datapoints and using the generated labels as training labels for a downstream machine learning model. Bansal teaches forming an augmented dataset of input samples paired with LLM generated annotations and finetuning a base model on that augmented dataset. He teaches structured LLM prompting for annotation, including defined input/output formats and few shot chain of thought prompting with self-generated explanations. Zhu teaches an e-commerce search relevance system in which query item pairs, product attributes, and product images are used to evaluate and classify item relevance to a user-query. One with the ordinary skill in the art would have been motivated to incorporate the teachings of Zhu into the combination of Smith, He, and Bansal to generate more reliable relevance labels for search result items. The combination would have been predictable application of LLM annotation, prompt formatting, explanation-based labeling, and e-commerce relevance classification techniques.
Regarding claim 6, the rejection of claim 1 is incorporated herein. The combination of Smith, Bansal, He, and Zhu teaches
wherein the classification model is a search engine for an item based on a search query, (Pg. 1 of Zhu states “Relevance module plays a fundamental role in e-commerce search as they are responsible for selecting relevant products from thousands of items based on user queries, thereby enhancing users experience and efficiency… However, the performance is sub-optimal because the vision-language pre-training models lack of alignment specifically designed for queries. In this paper, we propose a method called Query-LIFE (Query-aware Language Image Fusion Embedding) to address these challenges. Query-LIFE utilizes a query-based multimodal fusion to effectively incorporate the image and title based on the product types… Moreover, Query-LIFE has been deployed on Miravia Search1, resulting in improved both relevance and conversion efficiency.” Relevance module from Zhu, which functions like classification model) works as a search engine for selecting items based on a search query.)
including a first instruction for the machine-learned language model to identify a search intent from the search query; (Pg. 12 of He states “Directions: Given a search engine query: "google data studio sharepoint", first, consider what the user could have in mind when they type in the query and allow for misspellings or other ambiguity,” He teaches that in instruction, prompt directs LLM to consider what the user had in mind, which could be the search intent, when writing search query.)
including a second instruction for the machine-learned language model to identify one or more attributes of the item returned by the search engine; (Pg. 5 of Zhu states “The first module is generating, we employ a large language model (LLM) and a multimodal model (InstructBLIP) [5] to extract key text features from the product title and image respectively.” Pg. 8 of Zhu states “input of the product title, extract the core word, material, brand, color, and model parameters from the title and provide structured output.” Zhu teaches using an LLM to identify product attributes of items returned by the search model. A POSITA would include second instruction to identify these attributes for accurate relevance labeling.)
and providing one or more examples to the machine-learned language model on how the evaluation label is assigned based on comparing the one or more attributes of the item to the search intent. (Pg. 1 of Zhu states “Everyday, millions of users search & browse products, and maybe finally place orders in e-commerce platforms. Consequently, the relevance of products exposed to users triggered by their queries plays a crucial role in users shopping experiences, and also the transaction efficiency. Therefore, it is essential to accurately judge whether the candidate products are relevant to the user intentions for an e-commerce search engine.” Pg. 6 of Zhu states “The relevance of a query-item can be divided into three types: Excellent, Fair and Bad. Excellent means same as the original highly standard, the item’s core products, functional attributes and other attributes perfectly match the query requirements. Fair means the core product is the same as query, but the functional attributes are inconsistent. Bad means the core products are different or the core products are the same, but the retained attributes in the brand or other key industries are different. We count the proportions of the three indicators of different models.” Pg. 1 of He states “In this paper, we first claim that large language models (LLMs), such as GPT-3.5, can serve as an excellent crowdsourced annotator by providing them with sufficient guidance and demonstrated examples. To make LLMs to be better annotators, we propose a two-step approach, ‘explain-then-annotate’. To be more precise, we begin by creating prompts for every demonstrated example, which we subsequently utilize to prompt a LLM to provide an explanation for why the specific ground truth answer/label was chosen for that particular example. Following this, we construct the few-shot chain-of-thought prompt with the self-generated explanation and employ it to annotate the unlabeled data.” A POSITA would combine these to provide evaluation label examples showing how each label is assigned by comparing item attributes to the identified search intent.)
Claims 13, 16 recite substantially similar subject matter to claims 3, 6 respectively and are rejected with the same rationale, mutatis mutandis.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Smith et al. (U.S. Pub. 2024/0160900) in view of Bansal et al (NPL: “Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost”), further in view of Saha et al. (U.S. Pub. 2024/0257175).
Regarding claim 10, the rejection of claim 1 is incorporated herein. The combination of Smith and Bansal teaches
applying the machine-learned language model to generate evaluation labels for the plurality of items; ([0277] of Smith states “generating a prediction for one or more of datapoints in a dataset using a large language model (LLM), wherein each of the datapoints is input to the large language model with a prompt and the generated prediction is an output of the large language model representing a label for the datapoint; based on a confidence level of the predictions output by the large language model and the associated datapoints, selecting a subset of the datapoints to apply the label to by interactively thresholding a confidence level of the predictions;” Pg. 6 of Bansal states “Step 3: Annotating sampled inputs using LLM Given a sampled set of unlabelled input Usampled, we use LLM annotations for these inputs to get an annotated set as L′ sampled. We denote LLM annotations function by T ′ : X → {0, 1}, and hence the LLM annotation for input xi as t′ ij ∈ {0, 1}. The augmented dataset made from U is hence L′ L′ sampled = {(xi, t′ i) : xi ∈ U} (7)”)
However, the combination does not explicitly teach that
receiving a query for the classification model; identifying, by the classification model, a plurality of items based on the query;
and selecting, based on the evaluation labels, one of the plurality of items as a sponsored item to be displayed as a response to the query.
Saha teaches that
receiving a query for the classification model; identifying, by the classification model, a plurality of items based on the query; ([0006] of Saha states “The at least one processor is configured to read the instructions to: obtain, from a computing device, a search request identifying a query and seeking items to be displayed on a webpage of a website to a user; search, based on the query, a first database to retrieve a first set of sponsored items associated with the website,… search, based on the query, a second database to retrieve a second set of sponsored items associated with the website,” [0133] of Saha states “FIG. 7 illustrates a process 700 for classifying sponsored items based on their relevancy to a query, in accordance with some embodiments of the present teaching. In some embodiments, the process 700 may be carried out by one or more computing devices, such as the item recommendation computing device 102 and/or the cloud-based engine 121 of FIG. 1 . In some embodiments, the process 700 may be performed as part of the operations 641, 642, 643 and/or 680 in FIG. 6 .”)
and selecting, based on the evaluation labels, one of the plurality of items as a sponsored item to be displayed as a response to the query. ([0027] of Saha states “In some embodiments, an item recommendation system determines whether a sponsored item is eligible to be recommended or not, in response to the query, based on a relevance score… The sponsored item is determined to be ineligible to be recommended when the relevance score is not beyond the threshold. Only the eligible items may be displayed, after some filtering and rankings, on a search results webpage in response to the query.” Saha teaches selecting sponsored items for display based on a relevance score. A POSITA would use the LLM generated evaluation labels (relevance classifications for each query item pair) as both the evaluation label and the relevance score serve the same function.)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Smith, Bansal, and Saha. Smith teaches using a large language model to generate labels for datapoints and using the generated labels as training labels for a downstream machine learning model. Bansal teaches forming an augmented dataset of input samples paired with LLM generated annotations and finetuning a base model on that augmented dataset. Saha teaches receiving a search query, retrieving sponsored items based on the query, determining item eligibility based on relevance, and displaying eligible sponsored items in response to the query. One with the ordinary skill in the art would have been motivated to incorporate the teachings of Saha into the combination of Smith and Bansal to assist in determining which retrieved sponsored items are relevant and eligible for display. The combination would have been predictable as both systems concern query item relevance evaluation and using LLM generated labels as relevance signals would have been a straightforward application of labeling techniques to sponsored item selection.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BYUNGKWON HAN whose telephone number is (571)272-5294. The examiner can normally be reached M-F: 9:00AM-6PM PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at (571)272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BYUNGKWON HAN/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights