Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claims 1-20 are pending. Claims 1, 8, and 15 are independent.
This Application was published as U.S. 20250238600.
Apparent priority: 24 January 2024.
Claims are not rejected by art. Claims 1-7 and 15-20 are outright allowed and Claims 8-14 are objected to and rejected for indefiniteness.
Claim Objections
Claim 8 is objected to because of informalities that may be addressed with the following suggested amendments:
8. A system, comprising:
a database comprising a dataset comprising a plurality of label names and a label description for each label name;
a memory storing a large language model (LLM); and
an electronic device executing a computer program to:
receive the dataset from the database;
prompt the LLM for initial annotation guidelines with the dataset;
receive, from the LLM, the initial annotation guidelines;
receive an unlabeled dataset comprising unlabeled text samples;
predict, a label name for each unlabeled text sample;
present each unlabeled text sample and its predicted label name for review; group incorrectly labeled text samples and their predicted label names;
prompt the LLM for a recommendation to improve the initial annotation guidelines with the group of the incorrectly labeled text samples and their predicted label names and the initial annotation guidelines;
receive, from the LLM, the recommendation;
prompt the LLM for improved annotation guidelines with the initial annotation guidelines and the recommendation;
receive the improved annotation guidelines; and
deploy the improved annotation guidelines to a production environment.
Note the 112(b) rejection below.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 8-14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 8 is a system claim whereas “a large language model” and “ a computer program” are not system/device/machine or manufacture limitations. The Claim as is recites a single structural limitation: “database.”
Dependent Claims inherit the indefiniteness.
The following amendments are suggested:
8. A system, comprising:
a database comprising a dataset comprising a plurality of label names and a label description for each label name;
a memory storing a large language model (LLM); and
an electronic device executing a computer program to:
receive the dataset from the database;
prompt the LLM for initial annotation guidelines with the dataset; receive, from the LLM, the initial annotation guidelines;
receive an unlabeled dataset comprising unlabeled text samples;
predict, a label name for each unlabeled text sample;
present each unlabeled text sample and its predicted label name for review; group incorrectly labeled text samples and their predicted label names;
prompt the LLM for a recommendation to improve the initial annotation guidelines with the group of the incorrectly labeled text samples and their predicted label names and the initial annotation guidelines;
receive, from the LLM, the recommendation;
prompt the LLM for improved annotation guidelines with the initial annotation guidelines and the recommendation;
receive the improved annotation guidelines; and
deploy the improved annotation guidelines to a production environment.
Allowable Subject Matter
The pending Claims 1-7 and 15-20 are allowed.
The following is an examiner’s statement of reasons for allowance: In view of each of the particular limitations of the independent Claims when considered in the order established by the Claim language and in the context of the language of the independent Claims when each Claim is considered as a whole, the independent Claims of this Application were not found in the prior art that was viewed.
In particular, receiving a dataset of Label Names and Label Description and providing it as an input to an LLM and prompting the LLM to generate guidelines for labeling of unlabeled data with the labels of the dataset, using the guidelines to label data items and then providing the labeled data back to the LLM asking for improved guidelines based on groupings of mislabeled data items when considered in the context of the independent Claims as a whole and considering each and every limitation of these claims was not found in the prior art.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Close Art of Record
Sommerfield (U.S. 20250139380), Zhang (U.S. 20250200356), and Pauli (U.S. 20240338532) are applied to the Claims as shown below. The mapping reveals the shortcomings of the references with respect to teaching the limitations of Claim 1 when the Claim is considered as a whole.
Regarding Claim 1, Sommerfield teaches:
1. A method, comprising: [Sommerfield, Figure 14 shows the hardware including Processing System 1404 and Computer-Readable Storage media 1406 including Instructions, Data, Settings, etc. 1408.]
receiving, by a computer program [Sommerfield, Figure 1, “computing system 102” which includes the “labeling control system 114” and the “language model 104.”] executed by a computer processor, a dataset comprising a plurality of label names and a label description for each label name; [Sommerfield, Figure 1, item 120 and label information 122 are received but do not teach the “dataset” of the Claim which is asking for Label Name and Label Description and not in association with the items being labeled/classified, “[0030] In some contexts, a developer uses the computing system 102 to apply labels to items in a data store 106 of items. A data store 108 stores the labeled items….” The examples shown in Figure 4 or the “Hints” in Figure 5 can teach the “label description” of the Claim. “[0058] FIG. 4 shows an overview of an illustrative task description 402. The task description 402 includes text having various sections. An optional first section 404 specifies a group of predefined classes 406, generically designated in FIG. 4 as class A, class B, class C, etc. The list of classes 406 also has an entry 408 that generically represents any new class that the language model 104 may detect during its operation. A second section 410 provides an explanation of the task that the language model 104 is being requested to perform. In some cases, the explanation also explicitly requests the language model 104 to set forth the “reasons” that it has selected a particular task. For example, the explanation provides a general directive to “give a detailed response that sets forth the reason for the classification.” A third section 412 optionally provides examples that demonstrate how the task is correctly performed. For example, a representative example specifies an input prompt and a correct classification of the input prompt. A fifth section 414 specifies the format that the language model 104 is requested to provide its conclusion. Other task descriptions include additional types of information, and/or omit one or more of the types of information shown in FIG. 4, and/or organize the types of information in a different manner than what is shown in FIG. 4.”]
prompting, by the computer program, a large language model (LLM) for initial annotation guidelines [Sommerfield, Figure 1, “task description 118.”] with the dataset;
receiving, by the computer program and from the LLM, the initial annotation guidelines; [Sommerfield, Figure 1, the “initial annotation guidelines” are taught by the “task description 118” and are an input to the system and are not generated by the LLM. “[0032] A labeling control system 114 generates a prompt 116 to be sent to the language model 104. The prompt 116 includes a task description 118 and a particular item 120, selected from the items in the data store 106. The task description 118 instructs the language model 104 how it is to process the particular item 120….” “[0034] As another function, the labeling control system 102 dynamically and automatically adjusts the task description 118….”]
receiving, by the computer program, an unlabeled dataset comprising unlabeled text samples; [Sommerfield, “[0032] A labeling control system 114 generates a prompt 116 to be sent to the language model 104. The prompt 116 includes a task description 118 and a particular item 120, selected from the items in the data store 106….” “Data store 106” has the “dataset comprising unlabeled text samples” of the Claim. “[0001] There exists a need to label large quantities of items for various downstream uses. As used herein, “items” include text-bearing items, image-bearing items, video-bearing items, etc., or any combination thereof….”]
predicting, by the computer program, a label name for each unlabeled text sample; [Sommerfield, Figure 1, the “language model 104” generates the “label information122.” “[0027] FIG. 1 shows a computing system 102 for automatically labeling items using a machine-trained language model (“language model”) 104. An “item” refers to a unit of information containing any content type or types. For example, an item includes a passage of text of any length, an image, a video, or any combination thereof.” “[0033] The language model 104 transforms the prompt 116 into label information 122, which expresses the class selected by the language model 104….”]
presenting, by the computer program, each unlabeled text sample and its predicted label name for review; [Sommerfield, Figure 2 shows the “Review 208” panel which permits the human user to edit 248 his instructions. Figure 3, 316 and 318. “[0045] The user interface presentation 202 includes three mains sections. A first section 204 allows a user to set up a labeling run. A second section 206 allows the user to start and stop the labeling run. A third section 208 allows the user to review the label information specified by the language model 104 for the images classified so far in the labeling run. When paused, the user also remains free to change any aspects of the labeling run via the first section 204….”]
grouping, by the computer program, incorrectly labeled text samples and their predicted label names; [Sommerfield teaches that confidence in that a new class should be added and accuracy of the labels is presented. It does not teach grouping of the low accuracy/incorrectly labeled items: “[0052] A summary subsection 250 presents information that summarizes the results produced by the language model 104, at the current juncture in a labeling run. For example, a first presentation 252 shows a count (or other summary information) of the number of times that the language model 104 has detected each class in the list of predefined classes. A second presentation 254 shows a count (or other summary information) of the number of times that the language model 104 has detected a particular new disease (e.g., downy mildew), not currently represented in the predefined classes. Each number in parentheses (e.g., 74% for downy mildew) represents a current level of confidence that a new class should be added to the predefined list of classes. In some implementations, this level of confidence is expressed as a ratio of the number of times that the language model 104 has chosen the new class to a number of times that are required to add the new class to the list of predefined classes. A third presentation 256 assesses the accuracy of the label information produced thus far, e.g., as assessed by one or more human users and/or an authoritative machine-trained model (including a language model that operates in an evaluation mode)…. In some implementations, the summary information explicitly allows a user to compare the classification behavior and/or accuracy produced for different task descriptions, and/or to view trends in classification behavior and/or accuracy across plural changes to the class description….”]
prompting, by the computer program, the LLM for a recommendation to improve the initial annotation guidelines with the group of the incorrectly labeled text samples and their predicted label names and the initial annotation guidelines; [Sommerfield, Figures 2 and 3 shows how the user can change the “annotation guidelines” based on the results generated for the first run of the classification. Here is not the LLM that recommends a modified guideline. It is the user that can change a task description that generated an inaccurate label. Figure 3 shows the editing process.]
receiving, by the computer program and from the LLM, the recommendation; [Sommerfield, Figure 3, 318. “[0054] In block 314, the labeling control system 114 detects whether the user has paused the labeling run, e.g., by activating the graphical control 226. If so, then in block 316, the labeling control system 114 detects whether the user has chosen to edit the setup information entered via the section 204 of FIG. 4. If so, in block 318, the labeling control system 114 receives the user's edits….”]
prompting, by the computer program, the LLM for improved annotation guidelines with the initial annotation guidelines and the recommendation; [Sommerfield, this process is taught by the reference. However, the “recommendation” part is not by LLM, and the improved guideline/task description is generated by the “labeling control system 114”: “[0034] As another function, the labeling control system 102 dynamically and automatically adjusts the task description 118. For example, assume that the task description 118 instructs the language model 104 to choose a class from a group of predefined classes. Assume that the task description 118 also instructs the language model 104 to assign a new class upon determining that none of the group of classes applies to the particular item 120. Here, the labeling control system 114 determines whether a predetermined number of responses (associated with a plurality of items) support the conclusion that a new class should be added to the current group of classes (because the responses specify the new class), and/or a predetermined percentage of responses classified so far support the conclusion that the new class should be added (because the responses specify the new class with a threshold normalized frequency). If so, the labeling control system 114 automatically updates the predefined group of classes in the task description 118 to include the new class.”]
receiving, by the computer program, the improved annotation guidelines; and [Sommerfield, Figure 1, the LLM 104 receives the improved instructions/annotation guidelines from the “labeling control system 114.”]
deploying, by the computer program, the improved annotation guidelines to a production environment. [Sommerfield, Figure 1, the “Language Model 104” uses the modified instruction/task description 118 to generate “label information 122.”]
PNG
media_image1.png
545
441
media_image1.png
Greyscale
Sommerfield performs the same operations of Claim 1 except not by an LLM. The “task description 118” / “initial annotation guidelines” and later recommendations are input by a user person and not generated by an LLM.
Also, there is no “grouping … incorrectly labeled text samples” in Sommerfield to lead to an improved annotation guidelines generated by the LLM.
Zhang teaches:
prompting, by the computer program, a large language model (LLM) for initial annotation guidelines with the dataset; [Zhang, Figure 4: “Instruction prompt 400.” Figure 3, 310. “Instruction prompt to generate evaluation labels” is the closest Zhang comes the “prompting … LLM for initial annotation guidelines” of the Claim: “[0071] In some embodiments, the online system provides 310 an instruction prompt to a machine-learned language model. The machine-learned language model can be any language model provided by the model serving system 150. The instruction prompt provides an instruction for the machine-learned language model to generate evaluation labels for the outputs of a classification model….” But this is not prompting the LLM to generate guidelines for labeling; rather it is prompting the LLM to do the labeling.]
receiving, by the computer program and from the LLM, the initial annotation guidelines; [Zhang, Figure 3, 330. “evaluation labels” and “reasoning of the evaluation label” received back from the LLM cannot teach the “initial annotation guidelines” of the Claim: “[0083] In some embodiments, the online system receives 330 a plurality of responses from the machine-learned language model. In response to receiving the batch of evaluation request prompts, the machine-learned language model provides the responses. Each response includes the evaluation label corresponding to the data in an evaluation request prompt. The response may include other things, such as reasoning of the evaluation label, metadata, and other relevant information that is requested in the instruction prompt. The plurality of responses may be sent as a plurality of messages or one or more combined messages from the machine-learned language model.”]
…
PNG
media_image2.png
473
425
media_image2.png
Greyscale
PNG
media_image3.png
671
449
media_image3.png
Greyscale
Regarding Claim 2, Sommerfield teaches the transformer based language model: “[0084] FIG. 9 shows a transformer-based language model (“language model”) 902 for implementing the language model 104 referenced by FIG. 1. The language model 902 is composed, in part, of a pipeline of transformer components, including a first transformer component 904….” But does not each the steps of the Claim.
Pauli teaches:
2. The method of claim 1, wherein the step of predicting, by the computer program and using an annotation agent, the label name for each unlabeled text sample comprises:
obtaining, by the annotation agent and using a sentence transformer, a semantic embedding for each unlabeled sample text and for the label names; [Pauli, “[0030] … The LLM 120 is also used to generate semantic embeddings 124 for the samples of the dataset 104, where the embeddings 124 are then used to train 112 the student model 130. … The LLM 120 may also be used to generate soft labels 126 for some samples, where soft labels 126 represent automatically-generated initial categorization guesses for those samples that may be used to train 112 the student model 130.” “[0063] FIG. 6 illustrates an example flow 600 of data within the architecture 100 of FIG. 1. An embedding model (e.g., the LLM 120) generates embeddings 124 for the dataset 104….” “[0038] … The LLM 120 is used to create soft labels 126 for training the student model 130 that can eventually transform the entire dataset 104 with high throughput and without necessarily requiring specialized hardware (GPU)….”]
ranking, by the annotation agent and for each unlabeled sample text, the label names based on a similarity between the semantic embeddings; and [Pauli, Figure 5, for “sample to be labeled 510” the x nearest labeled samples 516 are selected. The quantification of the distance is a natural ranking. The distances will be different and some will be closer. Figure 7, “[0065] … The distances between the data points in the t-SNE plot reflect the similarities between them in the high-dimensional space, with closer points indicating higher similarities….”]
identifying, by the annotation agent, one of the label names for each unlabeled sample text as the predicted label name. [Pauli, Figure 5, for “sample to be labeled 510” the x nearest labeled samples 516 are selected.]
PNG
media_image4.png
292
459
media_image4.png
Greyscale
Regarding Claim 3, Pauli teaches:
3. The method of claim 2, wherein the similarity comprises a cosine similarity.[Pauli, Figure 5, 516. “[0062] FIG. 5 illustrates an annotation flow 500 for applying labels to samples of the dataset 104 by the AI assistant 110. At operation 512, a sample 510 to be labeled is input to the student model 130 to produce class membership probabilities for each of the defined classes. At operation 516, the AI assistant selects one or more nearest labeled samples to the sample to be labeled (or just “sample”) 510 (e.g., based on cosine similarity to labeled samples 514)….”]
PNG
media_image5.png
439
865
media_image5.png
Greyscale
Regarding Claim 4, Sommerfield teaches:
4. The method of claim 2, wherein a number of ranked label names is limited. [Sommerfield, Figure 4 shows a limited number of classes/labels and the description also provides that certain classes/labels can be added if they appear in the classification over a certain percentage of time. See [0034].]
PNG
media_image6.png
456
398
media_image6.png
Greyscale
Regarding Claim 5, Sommerfield teaches:
5. The method of claim 1, wherein each unlabeled text sample and its predicted label name are presented for review to a human. [Sommerfield, Figure 2 shows the user interface with 3 panels where in the Review 208 panel the sample 234 and its predicted label 236 are shown to the user for editing 248.]
PNG
media_image7.png
661
463
media_image7.png
Greyscale
Regarding Claim 6, Sommerfield teaches:
6. The method of claim 1, wherein each unlabeled text sample and its predicted label name is identified as correctly labeled or incorrectly labeled. [Sommerfield, the “accuracy” measure determines whether the label is correct or incorrect. Figure 3, “model performance 256.” “[0052] … A third presentation 256 assesses the accuracy of the label information produced thus far, e.g., as assessed by one or more human users and/or an authoritative machine-trained model (including a language model that operates in an evaluation mode)….”]
Regarding Claim 7, Sommerfield teaches:
7. The method of claim 1, wherein multiple versions of the annotation guidelines are used in the production environment to predict label names, and a label model combines the predicted label names and outputs a final predicted label name. [Sommerfield, Figure 6 shows a number of different Descriptions 606, 620, 622 each of which is an “annotation guideline” that are used to generate the predicted label and the responses are processed at 626 and combined to generate a label. Figure 12 shows the flowchart of the same process. “Determine a final class for the particular item based on the particular classes generated by the plural language models 1208” when the process begins with “send plural task descriptions to plural language models, together with a particular item of a set of items, … 1204.”]
PNG
media_image8.png
394
426
media_image8.png
Greyscale
PNG
media_image9.png
645
462
media_image9.png
Greyscale
Claim 8 is a system claim with limitations corresponding to the limitations of Claim 1.
Claim 9 is a system claim with limitations corresponding to the limitations of Claim 2.
Claim 10 is a system claim with limitations corresponding to the limitations of Claim 3.
Claim 11 is a system claim with limitations corresponding to the limitations of Claim 4.
Claim 12 is a system claim with limitations corresponding to the limitations of Claim 5.
Claim 13 is a system claim with limitations corresponding to the limitations of Claim 6.
Claim 14 is a system claim with limitations corresponding to the limitations of Claim 7.
Claim 15 is a computer program product system claim with limitations corresponding to the limitations of method Claim 1.
Claim 16 is a computer program product system claim with limitations corresponding to the limitations of method Claim 2.
Claim 17 is a computer program product system claim with limitations corresponding to the limitations of method Claim 3.
Claim 18 is a computer program product system claim with limitations corresponding to the limitations of method Claim 4.
Claim 19 is a computer program product system claim with limitations corresponding to the limitations of method Claim 6.
Claim 20 is a computer program product system claim with limitations corresponding to the limitations of method Claim 7.
Li (U.S. 20250139446):
PNG
media_image10.png
534
353
media_image10.png
Greyscale
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to FARIBA SIRJANI whose telephone number is (571)270-1499. The examiner can normally be reached 9 to 5, M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre Desir can be reached at 571-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Fariba Sirjani/
Primary Examiner, Art Unit 2659
Prosecution Insights