Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 19 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 19 recites the limitation “The Graphic User Interface (GUI) system of claim 15, further comprising: an interface for applying the provided or updated one or more prompts to a set of data that the LLM has not seen in lines 1-2.” There is insufficient antecedent basis for this limitation in the claim as the GUI system is only introduced in claim 16 whilst claim 15 corresponds to the method of claim 1.
For compact examination purposes, the examiner interpreted claim 19 as being dependent on claim 16 instead of 15. The examiner suggests changing the language to state: “The Graphic User Interface (GUI) system of claim 16, further comprising: an interface for applying the provided or updated one or more prompts to a set of data that the LLM has not seen in lines 1-2.”
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-31 rejected under 35 U.S.C. 101 because the claimed invention is an mental process without significantly more.
Independent claims 1 and 16 recite a method and a GUI system that uses a processor that, as drafted under its broadest representation (BRI), covers providing a set of data (a person can provide a set of data written down on paper); instructing a large language model (LLM) to review the set of data for a predefined task with or without one or more prompts by a user (a person can prompt a generic medium such as ChatGPT to review the set of data for a predefined task); producing performance metrics comparing results provided by the LLM to a set of initial human-supplied ground truth data generated by the user (A person can take the results provided by ChatGPT to compare it to their own truth data they have written down); updating or providing one or more prompts for the LLM by the user based on the performance metrics in a case that the performance metrics does not reach a threshold (A person can compare the performance metrics either mentally or on paper to see if it crosses a threshold they have determined and update/reprompt ChatGPT if it does not reach threshold); generating one or more renewed results by the LLM according to the updated or provided one or more prompts and producing performance metrics of the one or more renewed results such that the performance metrics reach a threshold (A person can take results from ChatGPT from their updated prompts and compare them to their own performance metrics); and reviewing and submitting the set of initial human-supplied ground truth data and the results of the LLM for training the discriminative machine learning model (A human can review and input a set of their own truth data and the results off of ChatGPT to potentially use for training a machine learning model).
As described above these limitations can be carried out as a series of mental steps. The judicial exception is not integrated into a practical application because the only additional elements recited are a GUI system comprising a processor which is general purpose hardware being used as a tool to implement the mental process.
The remaining dependent claims fail to add patent eligible subject matter to independent claim 1:
Claims 2, 17 simply add the step of identifying a dataset to be labeled which a human can perform mentally or with a pen and paper
Claims 3, 18 simply adds raw text to data which a human can do with a pen and paper or using a medium such as ChatGPT
Claims 4, 19 simply involves adding new unseen data to a generic medium such as ChatGPT which a human can do
Claims 5, 20 simply add display of data for human review which a human can do via ChatGPT prompting or with a pen and paper
Claims 6, 21 simply add well-known mathematical algorithms (transformers) to an already abstract process with nothing more
Claims 7, 22 simply add well-known mathematical algorithms (transformers) to an already abstract process with nothing more
Claims 8, 23 simply add well-known mathematical algorithms (transformers) to an already abstract process with nothing more
Claims 9, 24 simply adds calculating performance metrics using certain sub metrics which a human could do via pen and paper
Claims 10, 25 simply adds storing a trained model which is a generic computer function (data storage) that doesn’t transform the abstract idea into a patent-eligible invention.
Claims 11, 26 simply adds user defined requirements and iterative prompt updating and a human can give instructions and refine their requests verbally or by pen and paper
Claims 12, 27 simply adds user defined requirements and iterative prompt updating and a human can give instructions and refine their requests verbally or by pen and paper
Claims 13, 28 simply adds user defined requirements and iterative prompt updating and a human can give instructions and refine their requests verbally or by pen and paper
Claims 14, 29 simply adds user defined requirements and iterative prompt updating and a human can give instructions and refine their requests verbally or by pen and paper
Claim 15 simply adds the productions of metadata indication a reasoning which a human can perform by writing notes on their own logic with a pen and paper
Claim 30 simply adds the display of results and performance metrics which is a routine presentation of data and mathematical results that a human can do given a GUI is a generic component
Claim 31 simply adds a GUI input configuration, which is simply a design choice that can be mentally conceptualized as using one or two different blank areas for information gathering
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-6, 8-21, 23-31 are rejected under 35 U.S.C. 103 as being unpatentable over Callegari et al. (hereinafter Callegari) (US 20240362422 A1) in view of Lourentzou et al. (hereinafter Lourentzou) (US 11551437 B2) (refer to attached copy for paragraph numbers)
Regarding claim 1, Callegari teaches:
A method of data analysis for training a discriminative machine learning model (Callegari, P[0073], system is designed to analyze and assess LLM outputs iteratively and improve model behavior based on data from previous outputs, LLM’s iterative refinement is effectively learning which prompts yield better results (discriminative process), comprising:
providing a set of data (Callegari P[0073], “a test data set” is used as context for assessing LLM);
instructing a large language model (LLM) to review the set of data for a predefined task with or without one or more prompts by a user (Callegari, [0068], LLM is instructed for a predefined task (the intended output) with a user-suppled prompt. LLM is reviewing the data and implied as a part generating output from the input data).
updating or providing one or more prompts for the LLM by the user based on the performance metrics in a case that the performance metrics does not reach a threshold (Callegari, P[0070]: Discloses revising the prompt based on assessment (performance metrics). Threshold is implied by the iterative assessment/revision loop);
generating one or more renewed results by the LLM according to the updated or provided one or more prompts and producing performance metrics of the one or more renewed results such that the performance metrics reach a threshold (Callegari, P[0072], Final response after revisions maps to generating renewed results. Meeting assessment criteria corresponds to performance metrics reaching a threshold and “generating a response to the revised prompt” reads on according to the updated or provided one or more prompts); and
Callegari does not teach:
producing performance metrics comparing results provided by the LLM to a set of initial human-supplied ground truth data generated by the user;
reviewing and submitting the set of initial human-supplied ground truth data and the results of the LLM for training the discriminative machine learning model
However, Lourentzou teaches: producing performance metrics comparing results provided by the LLM to a set of initial human-supplied ground truth data generated by the user (Lourentzou, P[40]: “For each annotated data in the data set N.sub.Total, the calibration model evaluates the individual annotations and attaches a confidence score to each of the individual annotations” (teaches evaluating model output against reference data which reads on generating performance metrics comparing LLM results to human-supplied ground truth.));
reviewing and submitting the set of initial human-supplied ground truth data and the results of the LLM for training the discriminative machine learning model (Lourentzou, P[39]: "the calibration model selects two subsets of annotated data from the ML set, including a second subset identified as set HA from which machines annotations having a confidence score of 1 are assigned (320)." (reads on reviewing data and grouping them into subsets based on confidence score) “the data that populates the set AL and the SME annotated data in HA are merged and form a new set, L” (teaches combining machine-labeled and human-labeled data for subsequent model training)).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to have modified Callegari in view of Lourentzou. Doing so would have provided the recursive prompt-refinement methods of Callegari (Callegari, Abstract) with the active learning pipeline of Lourentzou (Lourentzou, Abstract). Doing so would have increased the precision of automatic data annotation while reducing the computational cost of human interventions by using the LLM to self-correct.
Regarding claim 2, The combination of Lourentzou and Callegari teach the method of claim 1.
Lourentzou further teaches:further comprising:
analyzing a set of predetermined amount of data by the user to generate the set of initial human-supplied ground truth data (Lourentzou, P[14]: “Active learning may begin with a small quantity of labelled examples to select an additional set of examples for which labeling is requested, learn from the request, and then use knowledge to select additional examples for labeling. Accordingly active learning selects a relatively small but informative sample of data to annotate to produce a highly performant model.” (small quantitity of labelled examples corresponds to the claimed subset of data analyzed by the user to establish the initial ground truth for the iterative loop)).
Regarding claim 3, The combination of Lourentzou and Callegari teach the method of claim 1.
Lourentzou further teaches:
wherein the set of data comprises one or more documents comprising raw text (Lourentzou, P[21]: “the NER Model, receives the unlabeled data set (166) and annotates the data therein” (the model processes raw textual data, satisfying “documents comprising raw text”)).
Regarding claim 4, The combination of Lourentzou and Callegari teach the method of claim 1.
Callegari further teaches:
applying the provided or updated one or more prompts to a set of data that the LLM has not seen (Callegari, P[0066]: “the revised prompt 69 may be used for larger projects to more efficiently generate a higher quality result. For example, if a website hosting the online article about the giant panda hosted a large repository of other articles and wished to provide a summary aimed at kids for each article, before having the LLM 26 generate all of the summaries at once, it would be prudent to ensure that the prompt used globally was thoroughly tested and would generate an acceptable response, rather than relying on the expertise of the user drafting the initial prompt 30 to do well on the first try.” (Testing a prompt on a small set then applying that refined prompt “globally” to a “large repository” (the second, unseen set of data)).
Regarding claim 5, The combination of Lourentzou and Callegari teach the method of claim 1.
Lourentzou further teaches:
The method of claim 1, wherein the predefined task comprises labeling, instant scaling, collection, classification or annotation (Lourentzou, P[3]: “The ML manager functions to apply an unlabeled data set to two or more neural models to automatically attach machine annotations”. Lourentzou, P[14]: “Specifically, active learning is a framework in which the learner has the freedom to select which data points are added to its training set.” (annotation and labeling are explicitly identified as predefined tasks on this system.))
Regarding claim 6, The combination of Lourentzou and Callegari teach the method of claim 6.
Lourentzou further teaches:
wherein the discriminative machine learning model (Lourentzou, P[20]: “and in one embodiment, different neural models may be employed or substituted in placed of these models. The NER model is a long-short term memory-convolutional neural network-conditional random field (LSTM-CNN-CRF) combination. “ (reads on discriminative models as they “extract entities” which is discriminative while mentioning the ability to substitute different neural models in other embodiments.))
Lourentzou does not teach comprises a small, for-purpose, fine-tuned transformer.
comprises a small, for-purpose, fine-tuned transformer.
Callegari further teaches:
comprises a small, for-purpose, fine-tuned transformer (Callegari, P[0021]: “ In order to efficiently utilize resources, the first LLM 46 may be a legacy model or a less computationally intensive model that requires fewer resources to run, and may be more limited in its capabilities than a second trained LLM 48. More specifically, the second LLM 26 may have a larger parameter size than the first LLM 46,”, P[0002]: “LLMS have been based on the transformer architecture, which utilizes tokenization and word embeddings to represent words in an input sequence, and a self-attention mechanism that is applied to allow each token to potentially attend to each other token in the input sequence during the training of the neural network. Examples of such LLMs include generative pre-trained transformers (GPTs) such as GPT-3, GPT-4, and GPT-J, as well as BLOOM, LLAMA, and others… Some of these models are fine-tuned using human reinforced learning or one-shot or few-shot learning based on ground truth examples.” (reads on transformer architecture and the process of “fine-tuning” using ground truth).
Regarding claim 8, The combination of Lourentzou and Callegari teach the method of claim 1.
Callegari further teaches:
wherein the LLM comprises Generative Pre-trained Transformer (GPT) (Callegari, P[0002]: “Examples of such LLMs include generative pre-trained transformers (GPTs) such as GPT-3, GPT-4, and GPT-J”).
Regarding claim 9, The combination of Lourentzou and Callegari teach the method of claim 1.
Lourentzou further teaches:
wherein the performance metrics comprise F1-score, Precision, Recall, False +, False ? or True+
Lourentzou, P[17]: “Introduction of unnecessary errors into the dictionary may limit potential recall of the extraction.” (reads directly on recall), Claim 1: “a data selector to evaluate the attached machine annotations for accuracy, including a calibration model to evaluate the attached machine annotations and assign a score to each machine annotation, wherein the assigned score reflects confidence of correctness of the individual machine annotations” (errors and accuracy are the direct result of false positives, false negatives, and true positives, assigned score reads on performance metrics))
Regarding claim 10, The combination of Lourentzou and Callegari teach the method of claim 1.
Lourentzou further teaches:
further comprises enabling human quality assurance of the results generated by the LLM before submitting, and the human quality assurance comprises checking or correcting the results by the user (Lourentzou, P[44]: “supports interaction of a human-in-the loop (HumL), e.g. SME, to control direction of the dictionary expansion, such as accepting or rejecting candidate entries” (SME is a subject matter expert which provides the checking and correcting required by a QA)).
Regarding claim 11, The combination of Lourentzou and Callegari teach the method of claim 1.
Callegari further teaches:
further comprising providing one or more requirements to the LLM by the user for the data to meet or adhere to (Callegari, P[0032]-P[0033]: “the user has indicated that user-specified assessment criteria should be used.”, “the prompt generation module 74 displays an assessment criteria text input pane 168 in which the user can input the assessment criteria 64.” (user-specified assessment criteria are literal requirements used to guide LLM prompting/evaluation process)).
Regarding claim 12, The combination of Lourentzou and Callegari teach the method of claim 11.
Callegari further teaches:
wherein the one or more requirements are provided to the LLM prior to or subsequent to the user providing or updating the one or more prompts to the LLM (Callegari, P[0018], criteria (requirements) are used to generate an assessment report, which is then used to revise the prompt. These requirements are provided and applied subsequently to the first response/prompt to evaluate performance).
Regarding claim 13, The combination of Lourentzou and Callegari teach the method of claim 11.
Callegari further teaches:
wherein the one or more requirements comprising predefined criteria or format for the data to meet or adhere to (Callegari, P[0033]: “a set of six assessment criteria may be used including conciseness, appropriate for audience, sufficient detail, provides citations, readability, and requested style.” (requested style and provides citations are predetermined formats/criteria)).
Regarding claim 14, The combination of Lourentzou and Callegari teach the method of claim 12.
Callegari further teaches:
wherein the one or more requirements are provided to the LLM subsequent to the user providing or updating the one or more prompts to the LLM (Callegari, P[0018], criteria (requirements) are used to generate an assessment report, which is then used to revise the prompt. These requirements are provided and applied subsequently to the first response/prompt to evaluate performance).
Regarding claim 15, The combination of Lourentzou and Callegari teach the method of claim 14.
Callegari further teaches:
further comprising generating one or more results by the LLM according to the provided one or more requirements, and producing performance metrics of the one or more results (Callegari, P[0033]: “The assessment report 34 may include numeric scores computed for each of the assessment criteria 64 on a scale from 1-10, as well as natural language (textual) description of the reasons for the score for each of the assessment criteria 64.” (the textual description of the reasons mentioned is a direct match for the claimed metadata indicating a reasoning for the LLM’s results)).
Regarding claim 16, claim 16 is directed to a graphical user interface system that corresponds to and performs substantially the same steps as the method of claim 1, implemented via conventional GUI components. Therefore, claim 16 is rejected for the same reasons as claim 1. (Refer to Callegari, Fig. 7 shows multiple interfaces and display)
Regarding claim 17, claim 17 recites the GUI system corresponding to the method presented in claim 2 and is rejected under the same grounds as above.
Regarding claim 18, claim 18 recites the GUI system corresponding to the method presented in claim 3 and is rejected under the same grounds as above.
Regarding claim 19, claim 19 recites the GUI system corresponding to the method presented in claim 4 and is rejected under the same grounds as above.
Regarding claim 20, claim 20 recites the GUI system corresponding to the method presented in claim 5 and is rejected under the same grounds as above.
Regarding claim 21, claim 21 recites the GUI system corresponding to the method presented in claim 6 and is rejected under the same grounds as above.
Regarding claim 23, claim 23 recites the GUI system corresponding to the method presented in claim 8 and is rejected under the same grounds as above.
Regarding claim 24, claim 24 recites the GUI system corresponding to the method presented in claim 9 and is rejected under the same grounds as above.
Regarding claim 25, claim 25 recites the GUI system corresponding to the method presented in claim 10 and is rejected under the same grounds as above.
Regarding claim 26, claim 26 recites the GUI system corresponding to the method presented in claim 11 and is rejected under the same grounds as above.
Regarding claim 27, claim 27 recites the GUI system corresponding to the method presented in claim 12 and is rejected under the same grounds as above.
Regarding claim 28, claim 28 recites the GUI system corresponding to the method presented in claim 13 and is rejected under the same grounds as above.
Regarding claim 29, claim 29 recites the GUI system corresponding to the method presented in claim 14 and is rejected under the same grounds as above.
Regarding claim 30, the combination of Callegari and Lourentzou teaches the GUI of claim 29.
Callegari further teaches:
wherein the display is further configured to display results generated by the LLM according to the provided one or more requirements, and the performance metrics of the one or more renewed results (Callegari, P[0034]: “the assessment report 34 is displayed to the user, including assessment report text 174, along with a gating control, which asks the user if the user would like to generate a revised response. The assessment report 34 may include numeric scores computed for each of the assessment criteria 64 on a scale from 1-10, as well as a natural language (textual) description of the reasons for the score for each of the assessment criteria 64” (final response 56 reads on LLM results being displayed in the GUI. The results are generated based on the “assessment criteria” (requirements) input by the user earlier in the process. The numeric scores (metrics) within the assessment report are displayed alongside the results to show improvement).
Regarding claim 31, Lourentzou teaches the GUI of claim 26.
Lourentzou further teaches:
wherein the input for providing the one or more requirements is configured to be the same as or different from the input for providing or updating the one or more prompts (Callegari, Fig. 7, P[0031]: “the prompt interface 24 of the GUI 28, the user has entered the prompt 30”, P[0033]: “the assessment and revision engine 76 of the prompt generation module 74 displays an assessment criteria text input pane 168 in which the user can input the assessment criteria 64.” (The “assessment criteria text input pane 168” reads on input for providing the one or more requirements. The “prompt interface 24” is the input for the prompt. Fig 7. shows both inputs)).
Claim(s) 7, 22 are rejected under 35 U.S.C. 103 as being unpatentable over Callegari et al. (hereinafter Callegari) (US 20240362422 A1) in view of Lourentzou et al. (hereinafter Lourentzou) (US 11551437 B2) in further view of Gashteovski et al. (hereinafter Gashteovski) (US 20240296285 A1)
Regarding claim 7, The combination of Lourentzou and Callegari teach the method of claim 1.
Lourentzou further teaches:
wherein the discriminative machine learning model comprises a Robustly Optimized BERT Pretraining Approach (RoBERTa) model, a Decoding-enhanced BERT with disentangled attention (DeBERTa) model, or Longformer (Gashteovski, P[0046]: “Therefore, training standard discriminative (classification) models is prohibitive.”, P[0048]: “The textual fact embedding model can be for example a pretrained language model (e.g., a pretrained natural language processing (NLP) model), such as RoBERTa” (reads on a discriminative machine learning model potentially comprising RoBERTa)).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to have modified Callegari in view of Lourentzou and in further view of Gashteovski. Doing so would have provided the pretraining language models from Gashteovski (Gashteovski, P[46]) with the recursive prompt-refinement methods of Callegari (Callegari, Abstract) with the active learning pipeline of Lourentzou (Lourentzou, Abstract). Doing so would have increased the precision of automatic data annotation while reducing the computational cost of human interventions by using the LLM to self-correct and improving computational efficiency via RoBERTa.
Regarding claim 22, claim 22 recites the GUI system corresponding to the method presented in claim 7 and is rejected under the same grounds as above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHASHIDHAR S MANOHARAN whose telephone number is (571)272-6772. The examiner can normally be reached M-F 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at 571-272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHASHIDHAR SHANKAR MANOHARAN/ Examiner, Art Unit 2655
/ANDREW C FLANDERS/ Supervisory Patent Examiner, Art Unit 2655