Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The abstract of the disclosure is objected to because the claim language is in legalese form without specifying the improvement in the art. The abstract states “embodiments include…”, repetitively, stating the claim scope. A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-21 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Chintagunta et al (20240029714).
As per claim 1, Chintagunta et al (20240029714) teaches a method for autonomous orchestration of a data labeling workflow, comprising:
generating a prediction based on input data using a classification machine learning model (para [0032], "A conversational dialogue between a medical professional and a patient has fundamental medical- related discourses. The discourse units include (a) gathering the patient's history of present illness (labeled "History taking"), (b) confirming captured patient symptomatology (labeled "Summarization"), (c) educating the patient (labeled "Education") and (d) communicating a treatment plan (labeled "Care plan"). There are also other non-medical discourses, such as the expression of empathy and discussion of technical difficulties. Dialog docs not explicitly codify this structure. FIG. 2 shows an abridged dialogue 200 in the sentences are labeled as predicted by the model introduced. Note that both the patient and the medical
professional may use language with high lexical overlap during each part of the dialogue 200. ");
performing, based on a confidence score associated with the prediction, a generative artificial intelligence process (para [0075], "Table 2 provides main results, comparing F1 and accuracy scores from each training round of the sentence-level model. The overall performance increased from an accuracy of 69.5% to 82.5%. This three-fold improvement of the F1 scores shows that the example iterative approach can improve labeling quality (and hence the model) even when the initial labels are quite noisy. The "Summarization" class has the most improvement (F1 score from 0.18 to 0.65). The sentences in this class are hard to identify solely from the turn-level-model-based pseudo-labeling because they vary in structure. The pseudolabeler successfully labels the sentences that contain "to summarize." The iterative clustering- based labeling introduces less-confident predictions that are semantically similar to more confident ones to improve the overall identifiability."), comprising:
generating, using a generative machine learning model, a natural language description of the input data (para [0076], "To understand the rationale for these metrics, FIG. 5 provides a graphical representation of the errors the model makes. Each column represents the human-assigned true label in this figure, and each row represents the proportion of the predicted labels in each true label for each training round. The two sections that see the significant F1 score uplift, "History taking" and "Summarization," start with a significant confusion with the "Other" section, which gradually decreases. Even though the additional iterations did not improve the care plan and education classes, their overall confusion changed between rounds. Initially, both "Education" and "Care plan" were confused with the "Other" section, while in later rounds, they were confused with each other. ");
selecting a set of candidate labels based on the natural language description of the input data (para [0131], "Example embodiments provide a medically-aware the summarization component 130, e.g., a Machine Learning (ML) model data labeler, GPT-3-ENS, that combines medical knowledge and an ensemble of GPT-3 for the purpose of medical dialogue summarization. While GPT-3 is used in an example, other machine learning models (large language models) may be used.");
providing natural language descriptions of the set of candidate labels and the natural language description of the input data to a language processing machine learning model (para [0134], "One of the main challenges in using deep learning for healthcare is the lack of large annotated datasets. It is usually costly and time-consuming to collect a large labeled dataset because annotations need to be provided by trained healthcare professionals. As deep models usually require a large amount of data to perform accurately and robustly, this deters their widespread application in healthcare. So, it is essential to develop low-shot models in healthcare i.e. models that can do well given a small number of labeled examples. In parallel, there has been a lot of progress in development of large scale models leveraging web-scale data, such as GPT-3, that show good low-shot performance. However, these models can be noisy, particularly in the medical domain, so we need approaches that mitigate this noise but arc still able to leverage these models' strengths. In this context, our approach of infusing medical knowledge in pretrained models such as GPT-3 to generate high-quality synthetic labels is an idea with wide applicability in low-resource settings like healthcare.);
and assigning a label to the input data based on the language processing machine learning model outputting the label in response to the natural language descriptions of the set of candidate labels and the natural language description of the input data (para [0135], "If pretrained models can be used to generate accurate labels, can they be directly leveraged for the task at hand? In many settings, they probably can but particularly in healthcare, this is nuanced. ML models in healthcare can learn and improve over time only if they are amenable to feedback loops i.e., they can be retrained with labels that are corrected/edited by medical practitioners. Moreover, if the model making the predictions is owned by a third party, privacy protocols (e.g. HIPAA) mandate that either they obey the same privacy protocols or that data be deidentified before being sent to such external services. Both these necessitate the need for a different approach. Accordingly, example embodiments infuse medical knowledge into an external non-HIPAA compliant model (GPT-3) and leverage it as a data generator to obtain a large training set, to then train an in-house model. Since the data exposed to GPT-3 is fixed and small (in our experiments, GPT-3 only saw 210 examples), it can be ensured to be privacy protected. Our proposed approach to develop an in-house model has two advantages (1) It can be used at inference time without the practical constraint of data de-identification and (2) It lends itself well to the aforementioned practitioner-in-the-loop setting.") and performing an action based on the assigning of the label to the input data (para [0141], "If we can generate multiple summaries from GPT-3 using a variety of priming contexts, then we should be able to ensemble these outputs to identify the summary that is ideal for the dialogue. This insight leads to a question on how to ensemble multiple text summaries. The answer to this question relies on the core requirement for medical summarization: we care about the coverage of medical concepts mentioned and therefore the best ensembling function is the one that returns the summary with the most medical information in the dialog input.").
As per claim 2, Chintagunta et al (20240029714) teaches the method of claim 1, wherein the performing of the generative artificial intelligence process is based on determining that the confidence score associated with the prediction is below a threshold (para [0122], "This is confirmed by FIGS. 10(a), (b) and (c). We observe that uncertainty drops monotonically as the a weight increases. In particular, uncertainty on early timesteps heavily drops as a result of utilization minimization. Hence, the model becomes more confident in selecting principal concepts at the beginning of an instruction. In contrast to the baseline, all regularized models' uncertainty stats to increase for t>10. As fewer concepts appear in the instruction end, the marginal probability maximization flattens the conditional distribution. However, the uncertainty does not degrade in comparison to the baseline. Thus, the proposed regularization effectively improves the confidence of the model on early timesteps.").
As per claim 3, Chintagunta et al (20240029714) teaches the method of claim 1, wherein the generating of the natural language description of the input data comprises prompting the generative machine learning model to generate the natural language description in a manner that adds context to the input data and uses human readable natural language (as combining medical knowledge and the gpt-3 machine learned models – para 0131, operating on human labeled(human natural language) datasets – para 0133, and using low-shot labeled machine learned models as the prompt for the models – para 0134-0135).
As per claim 4, Chintagunta et al (20240029714) teaches the method of claim 1, wherein the assigning of the label to the input data is further based on determining that the label matches the prediction (as combining the knowledge with the ensemble of the model – see para 0131, 0134).
As per claim 5, Chintagunta et al (20240029714) teaches the method of claim 1, wherein the assigning of the label to the input data is further based on determining that the label does not match the prediction and receiving manual confirmation of the label (as performing manual labeling of the clusters/human annotations to fine-tune/improve the matching – para 0064-0066; see also para 0029).
As per claim 6, Chintagunta et al (20240029714) teaches the method of claim 1, wherein the performing of the action comprises one or more of: updating a workflow of a computing application; populating a variable; OR displaying content via a user interface (paras [0102], "A care plan is a set of actions (instructions) that a medical professional writes in the patient's electronic health record (EHR) as a follow-up to the patient's visit. A care plan often includes a list of medications with appropriate directions, further medical evaluations, or educational information for preventive care. Before writing the care plan, the medical professional discusses it with the patient, and together, they jointly agree on the next course of action. This joint decision-making implies that most of the necessary information for writing the care plan is already available in the conversation.").
As per claim 7, Chintagunta et al (20240029714) teaches the method of claim 1, wherein the selecting of the set of candidate labels based on the natural language description of the input data comprises: generating an embedding of the natural language description (see para 0050, embeddings part of the sentence-level pseudolabels);
comparing the embedding of the natural language description to embeddings of descriptions of a plurality of labels and selecting the set of candidate labels based on the comparing, wherein the set of candidate labels contains fewer than all of the plurality of labels (as clustering, reviewing, and removing unnecessary data sets – see para 0061).
As per claim 8, Chintagunta et al (20240029714) teaches the method of claim 1, further comprising: generating an additional prediction based on additional input data using the classification machine learning model (as performing reiterative in-the-loop clustering of labels – para 0035);
determining, based on a corresponding confidence score associated with the additional prediction exceeding a threshold, not to perform a corresponding generative artificial intelligence process for the additional input data (as, if there is no improvement after a certain number of terms, stopping the training of the model – para 0113); and assigning a corresponding label to the additional data based on the additional prediction without performing the corresponding generative artificial intelligence process ( as applying the determine probability calculations for labeling – para 0111, 0112, then stopping the training/process—para 0113).
As per claim 9, Chintagunta et al (20240029714) teaches a method for autonomous orchestration of a data labeling workflow, comprising:
generating, using a generative machine learning model, a natural language description of input data (para [0032]); identifying candidate labels that are semantically similar to the natural language description of the input data (para [0075]); providing the candidate labels and the natural language description of the input data to a language processing machine learning model (para [0076]);
receiving an output from the language processing machine learning model in response to the candidate labels and the natural language description of the input data, wherein the output indicates a selected label from the candidate labels (para [0131]); validating the output based on an alternative label determination technique (para [0134]); and associating the selected label with the input data based on the validating (para [0135]).
As per claim 10, Chintagunta et al (20240029714) teaches the method of claim 9, further comprising receiving, from the language processing machine learning model, a natural language explanation of why the selected label was chosen (para [0122]).
As per claim 11, Chintagunta et al (20240029714) teaches the method of claim 9, wherein the validating comprises
determining whether the selected label matches a label determined using the alternative label determination technique or whether the selected label is within a category associated with the label determined using the alternative label determination technique (as performing in-loop-repetitive labeling – para 0029, 0035).
As per claim 12, Chintagunta et al (20240029714) teaches the method of claim 11, wherein the validating comprises providing the selected label to a user interface for manual review based on determining that the selected label does not match the label determined using the alternative label determination technique and that the selected label is not within the category associated with the label determined using the alternative label determination technique (as performing manual labeling of the clusters/human annotations to fine-tune/improve the matching – para 0064-0066; see also para 0029).
As per claim 13, Chintagunta et al (20240029714) teaches the method of claim 11, wherein the validating comprises associating an accuracy indicator with the selected label based on determining that the selected label does not match the label determined using the alternative label determination technique and that the selected label is within the category associated with the label determined using the alternative label determination technique (as performing in-loop-repetitive labeling – para 0029, 0035).
As per claim 14, Chintagunta et al (20240029714) teaches a method for autonomous orchestration of a transaction classification workflow, comprising:
determining a predicted classification for a transaction using a classification model (para [0032]);
identifying candidate classifications for the transaction from a set of classifications based on a semantic comparison of a description of the transaction with descriptions of the candidate classifications (para [0075]);
using a language processing machine learning model to determine a selected classification from the candidate classifications for the transaction (para [0076]);
and associating the selected classification with the transaction based on comparing the selected classification with the predicted classification (as performing reiterative in-the-loop clustering of labels – para 0035).
As per claim 15, Chintagunta et al (20240029714) teaches the method of claim 14, wherein the candidate classifications exclude one or more classifications from the set of classifications ((as performing manual labeling of the clusters/human annotations to fine-tune/improve the matching – para 0064-0066; see also para 0029)).
As per claim 16, Chintagunta et al (20240029714) teaches the method of claim 14, wherein the language processing machine learning model outputs the selected classification based on analyzing the descriptions of the candidate classifications and the description of the transaction ( as candidate classified description – para 0134, 0135).
As per claim 17, Chintagunta et al (20240029714) teaches the method of claim 14, further comprising generating the description of the transaction using the language processing machine learning model or a different generative machine learning model based on a prompt indicating that the description should be generated from a perspective of an expert in a particular domain ( as using expert-based dataset – para 0036, 0020; see also para 0084 – using medical expert information).
As per claim 18, Chintagunta et al (20240029714) teaches the method of claim 17, wherein the particular domain is procurement of goods or services in a particular industry (paras [0134], [0135]; wherein the models are for the domain of medical knowledge and HIPAA rules).
As per claim 19, Chintagunta et al (20240029714) teaches a method for autonomous orchestration of an item classification workflow, comprising:
identifying candidate classifications for an item from a set of classifications based on a semantic comparison of a description of the item with descriptions associated with the set of classifications, wherein the candidate classifications include a configured number of highest ranked classifications from the set of classifications based on rankings assigned from the semantic comparison (para [0075]);
using a large language model to determine a selected classification from the candidate classifications for the item (para [0076]); associating the selected classification with the item (paras [0131]); and performing an action with respect to the item based on the associating (para [0141]).
As per claim 20, Chintagunta et al (20240029714) teaches the method of claim 19, wherein the performing of the action comprises:
one or more of: displaying the item via a user interface; recommending the item to a user (as displaying, results – para 0209);
determining pricing information associated with the item; determining market information associated with the item; OR determining material information associated with the item ( as, determining medical based information, such as care plan,.summary, etc. – para 0029, and summarizing/providing such relevant information – see para 0030).
As per claim 21, Chintagunta et al (20240029714) teaches the method of Claim 19, further comprising
utilizing parallel processing or multithread processing to determine a corresponding selected classification for a different item while determining the selected classification for the item ( as using the expert databases during evaluation, and maintaining/performing the matching on different classes – because it makes the model more robust and generalized – para 0036). ).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Please see related art listed on the PTO-892 form.
Furthermore, the following references were found to have elements common to applicants spec/claim features:
Sewak et al(20220414137) teaches user label feedback between candidate text and semantic based text 9para 0030, 0032, 0034).
Parham et al (20240248963) teaches candidate label identification, label embedding with machine learned models (para 0077) interlaced with human intervention with label verification/updating – para 0078).
Shamir (20250252292) teaches posterior fine tuning of labels using human feedback (para 009, 0050).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael Opsasnick, telephone number (571)272-7623, who is available Monday-Friday, 9am-5pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Mr. Richemond Dorvil, can be reached at (571)272-7602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/Michael N Opsasnick/Primary Examiner, Art Unit 2658 09/15/2026
