Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This office action is final and in response to the amendment filed on July 07, 2026, which was in response to the non-final office action mailed May 18, 2026.
Appropriate corrections have been made to the claim objections.
Claims 1, 5-7, 10, 11, 13, 14, 16, and 18-20 have been amended.
Claims 4, 9, and 17 have been cancelled.
Claims 21-23 are newly added.
The remaining Claims 1-20 as well as Claims 21-23 are pending and have been examined. Remaining Claims 1-20 and Claims 21-23 are rejected.
Information Disclosure Statement
Acknowledgment is made of the information disclosure statements filed February 7, 2023, which comply with 37 CFR 1.97. As such, the information disclosure statements have been placed in the application file and the information referred to therein has been considered by the examiner.
Acknowledgment is made of the information disclosure statements filed July 5,
2026, which comply with 37 CFR 1.97. As such, the information disclosure statements have been placed in the application file and the information referred to therein has been considered by the examiner.
Priority
Applicant's claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged. The present application claims priority to U.S. Provisional application No. 63/307,500 filed on February 7, 2022.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claims 1, 2, 3, 5, 6, 7, 11, 12, 13, 14, 15, 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Mahmud et al. (US20200372395A1, filed on May 20, 2019, hereinafter “Mahmud”), in view of Khapali et al. (US20210034960A1, filed on July 29, 2019, hereinafter “Khapali”), further in view of Sagawa et al. (Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization, published November 20, 2019, hereinafter “Sagawa”). Each of these dates is before the effective filing date of the instant application, i.e., February 07, 2023, as well as the provisional filing date of February 07, 2022 where it applies.
With respect to Independent Claims 1, 11, and 13:
Mahmud teaches:
“a data augmentor to: receive a task data pertaining to information related to a pre- defined task to be performed by a model;” (Paragraph [0009] teaches that the model can receive information related to a task that can be performed by said model, “Once the cognitive system has been so optimized, it may perform its intended operation, such as a deep question/answer system which receives a user query and responsively provides an answer to the user query.” Paragraph [0020] further teaches that the data augmentor (data augmentation engine) proceeds to augment data based on pre-defined input information, “A data scientist provides the input training and test data, a machine learning model for the specific application (e.g., a business or engineering problem), an initial set of augmentation methods, and optionally a set of parameters associated with each augmentation method. Data augmentation engines apply a set of augmentation methods as specified in the input and, using the parameter/rule specified, select the data to be augmented, augment the data and generate augmented training data sets. For each set of augmented data, a machine learning model is trained.”)
“and augment the task data to obtain an augmented aspect data, wherein the augmented aspect data comprises a plurality of aspect queries obtained by augmentation of a base query in the task data based on at least one aspect, and wherein the base query corresponds to a simple question, the plurality of aspect queries include various possible forms of the question related to different aspects pertaining to the base query;” (Paragraph [0055] discloses that the task data (training data set) comprises of a plurality of aspect queries (question/answer pairs) that are augmented and dissected to properly assess each aspect point of the base training data set, “Assessment process 100 begins when computer system 10 receives a training data set for the cognitive system (102). The training data takes a form suitable for the particular cognitive system involved, e.g., question/answer pairs for a text-based Al. A plurality of augmented training data sets are generated using various augmentation techniques (104). The augmented training data sets may also be different variations of a single augmentation method where one or more parameters are varied. The cognitive system is trained with the first set of augmented training data (106). The nature of the training involved again depends upon the particular type of cognitive system, e.g., a text-based Al may use tools such as text annotators, parse tree generators, etc., to classify the input variables.” Paragraph [0054] further reinforces that the base query corresponds to a simple question, “…yielding an optimized deep question/answer 98. A user can submit a query to optimized deep question/answer system 98, e.g., in a natural language form. The NLP engine of optimized deep question/answer system 98 can then process the query and apply it to the trained system to provide one or more answers.”) Examiner’s Note: [0004] denotes the various possible forms of questions related to different aspects i.e. (person, location, organization, particular objects, positive/negative sentiment) etc.
“and a model evaluator to: evaluate, based on the augmented aspect data, a trained model to obtain aspect evaluation metrics, wherein the trained model is trained using the task data,” (Paragraph [0020] teaches a model evaluator (validation engine) assessing a trained model to obtain aspect evaluation metrics in correspondence with the training task data, “A validation engine then computes a set of scores for each of the augmented models by applying different test sets that can include the original test set and augmented test set which contain augmented vocabulary or other features. Each augmented model thereby gets a set of scores, such as accuracy gain from base model on original test set, accuracy gain from base model on augmented test set, vocabulary gain comparing to base model, grammatical quality of augmented data, syntactic variation of augmented data, readability of augmented data, etc.” Paragraph [0007] teaches training the model using task data (set of training data), “Cognitive system 11 takes a set of training data 12 and applies it to its input algorithms to learn from the data. The training data set is thus used to help form the basis of the system's logic, e.g., adjust the weightings on a neural network. The result is a trained cognitive system 14.”) Examiner’s Note: Addresses the “training, by the processor, using the task data…” limitation of Claims 11 and 13.
“wherein the evaluation enables to assess performance of the trained model by computing a performance score based on the aspect evaluation metrics,” (Paragraph [0007] discloses that aspect evaluation metrics (set of validation data) can be utilized to assess the performance of a trained model prompting the generation of a performance score, “A set of validation data 15 can then be submitted to trained cognitive system 14. A validation set can be used to minimize over-fitting , i.e., not necessarily adjusting the weightings of the network but just verifying accuracy over the training data. There are a variety of validation tests that can be used for this purpose. The result is a validation score 18 which indicates whether additional training may be required.”)
“and wherein the performance score enables to evaluate reliability of the trained model in a pre-defined domain.” (Paragraph [0052] discloses the generation of a combined validation score which is utilized to determine reliability (goodness) of a trained model, in a pre-defined domain (a particular augmentation method), “Each of the validation tests results in a separate component score. These scores may be combined in various manners to arrive at a combined validation score for a particular augmentation method…,combined validation score 84 is a combination of the individual scores with respective weightings for each test, i.e., a weighted average of the scores. Such weights represent goodness of each augmentation method.”)
“training, by the processor, using the task data, the model to obtain a trained model;” (Paragraph [0049] discloses an original model trained from the original training data (training the model using the task data to obtain a trained model), “…the original training data could contain ten thousand examples, the test data could contain one thousand examples, and the model could be support vector machine based. Accuracy of the original model (trained from original training data) on the original test data might be 70%”. Paragraph [0007] discloses a cognitive system that learns from training data to become a trained cognitive system, “Cognitive system 11 takes a set of training data 12 and applies it to its input algorithms to learn from the data…The result is a trained cognitive system 14.”) Examiner’s Note: Limitation exclusive to Claim 11.
Mahmud does not appear to explicitly disclose:
“determine whether the performance score satisfies a pre-defined threshold value associated with the pre-defined domain;”
“and a fine-tune engine to: execute a fine-tuning iterative loop to re-train the trained model based on the performance score being lower than the pre-defined threshold value,”
“re-train the trained model during the fine-tuning iterative loop, based on the task data using a loss function-----------"
“wherein the fine-tuning iterative loop corresponds to repeated cycles including the re-training of the model based on the task data and a subsequent evaluation of the re-trained model based on the augmented aspect data;”
However, Khapali teaches:
“determine whether the performance score satisfies a pre-defined threshold value associated with the pre-defined domain;” (Paragraph [0038] discloses determining which threshold to use (accuracy or reliability), and adjusting its value, the threshold type and value being dictated by the model, system, or production server requirements (a pre-defined threshold value associated with the pre-defined domain), “…CLP 150 may utilize a plurality of models to dynamically adjust one or more thresholds that determine when to trigger or initiate CLP 150 to evaluate, train/retrain, classify/reclassify, and store models 152A through 1527 corresponding to a plurality of entries in self-learning table 126. CLP 150 determines which threshold (e.g., accuracy, reliability, storage, etc.) to utilize and adjusts the threshold value amount (e.g., a 20 % training set delta, etc.).” Paragraph [0040] discloses comparing the model feedback (the performance score) against the threshold at decision block 214, “If the model feedback does not exceed a threshold (“If the model feedback does not exceed a threshold (“ No” branch, decision block 214), then CLP 150 returns to adjusting hyperparameters and training data (step 202).”)
“and a fine-tune engine to: execute a fine-tuning iterative loop to re-train the trained model based on the performance score being lower than the pre-defined threshold value,” (Paragraph [0040] discloses that when the model feedback does not exceed the threshold, the program returns to adjusting hyperparameters and training data and retrains the model (re-training based on the performance score being lower than the threshold). Paragraph [0024] discloses that the program then generates additional feedback data on the retrained models (a fine-tuning iterative loop), “CLP 150 dynamically adjusts one or more thresholds associated with one or more model feedback statistics. CLP 150 monitors a plurality of features contained within the feedback sets, in addition to feedback data delta. Responsive to a exceeding a feedback data threshold, CLP 150 adjusts the hyperparameters and training sets and retrains the previously trained models. CLP 150 generates additional feedback data on the retrained models.” Paragraph [0029] discloses that the weights of a previously trained model that failed to exceed an accuracy threshold are utilized in a subsequent training iteration (re-training the trained model), “For example, the weights of a previously trained model, that failed to exceed an accuracy threshold, are utilized in a subsequent training iteration.”)
“re-train the trained model during the fine-tuning iterative loop, based on the task data using a loss function-----------" (Paragraph [0031] discloses retraining the models using the training data from the training corpus (based on the task data) with cross-entropy as the loss function (using a loss function), and continuing supervised training when the calculated accuracy is insufficient, “In another embodiment, CLP 150 utilizes cross entropy (e.g., Kullback-Leibler (KL) divergence, etc.) as a loss function to determine the level of accuracy of the model. In this embodiment, CLP 150 compares the predicted sequence with the expected sequence. In yet another embodiment, CLP 150 may utilize the cross-entropy loss value to calculate error rate which further denotes the level of accuracy of the model. If the calculated accuracy is insufficient, then CLP 150 continues with supervised training. If the calculated accuracy is determined sufficient, then CLP 150 ends the training process.”)
“wherein the fine-tuning iterative loop corresponds to repeated cycles including the re-training of the model based on the task data and a subsequent evaluation of the re-trained model based on the augmented aspect data;” (Paragraphs [0024], “CLP 150 monitors a plurality of features contained within the feedback sets, in addition to feedback data delta. Responsive to a exceeding a feedback data threshold, CLP 150 adjusts the hyperparameters and training sets and retrains the previously trained models. CLP 150 generates additional feedback data on the retrained models.” and [0040] disclose repeated cycles of retraining and subsequent evaluation.) Examiner’s Note: Limitation exclusive to Claim 11. Encompasses entirety of claim 7 as well.
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to have modified the system of Mahmud to include the threshold-based retraining of Khapali. Mahmud already recognizes that the validation score indicates whether additional training may be required (Mahmud [0007]), and Khapali provides a known, automated technique for acting on that indication by comparing model feedback against a requirement-based threshold and retraining models that fail to exceed it (Khapali [0038]-[0040]). The motivation to do so would be to improve model accuracy and reliability through responsive retraining (Khapali [0012]).
The combination of Mahmud-Khapali does not appear to explicitly disclose:
“-----comprising a penalization parameter, wherein the penalization parameter pertains to penalization corresponding to the at least one aspect of the augmented aspect data,”
“determine an updated learning rate of the trained model based on the model performance with respect to the at least one aspect of the augmented aspect data,”
“update weights of the trained model based on the updated learning rate,”
“and generate, based on the updated weights, an updated trained model having improved reliability in the pre-defined domain with respect to the at least one aspect of the augmented aspect data.”
However, Sagawa teaches:
“-----comprising a penalization parameter, wherein the penalization parameter pertains to penalization corresponding to the at least one aspect of the augmented aspect data,” ([section 2, equation (4) and section 5, equation (7)], discloses dividing the training data into pre-defined groups (aspects) and weighting the loss of each group by a group weight q_g (a penalization parameter corresponding to the at least one aspect), “we leverage prior knowledge of spurious correlations to define groups over the training data and then define the uncertainty set Q in terms of these groups.” [Section 2.1: MultiNLI] further discloses that for the natural language inference task, the groups are defined by the presence or absence of negation words (a negation aspect).)
“determine an updated learning rate of the trained model based on the model performance with respect to the at least one aspect of the augmented aspect data,” ([section C.2, pg. 18] discloses determining an updated learning rate, “The interaction between batch norm and 2 penalties means that we had to adjust learning rates for each different 2 penalty strength (and each dataset). The learning rates below were chosen to be the highest learning rates that still resulted in stable optimization.” [Section 5, pg. 9] discloses maintaining weights over the groups that grow for groups on which the model performs poorly, “we maintain a distribution q over groups, with high masses on high-loss groups.” [Algorithm 1] discloses updating each group weight according to the model’s current loss on that group (model performance with respect to the aspect) and scaling each parameter update by the step size multiplied by that group weight (an updated learning rate).)
“update weights of the trained model based on the updated learning rate,” (Algorithm 1 in addition to [Section 5, pg. 9] disclose interleaving stochastic gradient descent on the model parameters θ (weights of the trained model) with updates to the group weights, each parameter update being scaled by the step size multiplied by the group weight (based on the updated learning rate), “we interleave gradient-based updates on θ and q. Intuitively, we maintain a distribution q over groups, with high masses on high-loss groups, and update on each example proportionally to the mass on its group. Concretely, we interleave SGD on θ and exponentiated gradient ascent on q (Algorithm 1).”)
“and generate, based on the updated weights, an updated trained model having improved reliability in the pre-defined domain with respect to the at least one aspect of the augmented aspect data.” ([pg. 1] discloses that the trained models achieve 10-40 percentage point improvements in worst-group accuracy (pre-defined domain). [Section 3.2] further discloses that, on the negation-grouped natural language inference task, worst-group test accuracy and reliability improves (an updated trained model having improved reliability with respect to the at least one aspect), “A different, implicit form of regularization is early stopping (Hardt et al., 2016b). We use the same settings in Section 3.1 but only train each model for a fixed (small) number of epochs (Section C.2). As with strong 2 penalties, curtailing training reduces the generalization gap and prevents models from fitting the data perfectly. In this setting, DRO also does substantially better than ERM on worst-group test accuracy, improving from 6.7% to 86.0% on Waterbirds, 25.0% to 88.3% on CelebA, and 66.0% to 77.7% on MultiNLI.”)
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to have modified the combination of Mahmud and Khapali to perform the retraining using the group-weighted training of Sagawa, with the augmentation methods of Mahmud serving as the groups. Sagawa identifies the same problem that the per-augmentation scores of Mahmud would reveal, specifically models that are accurate on average but consistently fail on identifiable groups of data, including groups defined by negation (Sagawa, Section 1), and provides a training step that weighs those groups according to their loss with little computational overhead relative to standard training (Sagawa, Section 5). The motivation to do so would be to improve the accuracy of the retrained model on the specific aspects on which it underperforms, “By coupling group DRO models with increased regularization—a stronger-than-typical 2 penalty or early stopping—we achieve substantially higher worst-group accuracies, with 10–40 percentage point improvements on a natural language inference task and two image tasks, while maintaining high average accuracies.” (Sagawa, Abstract; Section 3.2).
Therefore, Claims 1, 11, and 13 are rejected.
With respect to Claims 2 and 19:
The combination of Mahmud-Khapali-Sagawa teaches:
“wherein the model is an artificial intelligence (AI) model.” (Mahmud Paragraph [0052] teaches that the model is of a machine learning type which is a known artificial intelligence model, “In order to train the machine learning model, the data scientist can collect...”)
Therefore, Claims 2 and 19 are rejected.
With respect to Claims 3 and 20:
The combination of Mahmud-Khapali-Sagawa teaches:
“wherein the model is a deep learning (DL) based language model.” (Mahmud Paragraph [0002] teaches that the artificial intelligence model is of a deep learning type, “A cognitive system (sometimes referred to as deep learning, deep thought, or deep question answering) is a form of artificial intelligence that uses machine learning and problem solving.”)
Therefore, Claims 3 and 20 are rejected.
With respect to Claims 5, 12, and 14:
The combination of Mahmud-Khapali-Sagawa teaches:
“wherein the trained model is implemented for prediction in the pre-defined domain based on the performance score being greater than the pre-defined threshold value.” (Khapali Paragraph [0041] discloses deploying the highest ranked model to the production environment when the model feedback exceeds the threshold (implementing the trained model for prediction based on the performance score being greater than the threshold), “If the model feedback exceeds a model threshold (“Yes” branch, decision block 214), then CLP 150 selects and deploys the model (step 216). In an embodiment, CLP 150 selects the highest ranked model, as described in step 210, and deploys said model to a production environment or server.”)
Therefore, Claims 5, 12, and 14 are rejected.
With respect to Claims 6 and 15:
The combination of Mahmud-Khapali-Sagawa teaches:
“wherein the pre-defined threshold value is configurable based on the pre-defined domain.” (Khapali Paragraph [0038] discloses that the threshold type and value may be specified by a user or dictated by the model, system, or production server requirements (configurable based on the pre-defined domain), “In an embodiment, the user specifies the threshold type and/or value. In other embodiment, the model, system, and/or production server requirements/purpose dictates the threshold type and/or value. For example, an exemplary production server may only have sufficient computational storage for a small model (e.g., > 500 megabytes). In this example, CLP 150 adjusts the threshold to track model storage requirements and adjusts the value to conform with the requirements of the production server (i.e., > 500 megabytes).”)
Therefore, Claims 6 and 15 are rejected.
With respect to Claim 7:
The combination of Mahmud-Khapali-Sagawa teaches:
“execute a fine-tuning iterative loop to re-train the model based on the performance score being lower than the pre-defined threshold value, wherein the fine-tuning iterative loop corresponds to repeated cycles comprising re-training of the model based on the task data and a subsequent evaluation of the re-trained model based on the augmented aspect data.” (Khapali Paragraph [0024] discloses retraining the previously trained models and generating additional feedback data on the retrained models, and Paragraph [0040] discloses returning from decision block 214 to step 202 such that training, feedback generation, and threshold comparison repeat (repeated cycles of re-training and subsequent evaluation). In the combination, the evaluation is performed on the augmented test set of Mahmud (based on the augmented aspect data) (Mahmud [0020], [0050]).) Examiner’s Note: See rejection for Claims 1, 11, and 13 (limitations under Mahmud-Khapali).
Therefore, Claim 7 is rejected.
Claims 8 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Mahmud, in view of Khapali, in view of Sagawa, further in view of Wu et al. (“Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models,” published June 2021, hereinafter “Wu”).
The combination of Mahmud-Khapali-Sagawa does not appear to completely disclose:
“wherein the aspect pertains to at least one of: a contradiction aspect, a counterfactual aspect, a negation aspect, a domain based aspect, and a style transfer aspect.” ([Sagawa pg. 3] teaches training aspects pertaining to negation and contradiction, “In natural language inference, the task is to determine if a given hypothesis is entailed by, neutral with, or contradicts a given premise. Prior work has shown that crowdsourced training datasets for this task have significant annotation artifacts, such as the spurious correlation between contradictions and the presence of the negation words…”)
However, Wu teaches:
“wherein the aspect pertains to at least one of: a contradiction aspect, a counterfactual aspect, a negation aspect, a domain based aspect, and a style transfer aspect.” ([Pg. 6711, Section 3.2, Paragraph 1] teaches a data aspect augmentation type eliciting counterfactual and domain-specific properties, “we augment training sets with counterfactual examples. In all experiments, we finetune roberta-base on datasets of n original examples and m counterfactuals, which are generated by Polyjuice (m-polyjuice) or crafted from scratch by humans (m-CAD from Kaushik et al. (2020), only available for NLI). To distinguish the benefit of counterfactuals from that of just adding more data, we further add a baseline that uses n+m original examples (m-baseline). In addition to in-domain test set accuracy, we measure models’ generalization on out-of-domain datasets, as well as contrast sets and challenge sets.” [Pg. 6714, Section 5, Paragraph 2] further teaches that negation in collaboration with contradiction aspect augmentation/conditioning can also be performed on the model training data, “There is a known correlation between the label Contradiction and hypotheses with negation in NLI datasets (Gururanganetal.,2018), which may cause models to fail on non-contradiction negations. We explore this in Figure 5A by generating counterfactual hypotheses for a random Neutral instance, conditioning only on the original x and the negation control code.”)
Mahmud, Khapali, Sagawa, and Wu are analogous art and in the same field of invention because all four references pertain to augmenting existing datasets with synthetic variations and diversifying input scenarios to significantly enhance model reliability, reduce overfitting and improve generalization. While Mahmud teaches utilizing data augmentation to transform sparse datasets into high-confidence, generalized models by iteratively updating training sets to enhance robustness, Khapali teaches designing automated validation frameworks to generate performance benchmarks from data for tasks like anomaly detection. Similarly, while Sagawa teaches introducing convergence guarantees to efficiently train group DRO model, Wu teaches improving model robustness, evaluation, and explanation by generating high-quality counterfactuals that reveal error patterns easily that are overlooked. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of Mahmud (improving the structural soundness of high-quality inputs/outputs through rigorous verification analysis) with the teachings of Khapali (re-training models on the basis of statistical reliability metrics) in view of the teachings of Sagawa (worst-case generalization) further in view of the teachings of Wu (producing diverse datasets of realistic counterfactuals) in order to leverage automated retesting, rigorous performance tracking, and dynamic retraining thresholds to ensure model stability. One of ordinary skill in the art would be motivated to do so because by integrating Gadde, Khapali, and Wu’s frameworks into the methods of Mahmud one would be able to, "produce training data that significantly improve model generalization, as well as contrast sets that help identify model vulnerabilities (Gardner et al., 2020), with around 70% less annotation effort. In another application, Polyjuice produces counterfactual explanations (§4), providing significant insight on top of state-of-the-art explanation techniques. Finally, Polyjuice supports counterfactual error analysis (§5). It allows users to explore related counterfactuals (e.g., the model responds differently to different negation forms in Figure 1B), and to aggregate individual counterfactuals into patterns in order to gain systematic understanding of model behavior, {[Pg. 6708 paragraph. 3] of Wu}."
Therefore, Claims 8 and 16 are rejected.
Claims 10 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Mahmud, in view of Khapali, in view of Sagawa, further in view of Goto et al. (U.S. Patent Application Publication No. US20200110994A1, filed on October 4, 2018, hereinafter “Goto”).
With respect to Claims 10 and 18:
The combination of Mahmud-Khapali-Sagawa does not appear to explicitly disclose:
“wherein an extent of the penalization depends on model performance and accuracy with respect to the augmented aspect data, and wherein the penalization of the loss function for each aspect is based on a pre-defined threshold.”
However, Goto teaches:
“wherein an extent of the penalization depends on model performance and accuracy with respect to the augmented aspect data, and wherein the penalization of the loss function for each aspect is based on a pre-defined threshold.” (Paragraph [0028] discloses that loss function penalization depends on model accuracy, “If the class accuracy is greater than a threshold of difference of class accuracy (
PNG
media_image1.png
27
64
media_image1.png
Greyscale
) for a particular class as compared to another class after an epoch training, the magnitude of augmentation may be updated so that the number of training samples for the more accurate class is reduced and/or the number of training samples in the less accurate class is increased in the next training loop. After the class accuracies become balanced, updating of the magnitude of augmentation ceases and the trained neural network is preserved.” Paragraph [0030] further discloses that the augmented aspect data-based loss function penalization is relative to a pre-defined threshold, “At operation 310, the system determines whether the total loss is less than a threshold of loss. If the total loss is less than the threshold of loss, the system proceeds to operation 315, wherein a flag for enabling the augment mode is set to true, allowing the magnitude of augmentation parameter to be updated.”)
Mahmud, Khapali, Sagawa, and Goto are analogous art and in the same field of invention because all four references pertain to assessing and selecting the optimal augmentation methods based on performance metrics like class accuracy and training loss to improve model efficiency. While Mahmud teaches modifying artificial intelligence models to act as a user-friendly, high-confidence question-answering tools by training on expert-provided data to identify entities and sentiment, Khapali teaches automatically generating identifier characteristics from data for tasks like image recognition, allowing the system to learn complex patterns without being programmed with specific, task-related rules, competitive, and dynamic against performance thresholds. Similarly, while Sagawa teaches improving the worst-case (or minority group) accuracy of overparameterized neural networks by showing that coupling group Distributionally Robust Optimization (DRO) with stronger regularization prevents models from relying on spurious correlations, Goto teaches actively adjusting dataset augmentation through affine transformations based on total loss and class-specific accuracy, creating a feedback loop to address specific training challenges. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of Mahmud (improving the structural soundness of high-quality inputs/outputs through rigorous verification analysis) with the teachings of Khapali (re-training models on the basis of statistical reliability metrics) in view of the teachings of Sagawa (scalable algorithm implementation) further in view of the teachings of Goto (preventing bias by augmenting data) in order to implement an automated, self-improving machine learning pipeline incorporating advanced data enhancement that leverages automated retraining, performance feedback, and dynamic thresholds to ensure production readiness. One of ordinary skill in the art would be motivated to do so because by integrating Gadde, Khapali, and Goto’s frameworks into the methods of Mahmud one would be able to, "offer the added benefit of suppressing overfitting as well as improving performance on generalized data, {[0036] of Khapali}."
Therefore, Claims 10 and 18 are rejected.
Claims 21-23 are rejected under 35 U.S.C. 103 as being unpatentable over Mahmud, in view of Khapali, in view of Sagawa, further in view of Ng (Andrew Ng, “CS229 Lecture Notes: Supervised Learning,” Stanford University, updated April 4, 2020, hereinafter “Ng”).
With respect to Claim 21:
The combination of Mahmud-Khapali-Sagawa appears to explicitly disclose:
“wherein the augmented aspect data comprises a first aspect and a second aspect, and wherein the model evaluator is further to compute, based on the augmented aspect data, a first aspect evaluation metric corresponding to the first aspect and a second aspect evaluation metric corresponding to the second aspect; and compute the performance score based on the first aspect evaluation metric and the second aspect evaluation metric,------” (Mahmud Paragraph [0036] discloses a plurality of augmentation methodologies (a first aspect and a second aspect), “Augmentation engine 64a augments training data 62 using dictionary - based replacement; augmentation engine 64b augments training data 62 using text generation; augmentation engine 64c augments training data 62 using paraphrased sentences; and augmentation engine 64d augments training data 62 using back translation .” and Paragraph [0052] discloses that each validation test results in a separate component score (first and second aspect evaluation metrics) that are combined into a combined validation score (the performance score based on the first and second metrics), “Each of the validation tests results in a separate component score. These scores may be combined in various manners to arrive at a combined validation score for a particular augmentation method. In a simple implementation where the scores have been normalized, an average of the scores can be taken. In the preferred implementation, however, combined validation score 84 is a combination of the individual scores with respective weightings for each test, i.e., a weighted average of the scores. Such weights represent goodness of each augmentation method.” Examiner’s Note: Sagawa Section 2 (separate loss computed for each group) also viable teaching.
The combination of Mahmud, Khapali, and Sagawa does not appear to explicitly disclose:
“------and wherein computing the first aspect evaluation metric, the second aspect evaluation metric and the performance score, based on: J1(θ0, θ1) = hθ(x1) − y1, J2(θ0, θ1) = hθ(x2) − y2, where x1 and x2 correspond to inputs associated with the first aspect and the second aspect, and y1 and y2 correspond to outputs associated with the first aspect and the second aspect, and hθ corresponds to a learned model prediction.”
However, Ng teaches:
“------and wherein computing the first aspect evaluation metric, the second aspect evaluation metric and the performance score, based on: J1(θ0, θ1) = hθ(x1) − y1, J2(θ0, θ1) = hθ(x2) − y2, where x1 and x2 correspond to inputs associated with the first aspect and the second aspect, and y1 and y2 correspond to outputs associated with the first aspect and the second aspect, and hθ corresponds to a learned model prediction.” ([Page 2] discloses that x(i) denotes the input variables and y(i) denotes the output or target variable, and that a hypothesis h is learned so that h(x) predicts the corresponding value of y (x corresponds to inputs, y corresponds to outputs, and hθ corresponds to a learned model prediction). [Page 3] further discloses hθ(x) = θ0 + θ1x1 + θ2x2, where the θi’s are the parameters, also called weights (θ0 and θ1 of J(θ0, θ1). [Page 4] discloses a cost function J(θ) that measures how close each prediction hθ(x(i)) is to the corresponding y(i), formed from the difference hθ(x(i)) − y(i), and derives, for a single training example (x, y), the error term hθ(x) − y (an evaluation metric J(θ0, θ1) = hθ(x) − y). Applying this metric separately to the inputs and outputs of each aspect yields J1 and J2.)
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to have modified the combination of Mahmud, Khapali, and Sagawa to compute each per-aspect metric as the difference between the model prediction and the target output, as taught by Ng. The motivation to do so would be to use the standard measure of prediction error for each aspect, which is a simple substitution of one known error measure for another to obtain predictable results.
Therefore, Claim 21 is rejected.
With respect to Claim 22:
The combination of Mahmud-Khapali-Sagawa appears to explicitly disclose:
“wherein fine-tuning the trained model comprises: applying, by the processor, the loss function that penalizes the trained model based on both the task data and the augmented aspect data;--------” ([Sagawa Section 2] discloses a training objective computed over all groups, including the majority group (the task data) and the minority groups (the augmented aspect data), with each group’s loss weighted (penalizing based on both), “We assume that we know which group each training point comes from—i.e., the training data comprises (x,y,g) triplets—though we do not assume we observe g at test time, so the model cannot use g directly. Instead, we learn a group DRO model minimizing the empirical worst-group risk ˆ R(θ): ˆ θDRO := argmin θ∈Θ ˆ R(θ) := max g∈G E(x,y)∼ ˆPg [ (θ;(x,y))] , (4) where each group ˆPg is an empirical distribution over all training points (x,y,g) with g = g (or equivalently, a subset of training examples drawn from Pg). Group DRO learns models with good worst-group training loss across groups. This need not imply good worst-group test loss because of the worst-group generalization gap δ := R(θ) − ˆ R(θ). We will show that for overparameterized neural networks, δ is large unless we apply sufficient regularization.”)
The combination of Mahmud-Khapali-Sagawa does not appear to explicitly disclose:
“--------and updating, by the processor, weights of the trained model using the updated learning rate according to θj = θi − α′ ∂/∂θj J(θ0, θ1), wherein α′ represents the updated learning rate that is determined based on the augmented aspect data and controls an extent to which the weights are updated during fine-tuning of the trained model.”
However, Ng teaches:
“--------and updating, by the processor, weights of the trained model using the updated learning rate according to θj = θi − α′ ∂/∂θj J(θ0, θ1), wherein α′ represents the updated learning rate that is determined based on the augmented aspect data and controls an extent to which the weights are updated during fine-tuning of the trained model.” ([Page 4] discloses the gradient descent algorithm, which repeatedly updates each parameter by the update θj := θj − α ∂J(θ)/∂θj (updating weights of the trained model according to θj = θj − α ∂J/∂θj), “Here, α is called the learning rate.” [Page 5] discloses that the magnitude of each parameter update is proportional to the error term, so that little change is made when the prediction nearly matches the actual value and a larger change is made when the prediction error is large (the learning rate controls an extent to which the weights are updated). [Page 7, footnote 2] further discloses changing the learning rate α as the algorithm runs (an updated learning rate).) Examiner’s Note: Sagawa (algorithm 1) denotes an updated learning rate determined from the per-group loss (determined based on the augmented aspect data). Under broadest reasonable interpretation to be able to implement the weight update of Sagawa using the gradient descent update rule taught by Ng, with the group-dependent step size of Sagawa serving as the learning rate, a PHOSITA would be able to denote Ng’s update rule as the standard formulation of gradient-based training and also be able to note that its use would produce predictable results.
Therefore, Claim 22 is rejected.
With respect to Claim 23:
The combination of Mahmud-Khapali-Sagawa-Ng appears to explicitly disclose:
“wherein the updated learning rate is representative of accuracy on task aspects in the augmented aspect data pertaining to the first aspect and the second aspect,--------” (Sagawa Algorithm 1 discloses group weights updated from the loss on each group, such that the step size applied to each group reflects the model’s performance on that group (an updated learning rate representative of accuracy on the first aspect and the second aspect), “loss : Θ×(X ×Y)→R+, and training data drawn from some distribution P, the standard goal is to find a model θ ∈ Θ that minimizes the expected loss EP[ (θ;(x, y)] under the same distribution P. The standard training procedure for this goal is empirical risk minimization (ERM): ˆ θERM := argmin θ∈Θ E(x, y)∼ ˆP[ (θ;(x, y))].”
Khapali further teaches:
“--------and wherein the penalization of the loss function for the first aspect is based on a first pre-defined threshold pertaining to the first aspect, and the penalization of the loss function for the second aspect is based on a second pre-defined threshold pertaining to the second aspect.” (Paragraphs [0038] and [0039] disclose a plurality of thresholds, each having an adjustable type and value (first and second pre-defined thresholds). In the combination, applying a separate threshold to the penalization of each of the groups of Sagawa yields a first pre-defined threshold pertaining to the first aspect and a second pre-defined threshold pertaining to the second aspect.) Examiner’s Note: Under broadest reasonable interpretation the application of the plurality of adjustable thresholds of Khapali to the group-specific penalization of Sagawa, serves to allow the required level of performance to be set independently for each aspect according to the requirements of the application such that, each aspect is governed by its own threshold.
Therefore, Claim 23 is rejected.
Response to Amendment
Claim Rejections - 35 USC § 101
Applicant’s amendments and arguments filed with respect to the rejection of Claims 1-20 under 35 U.S.C. § 101 have been fully considered. The rejection of Claims 1-3, 5-8, 10-16, and 18-20 under 35 U.S.C. § 101 is withdrawn. The rejection of claims 4, 9, and 17 is moot in view of their cancellation. New claims 21-23 are not rejected under 35 U.S.C. § 101.
The examiner agrees that the recited operations of re-training a trained model cannot practically be performed in the human mind. Those operations include executing the fine-tuning iterative loop, applying the loss function, and updating the model’s weights. The examiner therefore does not characterize them as mental processes. However, the examiner does not agree that the claims recite no judicial exception. The claims still recite mental processes, namely rephrasing a simple question into various forms, evaluating and scoring the results, and comparing the score to a threshold. They also recite mathematical concepts, namely a loss function with a penalization parameter, the determination of an updated learning rate, and the updating of weights based on that learning rate. Applicant’s specification describes these operations as equations, and new claims 21 and 22 recite those equations.
Applicant argues that the amended claims integrate any judicial exception into a practical application. According to Applicant, the claims address the technical problem of superficial, base-query-based reliability evaluation and ambiguous re-training [0003] through aspect-based evaluation and threshold-based re-training that improves the performance and reliability of the model [0040].
The examiner agrees. They recite a particular manner of re-training the model. The remaining issues of claim scope are addressed under 35 U.S.C. § 103.
Claim Rejections - 35 USC § 102
A. The rejection of Claims 1-3, 19 and 20 under 35 U.S.C. § 102(a)(2) as anticipated by Mahmud (US20200372395A1) is withdrawn in light of Applicant’s amendment to Independent Claims 1, 11, and 13.
Claim Rejections - 35 USC § 103
A. Applicant argues that Gadde fails to teach "re-train the trained model during the fine-tuning iterative loop, based on the task data using a loss function comprising a penalization parameter…corresponding to the at least one aspect."
The examiner respectfully disagrees. Gadde is no longer relied upon, so this argument is moot. In any event, the prior rejection of former claims 9 and 17, which recited this subject matter, relied on a combination of references and not on Gadde alone. Applicant addressed only Gadde. One cannot show non-obviousness by attacking references individually where the rejections are based on combinations of references. In re Keller, 642 F.2d 413, 426 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 1097 (Fed. Cir. 1986). Applicant argues that dependent claims 5–8, 10, 12, 14–16, and 18 are patentable by virtue of their dependency. Because independent claims 1, 11, and 13 remain rejected, this argument is not persuasive.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NOOR F CHEEMA whose telephone number is (571)272-9642. The examiner can normally be reached Monday-Friday 7:30am-5:00pm alternative Fridays off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.F.C./Examiner, Art Unit 2142 /Mariela Reyes/Supervisory Patent Examiner, Art Unit 2142
