Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This action is in response to the amendments filed 02 April 2026. Claims 3, 11, and 19 are cancelled. Claims 1, 2, 6, 7, 9, 10, 12-18, and 20 are amended. Claims 21-23 are newly added. Claims 1, 2, 4-10, 12-18, and 20-23 are pending and have been examined.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03 February 2026 being considered by the examiner.
Response to Arguments
Applicant's arguments, see pages 10-17, filed 02 April 2026, with respect to the rejections of Claims 1-20 under 35 U.S.C. 101 have been fully considered but they are not persuasive.
APPLICANT'S ARGUMENT: Applicant argues (page 10, paragraph 2) that "the claims recite a specific, computer-implemented machine-learning monitoring framework that integrates structured evaluation vectors, synthetic and historical training data management, and automated retraining triggers to address technical deficiencies in conventional model evaluation systems. Applicant's Specification explains that the claimed techniques improve the operation and reliability of machine-learning systems themselves, not merely the analysis of data."
Applicant argues (page 10, paragraph 2) that "the claims show an improvement in computer functionality that integrates any abstract idea into a practical application."
Applicant argues (page 13, paragraph 3) that "Claim 1 recites a specific, computer-implemented machine-learning monitoring framework that integrates structured evaluation vectors, synthetic and historical training data management, and automated retraining triggers to address technical deficiencies in conventional model evaluation systems."
EXAMINER'S RESPONSE: Examiner respectfully disagrees that amended Claim 1, as currently recited, recites a technical solution to a technical problem. The additional elements of amended Claim 1, taken singly and in combination, fail to integrate the mental processes recited by the claim into a practical application or provide significantly more. Thus, any improvement recited by amended Claim 1 is an improvement in a mental process rather than in a computer or computing technology.
Examiner further notes that neither amended Claim 1 nor any of the claims dependent thereupon recite "structured evaluation vectors" or "automated retraining triggers."
APPLICANT'S ARGUMENT: Applicant argues (page 14, continued paragraph) that "A human mind is not equipped to perform, at least, (i) receiving a request to process an input data object with a target machine learning model and (ii) modifying a parameter of the target machine learning model based on the influencing feature value to improve a performance of the target machine learning model, as these steps require the use of a specific machine learning computing system. Therefore, claim 1 does not recite a mental process as defined by the MPEP."
EXAMINER'S RESPONSE: Examiner respectfully disagrees.
The recited step of receiving a request appears to amount to mere data gathering, and is thus insignificant extra-solution activity under Step 2A Prong Two analysis. The recited step of receiving a request appears to represent receiving or transmitting data over a network, and is thus well-understood, routine, conventional activity under Step 2B analysis.
The recited step of modifying a model parameter based on a feature value appears, at the claimed level of generality, to represent a mental process step that can be performed entirely in the human mind or with pen and paper. As currently recited, the step does not recite a positive step of improving model performance.
APPLICANT'S ARGUMENT: Applicant argues (page 15, paragraph 1) that "even if claim 1 were directed to an abstract idea - which, Applicant submits, it is not - the claim recites a combination of additional elements that improves a technical field such that the claim as a whole integrates any alleged abstract idea into a practical application that is patent eligible under 35 U.S.C. § 101."
Applicant argues (page 15, paragraph 2) that "Claim 1 recites an improved machine learning monitoring and modification framework for improving a target machine learning model. Further, dependent claim 7 specifically retrains the target machine learning model using the augmented training data set. The overall techniques include the modification of a feature of the target machine learning model in response to a performance degradation of the target machine learning model. Additionally, the target machine learning model may be retrained on an augmented training data set to improve its performance."
Applicant argues (page 16, continued paragraph) that "The claimed modification step is tailored to the specific target machine learning model based on the influencing feature value determined to have a significant impact on the predictions of the model. ... For at least these reasons, claim 1 improves the monitoring, modification, retraining, and performance of the target machine learning model and, therefore, integrates any alleged abstract idea into a practical application."
Applicant argues (page 17, paragraph 3) that "the claims recite an unconventional combination of operations and data structures that provides non-routine results in the field of machine learning monitoring, modification, and retraining - and thus the claims provide an inventive concept."
EXAMINER'S RESPONSE: Examiner respectfully disagrees. As currently recited, amended Claim 1 does not recite singly or in combination any additional elements that integrate the recited abstract ideas into a practical application or provides significantly more.
Applicant' s arguments, see pages 17-19, filed 02 April 2026, with respect to the rejections of Claims 1-20 under 35 U.S.C. 103 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
APPLICANT'S ARGUMENT: Applicant argues (page 18, paragraph 1) that "Wang discovers defects at certain lifecycle stages of a pipeline and generates synthetic data to simulate these defects. This is not the same as identifying synthetic data similar to an input data object pulled from a training dataset used to previously train the target machine learning model."
Applicant argues (page 19, paragraph 1) that "The Office Action relates the 'generated defects' of Wang with the 'counterfactual proposals' of the present application. However, Wang does not teach the counterfactual proposals, as amended and clarified in claim 1 and new claims 21-23."
EXAMINER'S RESPONSE: Examiner notes that Applicant's arguments are moot. Amended Claim 1 is now rejected under 35 102 in light of McGrath. Newly recited Claims 21-23 are rejected in view of McGrath in view of Mothilal.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 2, 4-10, 12-18, and 20-23 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1
Step 1
Claim 1 recites method, and thus the claimed process falls within a statutory category of invention.
Step 2A Prong 1
The claim recites identifying ... the synthetic data object that corresponds to the input data object based on a corresponding input feature value shared by the synthetic data object and the input data object, which is a mental process. The claim recites in response to identifying the synthetic data object: modifying ... a holistic evaluation score for the target machine learning model, which is a mental process. The claim recites identifying ... a performance degradation for the target machine learning model based on the holistic evaluation score for the target machine learning model, which is a mental process. The claim recites identifying ... an influencing feature value corresponding to the performance degradation, wherein the influencing feature value is based on a counterfactual proposal that is generated by a counterfactual algorithm used to explain an individual predictive output during an inference phase once the target machine learning model is deployed, which is a mental process. The claim recites modifying ... a parameter of the target machine learning model based on the influencing feature value to improve a performance of the target machine learning model, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element computer-implemented invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element receiving, by one or more processors, a request amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering"). The additional element to process an input data object with a target machine learning model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element wherein the target machine learning model is previously trained using a training dataset comprising a synthetic data object and a historical data object does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element by the one or more processors invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element initiating ... the performance of a labeling process for assigning a ground truth label to the input data object invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element augmenting ... a supplemental training dataset with the input data object and the ground truth label amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering and outputting").
Step 2B
The additional element computer-implemented invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element receiving, by one or more processors, a request is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element to process an input data object with a target machine learning model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element wherein the target machine learning model is previously trained using a training dataset comprising a synthetic data object and a historical data object does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element by the one or more processors invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element initiating ... the performance of a labeling process for assigning a ground truth label to the input data object invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element augmenting ... a supplemental training dataset with the input data object and the ground truth label is well-understood, routine, conventional activity (see MPEP 2106.05(d), "storing and retrieving information in memory").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 2
Step 1
Regarding Claim 2, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
The claim recites determining an updated holistic evaluation score for the target machine learning model, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 4
Step 1
Regarding Claim 4, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
The claim recites wherein modifying the holistic evaluation score comprises reducing the holistic evaluation score, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 5
Step 1
Regarding Claim 5, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
The claim recites detecting a threshold augmentation stimulus based on the supplemental training dataset, which is a mental process. The claim recites in response to the threshold augmentation stimulus, generating an augmented training dataset by augmenting the training dataset with the supplemental training dataset, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 6
Step 1
Regarding Claim 6, the rejection of Claim 5 is incorporated.
Step 2A Prong 1
The claim recites identifying an influencing feature value corresponding to the performance degradation (as recited by Claim 1), wherein the corresponding input feature value is associated with an evaluation feature of the training dataset, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element wherein the target machine learning model is previously trained using a training dataset comprising a synthetic data object and a historical data object (as recited by Claim 1), wherein the synthetic data object is associated with the evaluation feature does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element wherein augmenting the training dataset comprises: replacing the synthetic data object with the supplemental training dataset amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering").
Step 2B
The additional element wherein the target machine learning model is previously trained using a training dataset comprising a synthetic data object and a historical data object (as recited by Claim 1), wherein the synthetic data object is associated with the evaluation feature does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element wherein augmenting the training dataset comprises: replacing the synthetic data object with the supplemental training dataset is well-understood, routine, conventional activity (see MPEP 2106.05(d), "storing and retrieving information in memory").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 7
Step 1
Regarding Claim 7, the rejection of Claim 5 is incorporated.
Step 2A Prong 1
The claim recites in response to the performance degradation, retraining the target machine learning model using the augmented training dataset, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 8
Step 1
Regarding Claim 8, the rejection of Claim 5 is incorporated.
Step 2A Prong 1
The claim recites detecting a threshold augmentation stimulus based on the supplemental training dataset (as recited by Claim 5), wherein the threshold augmentation stimulus is based on a threshold number of supplemental input data objects in the supplemental training dataset, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 9
Step 1
Claim 9 recites a system, and thus the claimed machine falls within a statutory category of invention.
Step 2A Prong 1
The claim recites identifying the synthetic data object that corresponds to the input data object based on a corresponding input feature value shared by the synthetic data object and the input data object, which is a mental process. The claim recites in response to identifying the synthetic data object: modifying a holistic evaluation score for the target machine learning model, which is a mental process. The claim recites identifying a performance degradation for the target machine learning model based on the holistic evaluation score for the target machine learning model, which is a mental process. The claim recites identifying an influencing feature value corresponding to the performance degradation, wherein the influencing feature value is based on a counterfactual proposal that is generated by a counterfactual algorithm used to explain an individual predictive output during an inference phase once the target machine learning model is deployed, which is a mental process. The claim recites modifying a parameter of the target machine learning model based on the influencing feature value to improve a performance of the target machine learning model, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element one or more processors; and one or more memories storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element receiving a request amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering"). The additional element to process an input data object with a target machine learning model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element wherein the target machine learning model is previously trained using a training dataset comprising a synthetic data object and a historical data object does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element initiating the performance of a labeling process for assigning a ground truth label to the input data object invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element augmenting a supplemental training dataset with the input data object and the ground truth label amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering and outputting").
Step 2B
The additional element one or more processors; and one or more memories storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element receiving a request is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element to process an input data object with a target machine learning model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element wherein the target machine learning model is previously trained using a training dataset comprising a synthetic data object and a historical data object does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element initiating the performance of a labeling process for assigning a ground truth label to the input data object invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element augmenting a supplemental training dataset with the input data object and the ground truth label is well-understood, routine, conventional activity (see MPEP 2106.05(d), "storing and retrieving information in memory").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Claims 10 and 12-16, dependent on Claim 9, incorporate the rejection of Claim 9. Claims 10-16 incorporate substantively all the limitations of Claims 2-8, respectively, in system form and are rejected under the same rationales.
Regarding Claim 17
Step 1
Claim 17 recites one or more non-transitory computer-readable media storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations, and thus the claimed manufacture falls within a statutory category of invention.
Step 2A Prong 1
The claim recites identifying the synthetic data object that corresponds to the input data object based on a corresponding input feature value shared by the synthetic data object and the input data object, which is a mental process. The claim recites in response to identifying the synthetic data object: modifying a holistic evaluation score for the target machine learning model, which is a mental process. The claim recites identifying a performance degradation for the target machine learning model based on the holistic evaluation score for the target machine learning model, which is a mental process. The claim recites identifying an influencing feature value corresponding to the performance degradation, wherein the influencing feature value is based on a counterfactual proposal that is generated by a counterfactual algorithm used to explain an individual predictive output during an inference phase once the target machine learning model is deployed, which is a mental process. The claim recites modifying a parameter of the target machine learning model based on the influencing feature value to improve a performance of the target machine learning model, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element receiving a request amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering"). The additional element to process an input data object with a target machine learning model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element wherein the target machine learning model is previously trained using a training dataset comprising a synthetic data object and a historical data object does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element initiating the performance of a labeling process for assigning a ground truth label to the input data object invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element augmenting a supplemental training dataset with the input data object and the ground truth label amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering and outputting").
Step 2B
The additional element receiving a request is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element to process an input data object with a target machine learning model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element wherein the target machine learning model is previously trained using a training dataset comprising a synthetic data object and a historical data object does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limit the use of the abstract idea to a particular technological environment"). The additional element initiating the performance of a labeling process for assigning a ground truth label to the input data object invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element augmenting a supplemental training dataset with the input data object and the ground truth label is well-understood, routine, conventional activity (see MPEP 2106.05(d), "storing and retrieving information in memory").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Claims 18 and 20, dependent on Claim 17, incorporate the rejection of Claim 17. Claims 18 and 20 incorporate substantively all the limitations of Claims 2-4, respectively, in non-transitory computer-readable media form and are rejected under the same rationales.
Regarding Claim 21
Step 1
Regarding Claim 21, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
The claim recites identifying ... an influencing feature value corresponding to the performance degradation, wherein the influencing feature value is based on a counterfactual proposal that is generated by a counterfactual algorithm used to explain an individual predictive output during an inference phase once the target machine learning model is deployed (as recited by Claim 1), wherein the counterfactual algorithm comprises one or more of: DICE, CCHVAE, CEM, CLUE, CRUDS, FACE, Growing Spheres, Revise, or Wacter, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Claim 22, dependent on Claim 9, incorporates the rejection of Claim 9. Claim 22 incorporates substantively all the limitations of Claim 21 in system form and is rejected under the same rationale.
Claim 23, dependent on Claim 17, incorporates the rejection of Claim 17. Claim 23 incorporates substantively all the limitations of Claim 21 in non-transitory computer-readable media form and are rejected under the same rationales.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 2, 4, 9, 10, 12, 17, 18, and 20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by McGrath, et al. (US 2022/0129794 A1, hereinafter "McGrath").
Regarding Claim 1, McGrath teaches:
A computer-implemented method (McGrath, Claim 1, "A method, comprising: receiving, by a device, user information; determining, by the device and based on a prediction model, a prediction output of an analysis of the user information") comprising:
... by one or more processors (McGrath, Fig. 1B, "Automated Analysis System") ... receiving ... a request to process an input data object (McGrath, Fig. 1B, "User Info" and [0019]: "As shown in FIG. 1B, and by reference number 110, the qualification model analyzes user information. For example, the qualification model may receive user information from the user device and the user information may be associated with a user. The qualification model may be configured to qualify the user for a service") with a target machine learning model (McGrath, Fig. 1B, "CE Generation Model"), wherein the target machine learning model is previously trained using a training dataset (McGrath, Fig. 3, "Trained Model 325") comprising a synthetic data object and a historical data object (McGrath, [0040]: "the automated analysis system, during a training period, may iteratively receive pairs of user information and a prediction output of the qualification model, iteratively select one or more relevant counterfactual explanations ( e.g., using the iteratively updated labels and/or retrained generator model, clustering model, or classification model), iteratively provide the selected counterfactual explanations for feedback, iteratively update labels based on iteratively received feedback data, and iteratively retrain the clustering model and/or the classification model according to the feedback data," where McGrath's generated counterfactual explanation corresponds to the instant synthetic data object, as in [0036]: "the automated analysis system may generate and store the counterfactual explanation along with a label determined from the feedback, as a labeled counterfactual explanation");
identifying, by the one or more processors, the synthetic data object that corresponds to the input data object (McGrath, Fig. 3, 340, "Determine cluster for new observation") based on a corresponding input feature value shared by the synthetic data object and the input data object (McGrath, Fig. 6, 650, "Select, based on a classification model, a counterfactual explanation from a cluster of the clusters of counterfactual explanations based on the prediction output and a relevance score of the counterfactual explanation" and [0002]: "selecting, based on a classification model, a counterfactual explanation from a cluster of the clusters of counterfactual explanations based on the prediction output and a relevance score of the counterfactual explanation, wherein the relevance score is determined based on the user information and a confidence score associated with the clustering of the plurality of counterfactual explanations and a confidence score associated with the prediction model");
in response to identifying the synthetic data object (McGrath, Fig. 6, 670, "Receive feedback data associated with the request for feedback" and 680, "Update a data structure associated with the clustering model based on the feedback data and the counterfactual explanation to form an updated data structure," where McGrath's system receives feedback from a user according to the identified counterfactual): modifying, by the one or more processors, a holistic evaluation score for the target machine learning model (McGrath, [0036]: "the automated analysis system may generate and store the counterfactual explanation along with a label determined from the feedback, as a labeled counterfactual explanation" and [0037]: "In this way, a label of the counterfactual explanation can be updated in the data structure to increase a quantity of labeled counterfactual explanations in the labeled counterfactual explanation data structure," where McGrath's label quantity is used with a threshold for system evaluation, as in [0041]: "The automated analysis system may iteratively perform the one or more processes until a quantity of counterfactual explanations in the labeled data structure satisfies a threshold and/or a ratio of a quantity of unlabeled data structures to the quantity of labeled data structures that satisfy a threshold"),
initiating, by the one or more processors, the performance of a labeling process for assigning a ground truth label to the input data object (McGrath, Fig. 6, 670, "Receive feedback data associated with the request for feedback" and 680, "Update a data structure associated with the clustering model based on the feedback data and the counterfactual explanation to form an updated data structure" [0036]: "the automated analysis system may generate and store the counterfactual explanation along with a label determined from the feedback, as a labeled counterfactual explanation," where McGrath's stored, labeled counterfactual corresponds to the instant ground truth), and
augmenting, by the one or more processors, a supplemental training dataset with the input data object and the ground truth label (McGrath, Fig. 6, 670, "Receive feedback data associated with the request for feedback" and 680, "Update a data structure associated with the clustering model based on the feedback data and the counterfactual explanation to form an updated data structure")
identifying, by the one or more processors, a performance degradation for the target machine learning model (McGrath, [0012]: "providing such infeasible counterfactual explanations to a user and/or operator, which may also degrade a user experience associated with using the automated analysis system, wastes computing resources (e.g., processor resources, memory resources, and/or the like) and/or communication resources by processing, generating, and providing such infeasible counterfactual explanations to the user") based on the holistic evaluation score for the target machine learning model (McGrath, [0034]: "The feedback data may indicate and/or include a label for the counterfactual explanation. For example, the feedback data may indicate whether the counterfactual explanation is useful or not useful, feasible or infeasible, relevant or irrelevant, and/or the like" and [0041]: "once a particular quantity or threshold percentage of counterfactual explanations associated with the qualification model are determined (e.g., 75% of counterfactual explanations are labeled, 90% of counterfactual explanations are labeled, 95% of counterfactual explanations are labeled, or the like). The labeled counterfactual explanations may be used to train a machine learning model to determine an optimal counterfactual explanation for a prediction output of the qualification model, as described elsewhere herein");
identifying, by the one or more processors, an influencing feature value corresponding to the performance degradation, wherein the influencing feature value is based on a counterfactual proposal that is generated by a counterfactual algorithm used to explain an individual predictive output during an inference phase once the target machine learning model is deployed (McGrath, [0040]: "the automated analysis system, during a training period, may iteratively receive pairs of user information and a prediction output of the qualification model, iteratively select one or more relevant counterfactual explanations ..., iteratively provide the selected counterfactual explanations for feedback, iteratively update labels based on iteratively received feedback data, and iteratively retrain the clustering model and/or the classification model according to the feedback data. In some implementations, each iteration may be associated with a different counterfactual explanation associated with a prediction output and/or a different analysis of different user information associated with different users"); and
modifying, by the one or more processors, a parameter of the target machine learning model based on the influencing feature value to improve a performance of the target machine learning model (McGrath, [0038]: "the automated analysis system retrains the counterfactual explanation generation model according to the feedback. For example, the automated analysis system may retrain the clustering model according to the updated labeled counterfactual explanation data structure," where McGrath's retraining reasonably suggests modifying a model parameter).
Regarding Claim 9, McGrath teaches:
A system comprising: one or more processors; and one or more memories storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations (McGrath, [0074]: "Device 500 may perform one or more processes described herein. For example, a non-transitory computer-readable medium (e.g., memory 530 and/or storage component 540) may store a set of instructions ( e.g., one or more instructions, code, software code, program code, and/or the like) for execution by processor 520. Processor 520 may execute the set of instructions to perform one or more processes described herein") comprising: precisely those steps recited by the method of Claim 1. Claim 9 is rejected under the same rationale as Claim 1.
Regarding Claim 17, McGrath teaches:
One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations (McGrath, [0074]: "Device 500 may perform one or more processes described herein. For example, a non-transitory computer-readable medium (e.g., memory 530 and/or storage component 540) may store a set of instructions ( e.g., one or more instructions, code, software code, program code, and/or the like) for execution by processor 520. Processor 520 may execute the set of instructions to perform one or more processes described herein") comprising: precisely those steps recited by the method of Claim 1. Claim 17 is rejected under the same rationale as Claim 1.
Regarding Claim 2, the rejection of Claim 1 is incorporated. McGrath teaches:
determining an updated holistic evaluation score for the target machine learning model (McGrath, [0037]: "a label of the counterfactual explanation can be updated in the data structure to increase a quantity of labeled counterfactual explanations in the labeled counterfactual explanation data structure" and [0041]: "The automated analysis system may iteratively perform the one or more processes until a quantity of counterfactual explanations in the labeled data structure satisfies a threshold and/or a ratio of a quantity of unlabeled data structures to the quantity of labeled data structures that satisfy a threshold").
Claims 10 and 18 incorporate substantively all the limitations of Claim 2 in computing apparatus and non-transitory computer-readable storage media forms, respectively, and are rejected under the same rationale.
Regarding Claim 4, the rejection of Claim 1 is incorporated. McGrath teaches:
wherein modifying the holistic evaluation score comprises reducing the holistic evaluation score (McGrath, [0014]: "the automated analysis system may conserve computing resources and/or communication resources associated with training an automated system by providing a most relevant set of counterfactual explanations for prediction outputs obtained during a training period, which reduces a quantity of needed feedback to improve the automated analysis system").
Claims 12 and 20 incorporate substantively all the limitations of Claim 4 in computing apparatus and non-transitory computer-readable storage media forms, respectively, and are rejected under the same rationale.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 5-8 and 13-16 are rejected under 35 U.S.C. 103 as being unpatentable over McGrath, et al. (US 2022/0129794 A1, hereinafter "McGrath") in view of Wan, et al., "Protein function prediction is improved by creating synthetic feature samples with generative adversarial networks" (hereinafter "Wan").
Regarding Claim 5, the rejection of Claim 1 is incorporated.
McGrath teaches augmenting a supplemental training dataset with the input data object and the ground truth label.
McGrath does not explicitly teach detecting a threshold augmentation stimulus based on the supplemental training dataset and in response to the threshold augmentation stimulus, generating an augmented training dataset by augmenting the training dataset with the supplemental training dataset.
Wan further teaches:
detecting a threshold augmentation stimulus based on the supplemental training dataset (Wan, p. 4: "The training quality of FFPred-GAN continues to improve with more iterations of training, with the LOOCV [Leave One Out Cross-Validation] accuracy reaching 0.515 after another 10,000 iterations. Finally, after 29,601 iterations’ training, FFPred-GAN has been successfully trained due to the desired LOOCV accuracy of 0.500," where Wan's desired LOOCV accuracy corresponds to the instant threshold); and
in response to the threshold augmentation stimulus, generating an augmented training dataset by augmenting the training dataset with the supplemental training dataset (Wan, p. 3, Results: "the FFPred-GAN framework consists of three steps to generate high-quality synthetic training protein feature samples.... On the last step, FFPred-GAN uses the Classifier Two-Sample Tests (CTST) [33] to select the optimal synthetic training protein feature samples, which are used to augment the original training samples").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of McGrath regarding augmenting a supplemental training dataset with the input data object and the ground truth label with those of Wan regarding detecting a threshold augmentation stimulus based on the supplemental training dataset and in response to the threshold augmentation stimulus, generating an augmented training dataset by augmenting the training dataset with the supplemental training dataset.
The motivation to do so would be to facilitate training of classifiers with higher predictive performance (Wan, p. 6, Results, Overview of FFPred-GAN: "the synthetic protein feature samples successfully improve the predictive performance of the original combination of training protein feature samples, and lead to the overall highest accuracy for predicting all three domains of GO terms with an SVM classification algorithm. For predicting Biological Process (BP) domain of GO terms, the combination of Synthetic Positive + Real Positive + Real Negative obtains the overall best average ranks of 5.88 and 4.84, respectively according to MCC and AUROC values by using SVM as the classification algorithm").
Claim 13 incorporates substantively all the limitations of Claim 5 in system form and is rejected under the same rationale.
Regarding Claim 6, the rejection of Claim 5 is incorporated. The McGrath/Wan combination teaches:
wherein the corresponding input feature value is associated with an evaluation feature of the training dataset, wherein the synthetic data object is associated with the evaluation feature (McGrath, Fig. 6, 670, "Receive feedback data associated with the request for feedback" and 680, "Update a data structure associated with the clustering model based on the feedback data and the counterfactual explanation to form an updated data structure"), wherein the synthetic data object is associated with the evaluation feature (McGrath, [0040]: "the automated analysis system, during a training period, may ... iteratively provide the selected counterfactual explanations for feedback, iteratively update labels based on iteratively received feedback data, and iteratively retrain the clustering model and/or the classification model according to the feedback data. In some implementations, each iteration may be associated with a different counterfactual explanation associated with a prediction output and/or a different analysis of different user information associated with different users"), and wherein augmenting the training dataset comprises:
replacing the synthetic data object with the supplemental training dataset (McGrath, [0035]: "the counterfactual explanation generation model may receive feedback to permit the counterfactual explanation generation model to update a label of the counterfactual explanation and/or retrain the one or more models according to the label").
Claim 14 incorporates substantively all the limitations of Claim 6 in system form and is rejected under the same rationale.
Regarding Claim 7, the rejection of Claim 5 is incorporated. The McGrath/Wan combination teaches:
in response to the performance degradation, retraining the target machine learning model using the augmented training dataset (McGrath, [0038]: "the automated analysis system retrains the counterfactual explanation generation model according to the feedback. For example, the automated analysis system may retrain the clustering model according to the updated labeled counterfactual explanation data structure," where McGrath's retraining reasonably suggests modifying a model parameter).
Claim 15 incorporates substantively all the limitations of Claim 7 in system form and is rejected under the same rationale.
Regarding Claim 8, the rejection of Claim 5 is incorporated. Wan further teaches:
wherein the threshold augmentation stimulus is based on a threshold number of supplemental input data objects in the supplemental training dataset (Wan, p. 4, Results, Overview of FFPred-GAN: "we adopt the 1-nearest neighbour classification algorithm and the Leave One Out Cross-Validation (LOOCV) to conduct the classifier two-sample tests, which is used for evaluating the quality of synthetic protein feature samples. The closer the value of LOOCV accuracy is to 0.500, the higher the quality of synthetic samples. ... [A]t the begin of FFPred-GAN training (i.e. after the 1st iteration), the real positive protein feature samples (green dots) are distributed distantly from the synthetic ones (red dots), leading to a LOOCV accuracy of 1.000, suggesting obvious differences between the real and synthetic sets of protein feature samples. ... Finally, after 29,601 iterations' training, FFPred-GAN has been successfully trained due to the desired LOOCV accuracy of 0.500" and p. 21: "the synthetic protein feature samples that obtain the best LOOCV accuracy (i.e. closest to 50.0%) are selected as the optimal synthetic feature samples," where Wan's desired 0.500 accuracy corresponds to the threshold number).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the McGrath/Wan combination regarding detecting a threshold augmentation stimulus with the further teachings of Wan regarding wherein the threshold augmentation stimulus is based on a threshold number of supplemental input data objects in the supplemental training dataset.
The motivation to do so would be to ensure generating samples of sufficient quality to improve prediction accuracy of trained models (Wan p. 4, Results, Overview of FFPred-GAN: "The closer the value of LOOCV accuracy is to 0.500, the higher the quality of synthetic samples" and p. 18, Discussion: "we have presented a novel generative adversarial networks-based method that successfully generates high-quality synthetic feature samples, which significantly improve the accuracy on predicting all three domains of GO terms through augmenting the original training data").
Claim 16 incorporates substantively all the limitations of Claim 8 in system form and is rejected under the same rationale.
Claims 21-23 are rejected under 35 U.S.C. 103 as being unpatentable over McGrath, et al. (US 2022/0129794 A1, hereinafter "McGrath") in view of Mothilal, et al., "Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations" (hereinafter "Mothilal").
Regarding Claim 21, the rejection of Claim 1 is incorporated.
McGrath teaches identifying an influencing feature value based on a counterfactual proposal that is generated by a counterfactual algorithm.
McGrath does not explicitly teach wherein the counterfactual algorithm comprises one or more of: DICE, CCHVAE, CEM, CLUE, CRUDS, FACE, Growing Spheres, Revise, or Wacter.
However, Mothilal teaches:
wherein the counterfactual algorithm comprises one or more of: DICE, CCHVAE, CEM, CLUE, CRUDS, FACE, Growing Spheres, Revise, or Wacter (Mothilal, p. 3, 3 Counterfactual Generation Engine: "we incorporate feasibility using the proximity constraint from Wachter et al. [39] and introduce other user-defined constraints" and p. 6, 4.4 Baselines: "We employ the following baselines for generating CF [counterfactual] examples. • SingleCF: We follow Wachter et al. [39] and generate a single CF example, optimizing for y-loss difference and proximity").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of McGrath regarding teaches identifying an influencing feature value based on a counterfactual proposal that is generated by a counterfactual algorithm with those of XXX regarding wherein the counterfactual algorithm comprises one or more of: DICE, CCHVAE, CEM, CLUE, CRUDS, FACE, Growing Spheres, Revise, or Wacter.
The motivation to do so would be to facilitate providing counterfactual values that reflect the prevalence of features' observed values (Mothilal, p. 4, 3.3 Practical considerations, Choice of distance function: "For continuous features, we define
d
i
s
t
as the mean of feature-wise
l
1
distances between the CF example and the original input. Since features can span different ranges, we divide each feature-wise distance by the median absolute deviation (MAD) of the feature’s values in the training set, following Wachter et al. [39] Deviation from the median provides a robust measure of the variability of a feature’s values, and thus dividing by the MAD allows us to capture the relative prevalence of observing the feature at a particular value").
Claims 22 and 23 incorporate substantively all the limitations of Claim 21 in system and non-transitory computer-readable storage media forms, respectively, and are rejected under the same rationale.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT N DAY whose telephone number is (703)756-1519. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.N.D./Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights