Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This Non-Final Rejection is filed in response to Request for Continued Examination (RCE) filed 09/30/2025.
Claims 1-3, 8, 11, 12, 14, 15, 19, and 20 are amended.
Claims 1-20 remain pending.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim 1 rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the enablement requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to enable one skilled in the art to which it pertains, or with which it is most nearly connected, to make and/or use the invention. Regarding the claim 1 limitations, “configuring the second machine learning model for evaluating the effectiveness of adding the one or more of the set of candidate features to the first set of features for performing the task wherein the second machine learning model is configured to accept input values corresponding to a set of input features that comprise an output of the first machine learning model and the set of candidate features but excludes the first set of features”, it is unclear how the second machine learning model may evaluate the effectiveness of adding the one or more of the set of candidate features to the first set of features without while still excluding the first set of features. It is not certain to what degree are the first features excluded as even using performance results of a first machine learning model trained on the first set of features might still be construed as included the first set of features. The specification provides no explicit definition for how a first set of features may be excluded while still including the output of the first machine learning model. For the purposes of examination the limitation, “configuring the second machine learning model for evaluating the effectiveness of adding the one or more of the set of candidate features to the first set of features for performing the task wherein the second machine learning model is configured to accept input values corresponding to a set of input features that comprise an output of the first machine learning model and the set of candidate features but excludes the first set of features”, is being interpreted as configuring a second machine learning model to intake performance results from a first and second set of features and not literally intaking the first set of features.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being incomplete for omitting essential structural cooperative relationships of elements, such omission amounting to a gap between the necessary structural connections. See MPEP § 2172.01. The omitted structural cooperative relationships are: “configuring the second machine learning model for evaluating the effectiveness of adding the one or more of the set of candidate features to the first set of features… wherein the second machine learning model is configured to accept input values corresponding to a set of input features that comprise an output of the first machine learning model and the set of candidate features but excludes the first set of features”. It is unclear how exactly the first set of features are excluded from a model that is configured to evaluate adding more features to the same first set of features.
Response to Arguments
Argument 1, Applicant argues in Applicant Arguments/Remarks Made in an Amendment filed 09/30/2025, pg. 9-12, that the proposed amendments overcome the 35 U.S.C. 101 rejections.
Response to Argument 1, in light of the amendments the 35 U.S.C. 101 rejections are respectfully withdrawn
Argument 2, Applicant argues in Applicant Arguments/Remarks Made in an Amendment filed 09/30/2025, pg. 13-14, that prior art fails to teach the primary claim limitations of, “configuring the second machine learning model for evaluating the effectiveness of adding the one or more of the set of candidate features to the first set of features for performing the task wherein the second machine learning model is configured to accept input values corresponding to a set of input features that comprise an output of the first machine learning model and the set of candidate features but excludes the first set of features”.
Response to Argument 2, the examiner respectfully disagrees. Asthana teaches a second machine learning model that compares the feature importance of a first and second group of features, wherein the second group of features may be a new group of features consisting of augmented or substituted features from a first group of features. Thus the BRI for the limitation, “wherein the second machine learning model is configured to accept input values corresponding to a set of input features that comprise an output of the first machine learning model and the set of candidate features”, encompasses how the versioning service configures a second model on a new or updated second dataset and compares the feature importance to a first feature dataset, in this way the second comparison model ingests the output of a first model trained on a first dataset and not the whole first set of features.
This is supported by the following paragraphs of Asthana
para. [0075-0080], “Embodiments of the feature extraction module 213 may extract feature importance f.sub.1 of the trained model using one or more explainable AI, algorithms, or techniques …The feature extraction module 213 may pre-process a new set of features (f.sub.2) extracted from the new or evolved dataset
para. [0075-0080], “Embodiments of the feature extraction module 213 may extract feature importance f.sub.1 of the trained model using one or more explainable AI, algorithms, or techniques …The feature extraction module 213 may pre-process a new set of features (f.sub.2) extracted from the new or evolved dataset.
While Asthana teaches a model for comparing outputs of a first and second feature dataset, Asthana may not explicitly teach the second machine learning model for evaluating effectiveness,
excludes the first set of features
However, Sathe teaches a comparison model that compares a first and then a new second set of heartbeat features in order to compare the predicted performances. In this way the comparison model excludes a first set of heart beat features by ingesting a new and separate second group of heartbeat features and compares the performance predictions to a first.
This is supported by the following paragraphs of Sathe
para. [0035], the joint optimizer 132 is capable of applying the trained model to a new set of heartbeat features, pipeline features, and dataset features in order to output the predicted performance measures for each worker node with respect to training a pipeline. The joint optimizer 132 may then tweak the models based on comparing the predicted performance measures to the actual performance measures, as will be described in greater detail forthcoming.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5-8, 11-13, 15, 16, & 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication NO. 20230144585 “Asthana” and further in light of U.S. Patent Application Publication NO. 20220114019 “Sathe”.
Claim 1:
Asthana teaches a system, comprising:
a non-transitory memory (i.e. para. [0034] Fig. 1, memory 105); and one or more hardware processors coupled with the non-transitory memory and configured to read instructions from the non-transitory memory (i.e. para. [0035], Program(s) 114, application(s), processes, services, and installed components thereof, described herein, may be stored in memory 105) to cause the system to perform operations comprising:
determining, for a first machine learning model configurating to perform a task based on a first set of features (i.e. para. [0093], “ingestion module 211 of the versioning service 201 may ingest the first dataset used to train the model… A feature extraction module 213 of the versioning service 201 may perform feature exploration and extract feature importance, f.sub.1”, wherein the BRI for a first machine learning model encompasses the model trained to find an importance of a first set of features f.sub.1), a set of candidate features usable for performing the task, wherein the set of candidate features is not included in the first set of features (i.e. para. [0097], “wherein the second dataset is found to comprise new data having a new feature set with attributes not previously found in the feature set of f.sub.1.”, wherein the BRI for candidate features encompasses the new set of features not part of the first second dataset f.sub.1);
generating a second machine learning model for evaluating an effectiveness of adding one or more of the set of candidate features to the first set of features for performing the task (i.e. para. [0083], “the comparison module 215 may refer to a common geographical ontology to compute whether the new feature of f.sub.2 is correlated with old features of set f.sub.1 which may be considered a “revision” of the participating country list… Moreover, the use of feature substitution may be performed even if statistically the new feature of set f.sub.2 is closely overlapping the prior feature of set f.sub.1 (i.e., only one country added or removed from the treaty membership).”, wherein the BRI for generating a second machine learning model encompasses a model is generated that may add new features to f.sub.1 to get an augmented dataset f.sub.2 that is then evaluated and compared between as to f.sub.1 for the same task of evaluating importance); configuring the second machine learning model for evaluating the effectiveness of adding the one or more of the set of candidate features to the first set of features for performing the task (i.e. para. [0094], “versioning service 201 may receive and/or ingest a new or updated dataset (i.e., a second dataset)… 1. Embodiments of the versioning service 201 may pre-process a new set of features (f.sub.2) from the new or evolved second dataset”, wherein the BRI for configuring the second machine learning model encompasses the preprocessing of feature set f.sub.2 to perform the same task as the machine trained on f.sub.1, wherein f.sub.2 may have new features evaluated for importance) wherein the second machine learning model is configured to accept input values corresponding to a set of input features that comprise an output of the first machine learning model and the set of candidate features (i.e. para. [0075-0080], “Embodiments of the feature extraction module 213 may extract feature importance f.sub.1 of the trained model using one or more explainable AI, algorithms, or techniques …The feature extraction module 213 may pre-process a new set of features (f.sub.2) extracted from the new or evolved dataset”, wherein the BRI for input values corresponding to a set of input features encompasses how the second MLM inputs feature importance values from the first MLM that uses the first set of features and compares it to the f.sub.2 that contains additional new features to determine the effectiveness of the feature sets)
determining a difference in a prediction performance associated with the task between the first machine learning model and the second machine learning model (i.e. para. [0088], “The top features were identified and the delta changes between (f.sub.1, f.sub.2) were found. The delta changes involve new invoice amounts for the contract.. Since the cosine similarity and vector distance between invoice amount and our target variable risk score was 0.8, which is near a high correlation score, the recommendation engine outputs retraining the model to include the new invoices for the contracts”, wherein a difference of an increase in accuracy is found when features f.sub.2 are included in a second machine learning model); and
modifying the first machine learning model based on the difference, wherein the modifying comprises incorporating one or more features from the set of candidate features as input features for the first machine learning model (i.e. para. [0083], the versioning service 201 may discover that feature augmentation may be more appropriate than feature substitution, whereby useful information of the model dataset is combined with new features of the new dataset which may lead to improved predictions and performance once the model is re-trained using the augmented dataset).
While Asthana teaches comparing two features sets and modifying a first machine learning model based on the comparison, Asthana may not explicitly teach wherein the second machine learning model
excludes the first set of features;
However, Sathe teaches
wherein the second machine learning model is configured to accept input values corresponding to a set of input features that comprise an output of the first machine learning model and the set of candidate features but excludes the first set of features (i.e. para. [0035], “Once the model is trained, the joint optimizer 132 is capable of applying the trained model to a new set of heartbeat features, pipeline features, and dataset features in order to output the predicted performance measures for each worker node with respect to training a pipeline. The joint optimizer 132 may then tweak the models based on comparing the predicted performance measures to the actual performance measures, as will be described in greater detail forthcoming”, wherein the BRI for a second MLM encompasses how a new model may input a completely new set of heartbeat features, different from a first set of features, and compare the performance of the new candidate set of features to the output predicted performance of a first set of different heartbeat features).
It would have been obvious to one of ordinary skill in the art at the time of filing to add wherein the second machine learning model excludes the first set of features, to Asthana’s feature set versioning for machine learning model training, with a different set of features, not included in a first set of features, may be evaluated and comparted to a first set of features, as taught by Sathe. One would have been motivated to combine Sathe and Asthana in order help a user understand the differences in performance and would allow for more intelligence allocation of resources.
Claim 2:
Asthana and Sathe teach the system of claim 1.
Asthana further teaches wherein the operations further comprise:
Generating a training data set for training the second machine learning model, based on data corresponding to a transaction (i.e. para. [0069], “Examples of workloads and functions which may be provided from this layer include software development and lifecycle management 491, data analytics processing 492, multi-cloud management 493, transaction processing 494”, wherein features may be corresponding to a processed transaction), wherein the generating the training dataset comprises obtaining an output value from the first machine learning model based on providing a first portion of the data corresponding to the first set of features to the first machine learning mode (i.e. para. [0074], Feature importance may refer to a class of techniques for assigning scores to input features found in datasets of predictive models. The assigned scores corresponding to feature importance indicate the relative importance of each feature when a prediction is made by the model. Feature importance scores may be calculated for problems that can involve predicting a numerical value) wherein the training data set comprises
the output value obtained from the first machine learning model (i.e. para. [0074], “Feature importance scores may be calculated for problems that can involve predicting a numerical value (i.e., referred to as a regression) and problems that may involve predicting a class label (i.e., classification)”, wherein the BRI for output values encompasses feature values in the form of contract duration, billing frequency, and the like) and data values corresponding to the set of candidate features (i.e. para. [0019], The versioning service finds the changes (referred to as the “delta”) between the top n or n % features of f.sub.1 and the features of f.sub.2)
a label indicating an actual result (i.e. para. [0098], “This feature set was labeled as f.sub.1. The target variable was the risk score of the contract”, wherein the BRI for a label indicating an actual result encompasses an assigned risk score for a risk analytic model trained on feature set f.sub.1); and
training the second machine learning model using the training data set (i.e. para. [0088], “The Pearson correlation metrics algorithm outputs the correlation of invoice amounts with the feature set f.sub.2. Since the cosine similarity and vector distance between invoice amount and our target variable risk score was 0.8, which is near a high correlation score, the recommendation engine outputs retraining the model to include the new invoices for the contracts”, wherein a risk analytic model is trained the same invoice data values in addition to additional invoices received from the contracts assigned as feature set f.sub.2. A comparison is made correlating the first and second feature set based on results from generated feature data part of training a machine learning model as a risk analytic model)
Claim 3:
Asthana and Sathe teach the system of claim 1.
Asthana further teaches wherein the operations further comprises:
Accessing data associated with a transaction (i.e. para. [0069], “Examples of workloads and functions which may be provided from this layer include software development and lifecycle management 491, data analytics processing 492, multi-cloud management 493, transaction processing 494”, wherein features may be corresponding to a processed transaction);
Obtaining a first output value from the first machine learning model based on providing a first portion of the data corresponding to the first set of features to the first machine learning model (i.e. para. [0093], feature extraction module 213 of the versioning service 201 may perform feature exploration and extract feature importance, f.sub.1);
Obtaining a second output value from the second machine learning model based on providing the first output value and a second portion of the data corresponding to the set of candidate features to the second machine learning model (i.e. para. [0094], Embodiments of the versioning service 201 may pre-process a new set of features (f.sub.2) from the new or evolved second dataset); and
comparing the first output value against the second output value (i.e. para. [0094], Using the feature set f.sub.2, in step 509, the difference (the delta) between f.sub.1 and f.sub.2 can be found for the top n or n % extracted features of f.sub.1. In step 511, a determination may be made whether the second dataset is a new dataset comprising feature set f.sub.2 with attributes that are not previously present within the feature set of f.sub.1)
Claim 5:
Asthana and Sathe teach the system of claim 1.
Asthana further teaches wherein the modifying the first machine learning model comprises:
re-configuring the first machine learning model to perform the task based on a second set of input features comprising the first set of features and the one or more features from the set of candidate features (i.e. para. [0083], the versioning service 201 may discover that feature augmentation may be more appropriate than feature substitution, whereby useful information of the model dataset is combined with new features of the new dataset which may lead to improved predictions and performance once the model is re-trained using the augmented dataset).
Claim 6:
Asthana and Sathe teach the system of claim 1.
Asthana further teaches wherein the set of candidate features is a first set of candidate features and wherein the operations further comprise:
determining a second set of candidate features usable for performing the task, wherein the second set of candidate features is different from the first set of candidate features; configuring a third machine learning model to perform the task based on a third set of input features comprising the output of the first machine learning model and the second set of candidate features; and determining a second difference in prediction performance between the second machine learning model and the third machine learning model, wherein the modifying the first machine learning model is further based on the second difference (i.e. para. [0043], “detecting changes in feature between datasets used by the machine learning models, and recommending whether or not to re-train machine learning models depending on whether concept drift is detected that might render the machine learning model less accurate or obsolete if re-training does not occur”, wherein the versioning service may repeat the process of determining a different and second set of candidate features, configuring a model with a different set of features, and recommending whether or not to re-train machine learning models in a case where drift is applicable to the model it is exposed to new data as time goes on).
Claim 7:
Asthana and Sathe teach the system of claim 6.
Asthana further teaches
wherein the operations further comprise: determining that the third machine learning model has a higher accuracy performance than the second machine learning model associated with performing the task (i.e. para. [0083], “useful information of the model dataset is combined with new features of the new dataset which may lead to improved predictions and performance once the model is re-trained using the augmented dataset”, wherein the process may be repeated); and re-configuring the first machine learning model to perform the task based on the first set of features and the one or more features from the set of candidate features (i.e. para. [0016], “To pursue a model with high fidelity and accuracy, machine learning models may need to be re-trained and exposed to the machine learning pipeline if new data has emerged or evolved over time and may need to be analyzed alongside historical data, to account for both long-term and short-term trends of the data”, wherein in a case where a first model is not recommended for retraining using a second feature set but undergoes a recommendation to augment the model with a third set of features to create an improved third model).
Claim 8:
Asthana teaches a method, comprising:
Determining, by one or more hardware processors (i.e. para. [0035], Program(s) 114, application(s), processes, services, and installed components thereof, described herein, may be stored in memory 105) and for a first machine learning model configured to perform a task based on a first set of input features , a set of candidate input features usable for performing the task (i.e. para. [0093], “ingestion module 211 of the versioning service 201 may ingest the first dataset used to train the model… A feature extraction module 213 of the versioning service 201 may perform feature exploration and extract feature importance, f.sub.1”, wherein a first set of input features f.sub.1 are different than a candidate set of features f.sub.2),
generating, by the one or more hardware processors, a second machine learning model for evaluating the set of candidate input features (i.e. para. [0075-0080], “Embodiments of the feature extraction module 213 may extract feature importance f.sub.1 of the trained model using one or more explainable AI, algorithms, or techniques …The feature extraction module 213 may pre-process a new set of features (f.sub.2) extracted from the new or evolved dataset”, wherein the BRI for a second machine learning model encompasses a model that uses an explainable AI generate a second evaluation of a candidate set of features f.sub.2 for the same task);
configuring, by the one or more hardware processors, the second machine learning model to perform the task based on a second set of input features, wherein the second set of input features comprises (i) a first input feature corresponding to an output of the first machine learning model (i.e. para. [0075-0080], “Embodiments of the feature extraction module 213 may extract feature importance f.sub.1 of the trained model using one or more explainable AI, algorithms, or techniques …The feature extraction module 213 may pre-process a new set of features (f.sub.2) extracted from the new or evolved dataset”, wherein the BRI for an output of the first MLM encompasses how the second MLM inputs feature importance values from the first MLM that uses the first set of features and compares it to the f.sub.2 that contains additional new features to determine the effectiveness of the feature sets) and (ii) the set of candidate input features (i.e. para. [0094], “versioning service 201 may receive and/or ingest a new or updated dataset (i.e., a second dataset)… 1. Embodiments of the versioning service 201 may pre-process a new set of features (f.sub.2) from the new or evolved second dataset”, wherein the BRI for configuring the second machine learning model encompasses the preprocessing of feature set f.sub.2 to perform the same task as the machine trained on f.sub.1, wherein f.sub.2 may be based on an updated dataset of f.sub.1);
determining, by the one or more hardware processors, a first performance improvement associated with the task of the second machine learning model over the first machine learning model (i.e. para. [0088], “The top features were identified and the delta changes between (f.sub.1, f.sub.2) were found. The delta changes involve new invoice amounts for the contract.. Since the cosine similarity and vector distance between invoice amount and our target variable risk score was 0.8, which is near a high correlation score, the recommendation engine outputs retraining the model to include the new invoices for the contracts”, wherein a difference of an increase in accuracy is found when features f.sub.2 are included in a second machine learning model); and
modifying, by the one or more hardware processors, the first machine learning model based on the first performance improvement, wherein the modifying comprises incorporating one or more input features from the set of candidate input features into the first set of input features for the first machine learning model (i.e. para. [0083], the versioning service 201 may discover that feature augmentation may be more appropriate than feature substitution, whereby useful information of the model dataset is combined with new features of the new dataset which may lead to improved predictions and performance once the model is re-trained using the augmented dataset).
While Asthana teaches comparing two features sets and modifying a first machine learning model based on the comparison, Asthana may not explicitly teach
wherein the set of candidate features is not included in the first set of features
However, Sathe teaches
wherein the set of candidate input features is not included in the first set of input features (i.e. para. [0035], “. Once the model is trained, the joint optimizer 132 is capable of applying the trained model to a new set of heartbeat features, pipeline features, and dataset features in order to output the predicted performance measures for each worker node with respect to training a pipeline. The joint optimizer 132 may then tweak the models based on comparing the predicted performance measures to the actual performance measures, as will be described in greater detail forthcoming” wherein the new set of heartbeat features may be from a different worker node and thus not be included in a first set of heartbeat features).
It would have been obvious to one of ordinary skill in the art at the time of filing to add wherein the set of candidate features is not included in the first set of features, to Asthana’s feature set versioning for machine learning model training, with a different set of features, not included in a first set of features, may be evaluated and comparted to a first set of features, as taught by Sathe. One would have been motivated to combine Sathe and Asthana in order help a user understand the differences in performance and would allow for more intelligence allocation of resources.
Claim 11:
Asthana and Sathe teach the method of claim 8.
Asthana further teaches further comprising:
determining that the first performance improvement exceeds a benchmark (i.e. para. [0020], “the versioning service 201 may discover that feature augmentation may be more appropriate than feature substitution, whereby useful information of the mode”, wherein the BRI for a benchmark encompasses when new features of the new dataset which may lead to improved predictions and performance), wherein the modifying the first machine learning model is in response to determining that the first performance improvement exceeds the benchmark (i.e. para. [0083], “useful information of the model dataset is combined with new features of the new dataset which may lead to improved predictions and performance once the model is re-trained using the augmented dataset”, wherein new features are incorporated in response to benchmarks such as prediction accuracy are improved).
Claim 12:
Asthana and Sathe teach the system of claim 8.
Asthana further teaches wherein the set of candidate input features is a first set of input features and wherein the operations further comprise:
determining a second set of candidate input features usable for performing the task, wherein the second set of candidate input features is different from the first set of candidate input features; configuring a third machine learning model to perform the task based on a third set of input features comprising the output of the first machine learning model and the second set of candidate input features; and determining a second performance improvement associated with the task of the third machine learning model over the first machine learning model, wherein the modifying the first machine learning model is further based on the second performance improvement (i.e. para. [0043], “detecting changes in feature between datasets used by the machine learning models, and recommending whether or not to re-train machine learning models depending on whether concept drift is detected that might render the machine learning model less accurate or obsolete if re-training does not occur”, wherein the versioning service may repeat the process of determining a different and second set of candidate features, configuring a model with a different set of features, and recommending whether or not to re-train machine learning models in a case where drift is applicable to the model it is exposed to new data as time goes on).
Claim 13:
Asthana and Sathe teach the method of claim 12.
Asthana further teaches
determining that the first performance improvement is greater than the second performance improvement, wherein the modifying the first machine learning model is further based on the determining that the first performance improvement is greater than the second performance improvement (i.e. para. [0083], “the useful information of the model dataset is combined with new features of the new dataset which may lead to improved predictions and performance once the model is re-trained using the augmented dataset”, wherein any features resulting in an improved prediction accuracy results in an incorporation of new features and re-training)
Claim 15:
Claim 15 is the medium claim reciting similar claims to Claim 1 and is rejected for similar reasons.
Claim 16:
Claim 16 is the medium claim reciting similar claims to Claim 2 and is rejected for similar reasons.
Claim 19:
Claim 19 is the medium claim reciting similar claims to Claim 5 and is rejected for similar reasons.
Claim 20:
Claim 20 is the medium claim reciting similar claims to Claim 6 and is rejected for similar reasons.
Claim(s) 4, 9-10, 17-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication NO. 20230144585 “Asthana” and further in light of U.S. Patent Application Publication NO. 20220114019 “Sathe”, as applied to Claim 1, 8, and 15 above, and further in light of U.S. Patent Application Publication NO. 20200250477 “Barthur”.
Claim 4:
Asthana and Sathe teach the system of claim 1.
Asthana may not explicitly teach,
wherein the determining the difference in the prediction performance comprises: determining that the second machine learning model has a lower false negative rate than the first machine learning model.
However Barthur teaches
wherein the determining the difference in the prediction performance comprises: determining that the second machine learning model has a lower false negative rate than the first machine learning model (i.e. para. [0080], the trained and validated machine learning model with the highest number of true positives, the highest number of true negatives, the lowest number of false positives, and/or the lowest number of false negatives may be selected. A top tier of trained and validated machine learning models may be selected (e.g., top 5 models, top 5% models, etc.)).
It would have been obvious to one of ordinary skill in the art at the time of filing to add wherein the determining the difference in prediction performance comprises: determining that the second machine learning model has a lower false negative rate than the first machine learning model, to Asthana-Sathe’s feature set versioning for machine learning model training, with how a best performing machine learning model is chosen, as taught by Barthur. One would have been motivated to combine Barthur and Asthana-Sathe in order help a user produce an up to date and accurate model by accurately classifying behavior may and applying one or more remediation measures that may resolve or mitigate the damage.
Claim 9:
Asthana and Sathe teach the method of claim 8.
Asthana teaches
further comprising: determining a first set of performance metrics for the first machine learning model (i.e. para. [0075], “This feature set was labeled as f.sub.1. The target variable was the risk score of the contract”, wherein the performance metric determined encompasses a risk score for a model with feature set f.sub.1); and determining a second set of performance metrics for the second machine learning model (i.e. para. [0088], “additional invoices were received for the contracts from which the features f.sub.2 were extracted… Sample risk analytics were outputted for a contract if the recommendation for merging additional features was not taken into consideration. We observed that the actual risk varies from the predicted risk in the last few billing cycles”, wherein a first and second set of performance metrics, in the form of a predicted risk score, are compared),
While Asthana teaches determining a difference in predicted performance (i.e. para. [0083], the versioning service 201 may discover that feature augmentation may be more appropriate than feature substitution, whereby useful information of the model dataset is combined with new features of the new dataset which may lead to improved predictions and performance once the model is re-trained using the augmented dataset).
While Asthana teaches determining a difference in predicted performance, Asthana may not explicitly teach
wherein the first and second sets of performance metrics comprise at least one of a false positive rate, a false negative rate, or a catch count.
However, Barthur teaches
wherein the first and second sets of performance metrics comprise at least one of a false positive rate, a false negative rate, or a catch count (i.e. para. [0079], Fig. 4, “One or more corresponding performance metrics may be determined for each of the trained and validated supervised machine learning models. For example, a number of false positives predicted by the one or more trained supervised machine learning models… a percentage of the total predictions that are false positives… the one or more performance metrics may be compared to one or more corresponding performance thresholds”, wherein it is noted that a first supervised model is trained in 408, thus the BRI for a rate encompasses a percentage of total predictions that are false positives); It would have been obvious to one of ordinary skill in the art at the time of filing to add wherein the first and second sets of performance metrics comprise at least one of a false positive rate, a false negative rate, or a catch count, to Asthana-Sathe’s feature set versioning for machine learning model training, with how a comparison of performance metrics, such as a false positive rate, may be used to select a first or second machine learning model, as taught by Barthur. One would have been motivated to combine Barthur and Asthana-Sathe in order help a user produce an up to date and accurate model by accurately classifying behavior may and applying one or more remediation measures that may resolve or mitigate the damage.
Claim 10:
Asthana, Sathe, and Barthur teach the method of claim 9.
Barthur further teaches wherein the determining the first performance improvement comprises: determining a difference between the first set of performance metrics and the second set of performance metrics (i.e. para. [0080], the trained and validated machine learning model with the highest number of true positives, the highest number of true negatives, the lowest number of false positives, and/or the lowest number of false negatives may be selected. A top tier of trained and validated machine learning models may be selected (e.g., top 5 models, top 5% models, etc.)).
Claim 17:
Asthana and Sathe teach the non-transitory machine-readable medium of claim 15.
While Asthana teaches determining a difference in the predicted performance (i.e. para. [0083], the versioning service 201 may discover that feature augmentation may be more appropriate than feature substitution, whereby useful information of the model dataset is combined with new features of the new dataset which may lead to improved predictions and performance once the model is re-trained using the augmented dataset), Asthana may not explicitly teach wherein the determining the difference in the prediction performance comprises:
determining a first false positive rate associated with the first machine learning model based on a set of testing data; determining a second false positive rate associated with the second machine learning model based on the set of testing data; and comparing the first false positive rate against the second false positive rate.
However, Barthur teaches
wherein the determining the difference in the prediction performance comprises:
determining a first false positive rate associated with the first machine learning model based on a set of testing data (i.e. para. [0079], Fig. 4, “One or more corresponding performance metrics may be determined for each of the trained and validated supervised machine learning models. For example, a number of false positives predicted by the one or more trained supervised machine learning models… a percentage of the total predictions that are false positives… the one or more performance metrics may be compared to one or more corresponding performance thresholds”, wherein it is noted that a first supervised model is trained in 408, thus the BRI for a rate encompasses a percentage of total predictions that are false positives); determining a second false positive rate associated with the second machine learning model based on the set of testing data (i.e. para. [0076], Fig. 4, “At 412, a labeled version of the second selected subset is used to train a second supervised machine learning model.”, wherein it is noted that a second supervised model is trained on the determined gradient values and a second set of values in 412, and that for each of the trained models a false positive percentage is found); and comparing the first false positive rate against the second false positive rate (i.e. para. [0078], Fig. 4, “At 414, a trained supervised machine learning model is selected among a group of trained supervised machine learning models based on one or more corresponding performance metrics”, wherein the first and second machine learning models may compare their performance metrics, such as their false positive rates, resulting in the selection of a model based on such a performance metric comparison).
It would have been obvious to one of ordinary skill in the art at the time of filing to add determining a first false positive rate associated with the first machine learning model based on a set of testing data; determining a second false positive rate associated with the second machine learning model based on the set of testing data; and comparing the first false positive rate against the second false positive rate, to Asthana-Sathe’s feature set versioning for machine learning model training, with how a comparison of performance metrics, such as a false positive rate, may be used to select a first or second machine learning model, as taught by Barthur. One would have been motivated to combine Barthur and Asthana-Sathe in order help a user produce an up to date and accurate model by accurately classifying behavior may and applying one or more remediation measures that may resolve or mitigate the damage.
Claim 18:
Claim 18 is the medium claim reciting similar claims to Claim 4 and is rejected for similar reasons.
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication NO. 20230144585 “Asthana” and further in light of U.S. Patent Application Publication NO. 20220114019 “Sathe”, as applied to Claim 12 above and further in light of U.S. Patent Application Publication NO. 20220114390 “Baughman”.
Claim 14:
Asthana teaches the method of claim 12.
While Asthana teaches combining a first and third set of features (i.e. para. [0083], the versioning service 201 may discover that feature augmentation may be more appropriate than feature substitution, whereby useful information of the model dataset is combined with new features of the new dataset), Asthana may not explicitly teach further comprising: encoding the first set of features and the third set of features into a common multi- dimensional space.
However, Baughman teaches
encoding the first set of features and the third set of features into a common multi-dimensional space (i.e. para. [0035], “Another embodiment uses two source vectors, and two trained encoder networks, each encoding a source vector into a multidimensional feature vector. The embodiment combines, or concatenates, the two resulting vectors into one larger vector, by appending one vector to the other. The resulting combined feature vector has double the number of dimensions as each individual source vector”, wherein the BRI for a first and third set of features encompasses an embodiment that may use a set of training data comprising a first and third set of feature vectors and encode these sets of source vectors into a multidimensional vector).
It would have been obvious to one of ordinary skill in the art at the time of filing to add encoding the first set of features and the third set of features into a common multi-dimensional space, to Asthana-Sathe’s feature set versioning for machine learning model training, with how a set of features may be combined into a multidimensional vector, as taught by Baughman. One would have been motivated to combine Baughman and Asthana-Sathe in order ensure that features are properly encoded into a multidimensional space, as the model can often achieve higher accuracy in prediction tasks by considering the nuances within the data.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
U.S. Patent Application Publication NO. 20200388397 “Shoaran” in para. [0149], Following model selection, we evaluated the predictive performance of features for the top performing classifier. The algorithm first evaluates all single-feature subsets to find the most predictive biomarker. The performance of the previous subset combined with a new element from the remaining feature set is investigated to find the next “best feature”, using F1 score.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID H TAN whose telephone number is (571)272-7433. The examiner can normally be reached M-F 7:30-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571) 272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/D.T./Examiner, Art Unit 2145
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145