Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The object regarding to drawing and claims 1, 4-7, 11-15, and 19-20 are withdrawn.
The rejection related to 35 USC § 112 regarding to 1 is withdrawn.
The rejection related to 35 USC § 101 regarding to 1-20 is withdrawn.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 5-7, 13-15, 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (“Methods and Systems for Detection and Isolation of Bias in Predictive Models”—U.S. Patent No. 11,593,648; hereinafter “Lee”) in view of Grossman US 12165017
Regarding claim 1
Regarding claim 1, Lee teaches a system for detecting bin-specific model bias and presenting of alternative predictions, the system comprising:
one or more processors; and one or more non-transitory computer-readable storage media storing instructions, which when executed by the one or more processors cause the one or more processors to (paragraph [0129], “The computing system 1000 executes program code that configures the processor 1102 to perform one or more of the operations described herein. The program code may be resident in the memory device 1008 or any suitable computer-readable medium and may be executed by the processor 1004 or any other suitable processor”; “memory device 1008” implies the use of tangible hardware, therefore a non-transitory computer-readable storage medium);
subsequent to a machine learning model being trained using a training dataset, generate a plurality of bins for a compare dataset based on a parameter within the compare dataset, wherein each bin of the plurality of bins is associated with a different range of parameter values for the parameter (paragraph [0066]-[0067], “A block 304, a computing device receives data associated with a trained predictive model. The data includes data output from the predictive model such as predictions associated with a class or a subclass. In some instances, the data includes the input data that was input to the predictive model to generate the output and training data used during a training phase of the predictive model… At block 308, the computing device identifies a feature group from the data”; paragraph [0054], “A feature group is made up of one or more features and represents a subclass of the class that includes members that each include characteristics that correspond to the one or more features”; feature groups can be considered analogous to parameters, and they may contain one or more features that intuitively correspond to bins of values);
for each bin of the plurality of bins: obtain a training set of entries corresponding to a bin from the training dataset and a compare set of entries corresponding to the bin from the compare dataset (paragraph [0042], “In some instances, model test node 104 receives a seed from the training datasets 132… The model test node 104 uses the seed to procedurally generate any amount of training data that corresponds to the predictive model. The seed may be combined with a particular feature or feature group to generate training data that corresponds to that feature or feature group”; paragraph [0038], “For instance, the bias detection module 124 enumerates a set of features by identifying types of predictions that are generated, characteristics of a class (e.g., employees) that are the subject of the generated predictions, and/or statistically relevant aspects of the training data 208 or input data 212 such as data types, data structures, reoccurring data values, and the like”; input data is analogous to the compare dataset in this example; model test node generates (“obtains”) training data and bias detection module takes input data from feature sets);
input the training set of entries corresponding to the bin into the machine learning model to obtain a first set of residual values, the first set of residual values outputted by the machine learning model based on the machine learning model's processing of the training set of entries (paragraph [0060], “The results of bias detection module 220 includes, but is not limited to, an identification of a bias associated with a feature group, a performance characteristic of the predictive model, an error, combinations thereof, an identification of a cause of the bias or error such as a characteristic of the predictive model 204, a portion of the training data 208, a portion of the input data 212, combinations thereof, or the like”; training data given to predictive model and error is output); and
input the compare set of entries into the machine learning model to obtain a second set of residual values, the second set of residual values outputted by the machine learning model based on the machine learning model's processing of the compare set of entries (paragraph [0060], “The results of bias detection module 220 includes, but is not limited to, an identification of a bias associated with a feature group, a performance characteristic of the predictive model, an error, combinations thereof, an identification of a cause of the bias or error such as a characteristic of the predictive model 204, a portion of the training data 208, a portion of the input data 212, combinations thereof, or the like”; input data given to predictive model and error is output);
using the updated training dataset to train the alternative machine learning model associated with the first bin (paragraph [0065], “If the model correction includes a modification to the predictive model 204, the system 200 applies the model correction to the predictive model 204. If the model correction includes a modification to the training data 208, the system 200 applies the model correction to modify training data 208. Once the training data 208 is modified, the predictive model 204 is retraining using the modified training data 208… Alternatively, a new model correction may be developed and applied to the predictive model 204 and/or the training data 208”; modified predictive model is analogous to an alternative machine model, which may then be retrained on modified training data);
in response to the machine learning model generating a prediction from production data input matching the first bin, input the production data input into the alternative machine learning model (paragraph [0042], “In some instances, model test node 104 receives a seed from the training datasets 132… The model test node 104 uses the seed to procedurally generate any amount of training data that corresponds to the predictive model. The seed may be combined with a particular feature or feature group to generate training data that corresponds to that feature or feature group”; paragraphs [0051], “Once the predictive model is trained using the training data 208, the system 200 executes the trained predictive model using the input data 212 to generate test data 216”; test data is the prediction outputs of the predictive model; paragraphs [0063]-[0064], “The model correction 224 receives the identification of the cause of the bias and generates a model correction that, when applied to the predictive model, reduces or eliminates the bias. The model correction includes modified training data, instructions to modify training data, a modification to the predictive model, or instructions to modify the predictive model… In some instances, the model correction can include modifications to the predictive model… the model correction 224 generates a model correction that modifies a portion of the predictive model that is associated with the bias”; alternative model is analogous to the version of the model in the reference that is modified for the purposes of removing bias; paragraph [0065], “If the model correction includes a modification to the predictive model 204, the system 200 applies the model correction to the predictive model 204… In other instances, the process repeats to determine of the model correction successfully reduced or eliminated the identified and if the model correction introduced a new bias”; if model itself is modified, it may be trained again on the training data (training data is considered analogous to production data)); and
to present an alternative prediction that is more accurate than the prediction of the in-production machine learning model. (paragraph [0089], “FIG. 6 depicts an example of a graphical user interface presenting bias analysis data, according to certain embodiments of the present disclosure. A bias detection system generates a graphical user interface to visually represent the biases in a predictive model”; paragraph [0091], “A second subpanel 612 indicates the overall accuracy of the predictive model in generating predictions. A third subpanel 616 represents four performance metrics associated with a particular prediction associated with the feature group. Returning to the employee evaluation example, a feature group of Asian males has three possible three possible predictions: below expectations, meets expectations, exceeds expectations. The subpanel 616 displays four performance metrics associated with the meets expectation prediction. The four performance metrics indicate the classification accuracy 620, classification rate 624, true positive rate 628, and false positive rate 632 for Asian males predicted to meet expectations” the alternative prediction exceeds expectations).
But Lee fail to explicitly disclose “wherein the machine learning model is an in-production machine learning model; determine that the in-production machine learning model is not performing as required for a first bin of the plurality of bins based on an average difference between the first and second sets of residual values for the first bin satisfies a residual threshold, train, based on determining that the in-production machine learning model is not performing as required for the first bin, an alternative machine learning model using an updated training dataset that includes a portion of the training dataset that corresponds to the first bin and that is smaller than an entirety of the training dataset;”
Grossman disclose “wherein the machine learning model is an in-production machine learning model; (col. 5, line 5-20, col. 7, line 46-col, 8, line 20, ML model may be deployed into a production env.)
determine that the in-production machine learning model is not performing as required for a first bin of the plurality of bins based on an average difference between the first and second sets of residual values for the first bin satisfies a residual threshold, (col. 17, line 28-col. 23, line 32, determine the ML model is not satisfying the threshold for the different sets values for the bin based on an average drift difference)
train, based on determining that the in-production machine learning model is not performing as required for the first bin, an alternative machine learning model using an updated training dataset that includes a portion of the training dataset that corresponds to the first bin and that is smaller than an entirety of the training dataset; (col. 17, line 28-col. 23, line 32, train, based on the ML model is not satisfying the threshold for the first bin, updating the ML using a different training set which is part of the first bin, such as different bin ranges, different value ranges, etc. Note: (1) please incorporate claim 21 to claim 5, and claim 22 to claim 13 to make it corresponding to claim 1; (2) in claim 5 and 13, there are a threshold and an error threshold, are they different, please further clarify and in claim 13, “the difference between the difference…”(4) please try to file an RCE so the examiner would have more time to consider since this is the first office action for the examiner.)
It would have been obvious to one having ordinary skill in the art before the effective filing data of the claimed invention was made to incorporate Grossman‘s generating an alert regarding a ML model into Lee’s invention as they are related to the same field endeavor of model training and learning. The motivation to combine these arts, as proposed above, at least because Grossman’s ML model performance evaluation would help to provide more ML model evaluation method into Lee’s system. Therefore it would have been obvious to one having ordinary skill in the art before the effective filing data of the claimed invention was made that providing model performance evaluation in training ML model would help to improve accuracy of prediction precision and training efficiency.
In regard to claim 21, claim 21 is a method claim corresponding to the system claim 1 above and, therefore, is rejected for the same reasons set forth in the rejections of claim 1.
In regard to claim 22, claim 22 is a medium claim corresponding to the system claim 1 above and, therefore, is rejected for the same reasons set forth in the rejections of claim 1.
Regarding claim 2
Regarding claim 2, Lee teaches the system of claim 1 (as mentioned above), wherein the instruction for generating the plurality of bins for the compare dataset, when executed by the one or more processors, further cause the one or more processors to:
select a first feature within the compare dataset (paragraph [0038], “The bias detection module 124 enumerates the features over which the predictive model is tested. In some instances, the bias detection module 124 enumerates the features to test the predictive model from an analysis of the training data 116 and/or the test data 120. For instance, the bias detection module 124 enumerates a set of features by identifying types of predictions that are generated, characteristics of a class (e.g., employees) that are the subject of the generated predictions, and/or statistically relevant aspects of the training data 208 or input data 212 such as data types, data structures, reoccurring data values, and the like”; enumeration allows the bias detection module to identify particular features of interest);
generate a plurality of bin labels for the first feature, wherein each bin is associated with a subset of values for the first feature (paragraph [0031], “For example, for a predictive model generating predictions of employee performance, the employees are representative of objects and a feature group corresponds to a group of employees that each share a common set of characteristics such as demographic information (e.g., Asian males), positional information (e.g., executive officers), geographical information (e.g., in the New York office, in the Northeast, etc.), salary information, historical predictions, combinations thereof, or the like”); and
assign entries of the compare dataset to the plurality of bin labels according to a corresponding value for the first feature for each entry in the compare dataset (paragraph [0043], “the model test node 104 determines the portion of the training data 116 that corresponds to the feature group female executives. The model test node 104 analysis the training data 116 to determine a type model correction to reduce the bias such as increasing training data corresponding to female executives with labels that correspond to meets expectations and exceeds expectations, decreasing training data associated with female executives that have a label of below expectations, increasing or decreasing training data corresponding to another feature or feature groups, or the like”).
Regarding claim 5
Regarding claim 5, Lee teaches a method comprising:
subsequent to a machine learning model being trained using a first dataset, generating a plurality of groups for a second dataset based on a parameter within the second dataset, wherein each group of the plurality of groups is associated with a different range of parameter values for the parameter, (paragraph [0066]-[0067], “A block 304, a computing device receives data associated with a trained predictive model. The data includes data output from the predictive model such as predictions associated with a class or a subclass. In some instances, the data includes the input data that was input to the predictive model to generate the output and training data used during a training phase of the predictive model… At block 308, the computing device identifies a feature group from the data”; paragraph [0054], “A feature group is made up of one or more features and represents a subclass of the class that includes members that each include characteristics that correspond to the one or more features”; feature groups can be considered analogous to parameters, and they may contain one or more features that intuitively correspond to groups of values);
obtaining a first set of entries corresponding to a group from the first dataset and a second set of entries corresponding to the group from the second dataset;(paragraph [0042], “In some instances, model test node 104 receives a seed from the training datasets 132… The model test node 104 uses the seed to procedurally generate any amount of training data that corresponds to the predictive model. The seed may be combined with a particular feature or feature group to generate training data that corresponds to that feature or feature group”; paragraph [0038], “For instance, the bias detection module 124 enumerates a set of features by identifying types of predictions that are generated, characteristics of a class (e.g., employees) that are the subject of the generated predictions, and/or statistically relevant aspects of the training data 208 or input data 212 such as data types, data structures, reoccurring data values, and the like”; input data is analogous to the second dataset in this example; model test node generates (“obtains”) training data and bias detection module takes input data from feature sets);
inputting the first set of entries corresponding to the group into the machine learning model to obtain a first set of error values, the first set of error values outputted by the machine learning model based on the machine learning model's processing of the first set of entries; (paragraph [0060], “The results of bias detection module 220 includes, but is not limited to, an identification of a bias associated with a feature group, a performance characteristic of the predictive model, an error, combinations thereof, an identification of a cause of the bias or error such as a characteristic of the predictive model 204, a portion of the training data 208, a portion of the input data 212, combinations thereof, or the like”; training data given to predictive model and error is output);
inputting the second set of entries into the machine learning model to obtain a second set of error values, the second set of error values outputted by the machine learning model based on the machine learning model's processing of the second set of entries (paragraph [0060], “The results of bias detection module 220 includes, but is not limited to, an identification of a bias associated with a feature group, a performance characteristic of the predictive model, an error, combinations thereof, an identification of a cause of the bias or error such as a characteristic of the predictive model 204, a portion of the training data 208, a portion of the input data 212, combinations thereof, or the like”; input data given to predictive model and error is output); and
But Lee fail to explicitly disclose “wherein the machine learning model is an in-production machine learning model; wherein the machine learning model is an in-production machine learning model; determine the in-production machine learning model is not performing as required for a first group of the plurality of group based on a difference between the first set of error values and the second set of error values satisfying a threshold; in response to determining the difference between the first set of error values and the second set of error values satisfies an error threshold, transmitting a notification to a user indicating the first group.”
Grossman disclose “wherein the machine learning model is an in-production machine learning model; wherein the machine learning model is an in-production machine learning model; (col. 5, line 5-20, col. 7, line 46-col, 8, line 20, ML model may be deployed into a production env.)
determine the in-production machine learning model is not performing as required for a first group of the plurality of group based on the difference between the first set of error values and the second set of error values satisfies an error threshold, in response to determining the difference between the first set of error values and the second set of error values satisfies an error threshold, transmitting a notification to a user indicating the first group. (col. 17, line 28-col. 23, line 32, determine the ML model is not satisfying the threshold for the different sets values for the bin based on an average drift difference and sending an alert to the user indicating the dataset)
It would have been obvious to one having ordinary skill in the art before the effective filing data of the claimed invention was made to incorporate Grossman‘s generating an alert regarding a ML model into Lee’s invention as they are related to the same field endeavor of model training and learning. The motivation to combine these arts, as proposed above, at least because Grossman’s ML model performance evaluation would help to provide more ML model evaluation method into Lee’s system. Therefore it would have been obvious to one having ordinary skill in the art before the effective filing data of the claimed invention was made that providing model performance evaluation in training ML model would help to improve accuracy of prediction precision and training efficiency.
Regarding claim 6
Regarding claim 6, Lee teaches the method of claim 5 (as explained above), wherein generating the plurality of groups comprises:
selecting a first parameter within the first dataset, wherein the first parameter corresponds to a set of parameter values (paragraph [0038], “The bias detection module 124 enumerates the features over which the predictive model is tested. In some instances, the bias detection module 124 enumerates the features to test the predictive model from an analysis of the training data 116 and/or the test data 120. For instance, the bias detection module 124 enumerates a set of features by identifying types of predictions that are generated, characteristics of a class (e.g., employees) that are the subject of the generated predictions, and/or statistically relevant aspects of the training data 208 or input data 212 such as data types, data structures, reoccurring data values, and the like”; enumeration allows the bias detection module to identify particular features of interest and group them into feature groups); and
generating a plurality of group definitions for the set of parameters values, wherein each group definition is associated with a subset of the set of parameter values (paragraph [0031], “For example, for a predictive model generating predictions of employee performance, the employees are representative of objects and a feature group corresponds to a group of employees that each share a common set of characteristics such as demographic information (e.g., Asian males), positional information (e.g., executive officers), geographical information (e.g., in the New York office, in the Northeast, etc.), salary information, historical predictions, combinations thereof, or the like”).
Regarding claim 7
Regarding claim 7, Lee teaches the method of claim 6 (as explained above), wherein obtaining the first set of entries from the first dataset comprises:
obtaining the first set of entries from the first dataset comprises: determining, for each entry in the first dataset, a group definitions of the plurality of group definition that matches a parameter value for the entry; and assigning each entry of the first dataset to a group of the plurality of groups according to the determining (paragraph [0041], “The bias detection module 124 identifies one or more types of bias associated with a particular feature or feature group then identifies the portion of the training data 116 that corresponds to the feature or feature group”; feature groups are well-defined according to paragraph [0031]; identifying training data corresponding to feature groups is indicative of matching data entries to group definitions; training data entries are considered assigned to the feature group they belong to).
Regarding claim 13
Regarding claim 13, Lee teaches:
A non-transitory, computer-readable medium for detecting category-specific model bias that, when executed by one or more processors, cause the one or more processors to perform operations comprising (paragraph [0129], “The computing system 1000 executes program code that configures the processor 1102 to perform one or more of the operations described herein. The program code may be resident in the memory device 1008 or any suitable computer-readable medium and may be executed by the processor 1004 or any other suitable processor”; “memory device 1008” implies the use of tangible hardware, therefore a non-transitory computer-readable storage medium);
subsequent to a machine learning model being trained using a first dataset, generating a plurality of groups based on a parameter within a first dataset, wherein each group of the plurality of groups is associated with a different range of parameter values for the parameter, (paragraph [0066]-[0067], “A block 304, a computing device receives data associated with a trained predictive model. The data includes data output from the predictive model such as predictions associated with a class or a subclass. In some instances, the data includes the input data that was input to the predictive model to generate the output and training data used during a training phase of the predictive model… At block 308, the computing device identifies a feature group from the data”; paragraph [0054], “A feature group is made up of one or more features and represents a subclass of the class that includes members that each include characteristics that correspond to the one or more features”; feature groups can be considered analogous to parameters, and they may contain one or more features that intuitively correspond to groups of values);
obtaining a first set of entries corresponding to a group from the first dataset and a second set of entries corresponding to the group from the second dataset;(paragraph [0042], “In some instances, model test node 104 receives a seed from the training datasets 132… The model test node 104 uses the seed to procedurally generate any amount of training data that corresponds to the predictive model. The seed may be combined with a particular feature or feature group to generate training data that corresponds to that feature or feature group”; paragraph [0038], “For instance, the bias detection module 124 enumerates a set of features by identifying types of predictions that are generated, characteristics of a class (e.g., employees) that are the subject of the generated predictions, and/or statistically relevant aspects of the training data 208 or input data 212 such as data types, data structures, reoccurring data values, and the like”; input data is analogous to the second dataset in this example; model test node generates (“obtains”) training data and bias detection module takes input data from feature sets);
inputting the first set of entries corresponding to the group into the machine learning model to obtain a first set of error values, the first set of error values outputted by the machine learning model based on the machine learning model's processing of the first set of entries (paragraph [0060], “The results of bias detection module 220 includes, but is not limited to, an identification of a bias associated with a feature group, a performance characteristic of the predictive model, an error, combinations thereof, an identification of a cause of the bias or error such as a characteristic of the predictive model 204, a portion of the training data 208, a portion of the input data 212, combinations thereof, or the like”; training data given to predictive model and error is output);
inputting the second set of entries into the machine learning model to obtain a second set of error values, the second set of error values outputted by the machine learning model based on the machine learning model's processing of the second set of entries (paragraph [0060], “The results of bias detection module 220 includes, but is not limited to, an identification of a bias associated with a feature group, a performance characteristic of the predictive model, an error, combinations thereof, an identification of a cause of the bias or error such as a characteristic of the predictive model 204, a portion of the training data 208, a portion of the input data 212, combinations thereof, or the like”; input data given to predictive model and error is output); and
But Lee fail to explicitly disclose “wherein the machine learning model is an in-production machine learning model; wherein the machine learning model is an in-production machine learning model; determine the in-production machine learning model is not performing for a first group of the plurality of groups based on an average difference between the first set of error values and the second set of error values for the first group satisfies a threshold; in response to determining the difference between the difference between the first set of error values and the second set of error values satisfies an error threshold, transmitting a notification to a user indicating the first group.”
Grossman disclose “wherein the machine learning model is an in-production machine learning model; (col. 5, line 5-20, col. 7, line 46-col, 8, line 20, ML model may be deployed into a production env.)
determine the in-production machine learning model is not performing for a first group of the plurality of groups based on an average difference between the first set of error values and the second set of error values for the first group satisfies a threshold; in response to determining the difference between the difference between the first set of error values and the second set of error values satisfies an error threshold, transmitting a notification to a user indicating the first group. (col. 17, line 28-col. 23, line 32, determine the ML model is not satisfying the threshold for the different sets values for the bin based on an average drift difference and sending an alert to the user indicating the dataset)
It would have been obvious to one having ordinary skill in the art before the effective filing data of the claimed invention was made to incorporate Grossman‘s generating an alert regarding a ML model into Lee’s invention as they are related to the same field endeavor of model training and learning. The motivation to combine these arts, as proposed above, at least because Grossman’s ML model performance evaluation would help to provide more ML model evaluation method into Lee’s system. Therefore it would have been obvious to one having ordinary skill in the art before the effective filing data of the claimed invention was made that providing model performance evaluation in training ML model would help to improve accuracy of prediction precision and training efficiency.
Regarding claim 14
Regarding claim 14, Lee teaches the non-transitory, computer-readable medium of claim 13 (as explained above), wherein the instructions for generating the plurality of groups further cause the one or more processors to perform operations comprising:
selecting a first parameter within the first dataset, wherein the first parameter corresponds to a set of parameter values (paragraph [0038], “The bias detection module 124 enumerates the features over which the predictive model is tested. In some instances, the bias detection module 124 enumerates the features to test the predictive model from an analysis of the training data 116 and/or the test data 120. For instance, the bias detection module 124 enumerates a set of features by identifying types of predictions that are generated, characteristics of a class (e.g., employees) that are the subject of the generated predictions, and/or statistically relevant aspects of the training data 208 or input data 212 such as data types, data structures, reoccurring data values, and the like”; enumeration allows the bias detection module to identify particular features of interest and group them into feature groups); and
generate a plurality of group definitions for set of parameters values, wherein each group definition is associated with a subset of the set of parameter values (paragraph [0031], “For example, for a predictive model generating predictions of employee performance, the employees are representative of objects and a feature group corresponds to a group of employees that each share a common set of characteristics such as demographic information (e.g., Asian males), positional information (e.g., executive officers), geographical information (e.g., in the New York office, in the Northeast, etc.), salary information, historical predictions, combinations thereof, or the like”).
Regarding claim 15
Regarding claim 15, Lee teaches the non-transitory, computer-readable medium of claim 14 (as explained above), wherein the instructions for obtaining the first set of entries from the first dataset further cause the one or more processors to perform operations comprising:
determining, for each entry in the first dataset, a group definition of the plurality of group definitions that matches a parameter value for the entry; and assigning each entry of the first dataset to a group of the plurality of groups according to the determining (paragraph [0041], “The bias detection module 124 identifies one or more types of bias associated with a particular feature or feature group then identifies the portion of the training data 116 that corresponds to the feature or feature group”; feature groups are well-defined according to paragraph [0031]; identifying training data corresponding to feature groups is indicative of matching data entries to group definitions; training data entries are considered assigned to the feature group they belong to).
Claims 3, 4, 8-11, and 16-19 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (“Methods and Systems for Detection and Isolation of Bias in Predictive Models”—U.S. Patent No. 11,593,648; hereinafter “Lee”) and Grossman US 12165017 in view of Rama et al. (“Apparatus and Method For Anomaly Detection Using Weighted Autoencoder”—US 20220198267 A1; hereinafter “Rama”).
Claims 3, 8, and 16
Regarding claim 3
Regarding claim 3, dependent on claim 2, the rejection of claim 2 is incorporated. In addition, Lee teaches The system of claim 2, wherein the instructions when executed by the one or more processors, further cause the one or more processors to: determine a type of data associated with the first feature (paragraph [0038], “For instance, the bias detection module 124 enumerates a set of features by identifying types of predictions that are generated, characteristics of a class (e.g., employees) that are the subject of the generated predictions, and/or statistically relevant aspects of the training data 208 or input data 212 such as data types, data structures, reoccurring data values, and the like”).
Lee and Grossman do not teach the limitations wherein the instructions when executed by the one or more processors, further cause the one or more processors to… sort, according to the type of data, the values associated with the first feature, determine a number of bins for the first feature, or assign the entries of the compare dataset to the plurality of bin labels according to the sorting. However, Rama teaches the limitations wherein the instructions when executed by the one or more processors, further cause the one or more processors to sort, according to the type of data, the values associated with the first feature and determine a number of bins for the first feature (Rama, paragraph [0029], “The feature values of each observation are grouped into bins such that there are not overlapping intervals between the bins. The feature values of the observations are first sorted in ascending order and divided into buckets, i.e., bins, such that each bin has an equal number of observations. This is shown as Stage 1 in which each bin has three observations with the smallest feature value at the top. Let featureistart and featureiend denote the starting and ending values of each of the i groups. All overlapping intervals with the same value are merged into the same interval as shown in Stage 2. This is also represented by Equation (2) below, where b denotes the number of bins”), as well as assign the entries of the compare dataset to the plurality of bin labels according to the sorting (Rama, paragraph [0029], “FIG. 3 is a diagram of one example of an interval width binning heuristic that may be applied to the input training observations in order to determine weights for the autoencoder…”).
Rama is considered analogous to the claimed invention since they both utilize feature grouping (“binning”) techniques as part of a training process. It would have been obvious to a person having ordinary skill in the art (hereinafter “PHOSITA”), before the effective filing date of the claimed invention, to incorporate the methodologies of Rama into those of Grossman and Lee. Sorting a list of items prior to their processing is a technique commonly employed by developers of software and AI models, and would be considered combining prior art elements according to known methods to yield predictable results as per MPEP 2141(III).
Regarding claim 8
Regarding claim 8, dependent on claim 5, the rejection of claim 5 is incorporated. In addition, Lee teaches The method of claim 5 further comprising: determining a type of data associated with the first group (Lee, paragraph [0038], “For instance, the bias detection module 124 enumerates a set of features by identifying types of predictions that are generated, characteristics of a class (e.g., employees) that are the subject of the generated predictions, and/or statistically relevant aspects of the training data 208 or input data 212 such as data types, data structures, reoccurring data values, and the like”).
Lee and Grossman do not teach the limitations further comprising… sorting, according to the type of data, the parameter values associated with the first group, determining a number of groups for the parameter, or assigning entries of the first dataset to the plurality of groups according to the sorting. However, Rama teaches the limitations further comprising… sorting, according to the type of data and the parameter values associated with the first group and determining a number of groups for the parameter (Rama, paragraph [0029], “The feature values of each observation are grouped into bins such that there are not overlapping intervals between the bins. The feature values of the observations are first sorted in ascending order and divided into buckets, i.e., bins, such that each bin has an equal number of observations. This is shown as Stage 1 in which each bin has three observations with the smallest feature value at the top. Let featureistart and featureiend denote the starting and ending values of each of the i groups. All overlapping intervals with the same value are merged into the same interval as shown in Stage 2. This is also represented by Equation (2) below, where b denotes the number of bins”, as well as assigning entries of the first dataset to the plurality of groups according to the sorting (Rama, paragraph [0029], “FIG. 3 is a diagram of one example of an interval width binning heuristic that may be applied to the input training observations in order to determine weights for the autoencoder…”). It would have been obvious to PHOSITA, before the effective filing date of the claimed invention, to incorporate the methodologies of Rama into those of Grossman and Lee. Sorting a list of items prior to their processing is a technique commonly employed by developers of software and AI models, and would be considered combining prior art elements according to known methods to yield predictable results as per MPEP 2141(III).
Regarding claim 16
Regarding claim 16, dependent on claim 13, the rejection of claim 13 is incorporated. In addition, Lee teaches The non-transitory computer-readable medium of claim 13, the instructions further causing the one or more processors to perform operations comprising: determining a type of data associated with the first group (Lee, paragraph [0038], “For instance, the bias detection module 124 enumerates a set of features by identifying types of predictions that are generated, characteristics of a class (e.g., employees) that are the subject of the generated predictions, and/or statistically relevant aspects of the training data 208 or input data 212 such as data types, data structures, reoccurring data values, and the like”).
Lee and Grossman do not teach the limitations the instructions further causing the one or more processors to perform operations comprising… sorting, according to the type of data, the parameter values associated with the first group, determining a number of groups for the parameter, or assigning entries of the first dataset to the plurality of groups according to the sorting. However, Rama teaches the limitations the instructions further causing the one or more processors to perform operations comprising: sorting, according to the type of data and the parameter values associated with the first group and determining a number of groups for the parameter (Rama, paragraph [0029], “The feature values of each observation are grouped into bins such that there are not overlapping intervals between the bins. The feature values of the observations are first sorted in ascending order and divided into buckets, i.e., bins, such that each bin has an equal number of observations. This is shown as Stage 1 in which each bin has three observations with the smallest feature value at the top. Let featureistart and featureiend denote the starting and ending values of each of the i groups. All overlapping intervals with the same value are merged into the same interval as shown in Stage 2. This is also represented by Equation (2) below, where b denotes the number of bins” as well as assigning entries of the first dataset to the plurality of groups according to the sorting (Rama, paragraph [0029], “FIG. 3 is a diagram of one example of an interval width binning heuristic that may be applied to the input training observations in order to determine weights for the autoencoder…”). It would have been obvious to PHOSITA, before the effective filing date of the claimed invention, to incorporate the methodologies of Rama into those of Grossman and Lee. Sorting a list of items prior to their processing is a technique commonly employed by developers of software and AI models, and would be considered combining prior art elements according to known methods to yield predictable results as per MPEP 2141(III).
Claim 4
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Lee and Grossman US 12165017 in view of in view of Lopes et al. (“Machine Learning System for Transaction Reconciliation”—US 20210350382 A1; hereinafter “Lopes”) and Fukushima et al. (“Information Processing Apparatus, Information Processing Method, And Computer Program”—U.S. Patent No. 12,136,023; hereinafter “Fukushima”).
Regarding claim 4, dependent on claim 1, the rejection of claim 1 is incorporated. In addition, Lee teaches The system of claim 1 wherein the instructions when executed by the one or more processors, further cause the one or more processors to… determining a base residual score for the first set of entries and a compare residual score for the second set of entries (Lee, paragraph [0060], “The results of bias detection module 220 are passed to model corrections 224. The results of bias detection module 220 includes, but is not limited to, an identification of a bias associated with a feature group, a performance characteristic of the predictive model, an error, combinations thereof, an identification of a cause of the bias or error such as a characteristic of the predictive model 204, a portion of the training data 208, a portion of the input data 212, combinations thereof, or the like”).
Lee and Grossman do not teach the limitation wherein the instructions when executed by the one or more processors, further cause the one or more processors to: calculating the average difference between the first and second sets of residual values based on the base residual score and the compare residual score. However, Lopes teaches wherein the instructions when executed by the one or more processors, further cause the one or more processors to: calculating the (Lopes, paragraph [0059], “For example, the transaction reconciliation system may determine a first error metric associated with the set of matched entries, may determine a second error metric associated with the set of classified entries, may compare the two error metrics, and may perform an action based on the comparison. For example, the transaction reconciliation system may perform an action based on a difference between the error metrics, whether the difference satisfies a condition, whether the difference satisfies a threshold, and/or the like”), while Fukushima teaches calculating the average of first and second sets of residual values (Fukushima, paragraph (57) and (58), “Examples of calculating, as the improved score, an aggregation of errors that is a divided group include the following examples 3 and 4… where the average of the errors of the first group is… and the average of the errors of the second group is…”; residual values are considered error values; and while Fukushima teaches averaging two sets of errors between which a difference can be found as taught by Lopes, it can be shown mathematically that the difference of two averages is equivalent to averaging a corresponding set of differences).
Lopes and Fukushima are considered analogous to the claimed invention since they both utilize error values in machine learning. It would have been obvious to a PHOSITA, before the effective filing date of the claimed invention, to incorporate the methodologies of Lopes and Fukushima into those of Grossman and Lee. Taking an average of error values (as is frequently done with statistical measures such as mean absolute error) is considered combining prior art elements in known ways to yield predictable results according to MPEP 2141(III).
Claims 9, 11, 17, and 19
Claims 9, 11, 17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Lee, Grossman in view of Lopes.
Regarding claim 9
Regarding claim 9, dependent on claim 5, the rejection of claim 5 is incorporated. In addition, Lee teaches The method of claim 5 further comprising: determining a first error score for the first set of entries and a second error score for the second set of entries (paragraph [0060], “The results of bias detection module 220 are passed to model corrections 224. The results of bias detection module 220 includes, but is not limited to, an identification of a bias associated with a feature group, a performance characteristic of the predictive model, an error, combinations thereof, an identification of a cause of the bias or error such as a characteristic of the predictive model 204, a portion of the training data 208, a portion of the input data 212, combinations thereof, or the like”).
Lee does not teach the limitation further comprising… calculating the difference between the first and second sets of error values based on the first error score and the second error score. However, Lopes teaches this limitation (paragraph [0059], “For example, the transaction reconciliation system may determine a first error metric associated with the set of matched entries, may determine a second error metric associated with the set of classified entries, may compare the two error metrics, and may perform an action based on the comparison. For example, the transaction reconciliation system may perform an action based on a difference between the error metrics, whether the difference satisfies a condition, whether the difference satisfies a threshold, and/or the like. As a specific example, if the second error metric indicates a greater error than the first error metric and/or if the difference between the error metrics satisfies a threshold, then the transaction reconciliation system may predict missing transaction data”). It would have been obvious to a PHOSITA, before the effective filing date of the claimed invention, to incorporate the methodologies of Lopes into those of Grossman and Lee. Calculating a difference between sets of error values is considered combining prior art elements in known ways to yield predictable results according to MPEP 2141(III).
Regarding claim 11
Regarding claim 11, dependent on claim 9, the rejection of claim 9 is incorporated. In addition, the combination of Lee, Grossman and Lopes teaches the limitation The method of claim 9 further comprising determining the difference between the first set error of values and the second set of error values based on the second error score being higher than the first error score by the error threshold. In particular, Lopes teaches this limitation (Lopes, paragraph [0059], “For example, the transaction reconciliation system may determine a first error metric associated with the set of matched entries, may determine a second error metric associated with the set of classified entries, may compare the two error metrics, and may perform an action based on the comparison. For example, the transaction reconciliation system may perform an action based on a difference between the error metrics, whether the difference satisfies a condition, whether the difference satisfies a threshold, and/or the like. As a specific example, if the second error metric indicates a greater error than the first error metric and/or if the difference between the error metrics satisfies a threshold, then the transaction reconciliation system may predict missing transaction data”). It would have been obvious to a PHOSITA, before the effective filing date of the claimed invention, to incorporate the methodologies of Lopes into those of Grossman and Lee. Calculating a difference between sets of error values is considered combining prior art elements in known ways to yield predictable results according to MPEP 2141(III).
Regarding claim 17
Regarding claim 17, dependent on claim 13, the rejection of claim 13 is incorporated. In addition, Lee teaches The non-transitory, computer-readable medium of claim 13, the instructions further causing the one or more processors to perform operations comprising: determining a first error score for the first set of entries and a second error score for the second set of entries (paragraph [0060], “The results of bias detection module 220 are passed to model corrections 224. The results of bias detection module 220 includes, but is not limited to, an identification of a bias associated with a feature group, a performance characteristic of the predictive model, an error, combinations thereof, an identification of a cause of the bias or error such as a characteristic of the predictive model 204, a portion of the training data 208, a portion of the input data 212, combinations thereof, or the like”).
Lee and Grossman do not teach the limitation the instructions further causing the one or more processors to perform operations comprising… calculating the difference between the first and second sets of error values based on the first error score and the second error score. However, Lopes teaches this limitation (paragraph [0059], “For example, the transaction reconciliation system may determine a first error metric associated with the set of matched entries, may determine a second error metric associated with the set of classified entries, may compare the two error metrics, and may perform an action based on the comparison. For example, the transaction reconciliation system may perform an action based on a difference between the error metrics, whether the difference satisfies a condition, whether the difference satisfies a threshold, and/or the like. As a specific example, if the second error metric indicates a greater error than the first error metric and/or if the difference between the error metrics satisfies a threshold, then the transaction reconciliation system may predict missing transaction data”). It would have been obvious to a PHOSITA, before the effective filing date of the claimed invention, to incorporate the methodologies of Lopes into those of Grossman and Lee. Calculating a difference between sets of error values is considered combining prior art elements in known ways to yield predictable results according to MPEP 2141(III).
Regarding claim 19
Regarding claim 19, dependent on claim 17, the rejection of claim 17 is incorporated. In addition, the combination of Lee, Grossman and Lopes teaches the limitation The non-transitory, computer-readable medium of claim 17 further comprising determining the difference between the first set of error values and the second set of error values based on the second error score being higher than the first error score by the error threshold. In particular, Lopes teaches this limitation (Lopes, paragraph [0059], “For example, the transaction reconciliation system may determine a first error metric associated with the set of matched entries, may determine a second error metric associated with the set of classified entries, may compare the two error metrics, and may perform an action based on the comparison. For example, the transaction reconciliation system may perform an action based on a difference between the error metrics, whether the difference satisfies a condition, whether the difference satisfies a threshold, and/or the like. As a specific example, if the second error metric indicates a greater error than the first error metric and/or if the difference between the error metrics satisfies a threshold, then the transaction reconciliation system may predict missing transaction data”). It would have been obvious to a PHOSITA, before the effective filing date of the claimed invention, to incorporate the methodologies of Lopes into those of Grossman and Lee. Calculating a difference between sets of error values is considered combining prior art elements in known ways to yield predictable results according to MPEP 2141(III).
Claims 10 and 18
Claims 10 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Lee and Grossman in view of Lopes as applied to claims 5 and 13 above, and further in view of Fukushima.
Regarding claim 10
Regarding claim 10, dependent on claim 9, the rejection of claim 9 is incorporated. In addition, the combination of Lee, Grossman and Lopes teaches the limitation The method of claim 9 but does not teach wherein determining the first error score for the first set of entries and the second error score for the second set of entries comprises generating a first average value for the first set of error values and a second average value for the second set of error values. However, Fukushima teaches this limitation (Fukushima, paragraph (57) and (58), “Examples of calculating, as the improved score, an aggregation of errors that is a divided group include the following examples 3 and 4… where the average of the errors of the first group is… and the average of the errors of the second group is…”). It would have been obvious to a PHOSITA, before the effective filing date of the claimed invention, to incorporate the teachings of Fukushima into those of Grossman, Lee and Lopes. Taking the average of multiple error values is common practice in statistical settings to determine, for example, a mean squared error. This is considered combining prior art elements in known ways to yield predictable results according to MPEP 2141(III).
Regarding claim 18
Regarding claim 18, dependent on claim 17, the rejection of claim 17 is incorporated. In addition, the combination of Lee, Grossman and Lopes teaches the limitation The non-transitory computer-readable medium of claim 17 but does not teach wherein the instructions for determining the first error score for the first set of entries and the second error score for the second set of entries comprises generating a first average value for the first set of error values and a second average value for the second set of error values. However, Fukushima teaches this limitation (Fukushima, paragraph (57) and (58), “Examples of calculating, as the improved score, an aggregation of errors that is a divided group include the following examples 3 and 4… where the average of the errors of the first group is… and the average of the errors of the second group is…”). It would have been obvious to a PHOSITA, before the effective filing date of the claimed invention, to incorporate the teachings of Fukushima into those of Grossman, Lee and Lopes. Taking the average of multiple error values is common practice in statistical settings to determine, for example, a mean squared error. This is considered combining prior art elements in known ways to yield predictable results according to MPEP 2141(III).
Response to Arguments
Applicant’s arguments with respect to claims 1-20 filed on 7/17/2025 have been considered but are moot because the arguments do not apply to the current rejection.
Conclusion
The prior art made of record and not relied upon is considered pertinent to Applicant's disclosure.
PATENT PUB. # PUB. DATE INVENTOR(S) TITLE
US 20230112063 A1 2023-04-13 Kwon et al.
INTERACTIVE SUBGROUP DISCOVERY
Kwon et al. disclose obtain covariates and an outcome data for a population. Partition the population into a plurality of subgroups. Produce outcomes predictions by applying a machine learning model to the covariate data for the population. Establish performance measures based on the outcomes predictions. Compare the performance measures for at least one subgroup to the performance measures for at least one other subgroup. Identify an outlying subgroup for which the machine learning model produces performance measures that are different than the performance measures for one or more other subgroups. Optionally, retrain the machine learning model on additional covariate and outcomes data for the outlying subgroup… see abstract.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XUYANG XIA whose telephone number is (571)270-3045. The examiner can normally be reached Monday-Friday 8am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached at 571-272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
XUYANG XIA
Primary Examiner
Art Unit 2143
/XUYANG XIA/Primary Examiner, Art Unit 2143
Prosecution Insights