Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Status
This instant application No. 18/111825 has Claims 1-20 pending.
Priority / Filing Date
Applicant did not claim for any domestic or foreign priority. The effective filing date of this application is February 20, 2023.
Information Disclosure Statement
As required by M.P.E.P. 609(C), the Applicant’s submissions of the Information Disclosure Statements dated June 9, 2023, September 13, 2023, July 1, 2024, August 12, 2024, February 11, 2025, April 1, 2024, September 8, 2025 and December 30, 2025 are acknowledged by the Examiner and the cited references have been considered in the examination of the claims now pending. As required by M.P.E.P. 609 C(2), a copy of each of the PTOL-1449s initialed and dated by the Examiner is attached to the instant Office action.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
4. Claims 1-20 are rejected on the ground of non-statutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 12,475,132 from the same inventors and assignee. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of the ‘132 Patent include all the limitations of this Application as well as additional limitations.
5. Claims 1-20 are rejected on the ground of non-statutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 12,614,083 from the same inventors and assignee. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of the ‘083 Patent include all the limitations of this Application as well as additional limitations.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
6. Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
Step 2A Prong One:
Independent Claims 1, 10 and 19 recite
use the given input data record, the sample historical data record, the randomly-selected group coalition, and the randomly-selected variable coalition to compute an iteration-specific contribution value for the respective input variable; and
for each respective input variable of the model object, aggregate the iteration-specific contribution values calculated for each iteration and thereby determine an aggregated contribution value for the respective input variable.,
-all of which are mathematical calculations and/or mathematical relationships-which is categorized as mathematical concepts.
The limitation:
arrange the set of input variables into two or more variable groups based on
dependencies between respective input variables, where each variable group comprises at
least one input variable;
involves mathematical calculation as well as mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper.
Additionally, the limitations:
identify a sample historical data record from a set of historical data records;
select a random group coalition comprising the respective variable group and zero or more other variable groups;
select a random variable coalition, within the respective variable group,
comprising the respective input variable and zero or more other input variables;
encompasses mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper. Said limitations in Claims 1, 10 and 19 are a process that under its broadest reasonable interpretation, covers performance of the limitations in the mind and/or mathematical calculation/relationships but for the recitation of generic computer components. Other than reciting “a computing platform”, “at least one processor”; “non-transitory computer-readable medium”, and “program instructions stored on the non-transitory computer-readable medium that are executable by the at least one processor” in the claims nothing in the claim elements precludes the steps from practically being performed in the mind and/or a mathematical concept. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind and/or a mathematical concept but for the recitation of generic computer components, then it falls within the “mental processes” and “mathematical concept” grouping of abstract ideas. As such Claims 1, 10 and 19 recite an abstract idea.
Step 2A Prong Two:
This judicial exception is not integrated into a practical application. The claims recite the additional element of “a computing platform”, “at least one processor”; “non-transitory computer-readable medium”, “program instructions stored on the non-transitory computer-readable medium that are executable by the at least one processor” and “a machine learning process” to perform the claimed steps at a high level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component. This additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The additional element of receive an input data record comprising a set of
input variables and output a score for the input data record is a data gathering/data display steps and an insignificant extra-solution activity. Training a model object for a data science model using a machine learning process to accomplish these steps is just using generic tool to accomplish these tasks perceived as insignificant-extra solution activity that could be accomplished using by any generic means. As such this additional element also does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Step 2B:
Finally, the pre-processing step of receiving input data and post-processing step of output a score of input data is categorized as insignificant extra solution activity under 2106.05(g). Claims 1, 10 and 19 only recite ““a computing platform”, “at least one processor”; “non-transitory computer-readable medium”, “program instructions stored on the non-transitory computer-readable medium that are executable by the at least one processor” and “a machine learning process” to perform the claimed steps and therefore only recite a general purpose computer rather than a specific machine under MPEP 2106.05(b), and are directed to mere instructions to apply the exception under MPEP 2106.05(f), and do not result in anything significantly more than the judicial exception. The additional elements have been considered both individually and as an ordered combination in the significantly more consideration. The inclusion of the computer or memory and machine learning process to perform the arranging, identifying, selecting and determining steps amount to nor more than mere instructions to apply the exception using generic computer components. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Claims 1, 10 and 19 are not patent eligible.
The dependent claims include the same abstract ideas and mathematical techniques recited as recited in the independent claims, and merely incorporate additional details that narrow the abstract ideas and fail to add significantly more to the claims.
Dependent Claims 2, 3, 11 and 12 are directed to further limiting generate a random ordering of the two or more variable groups; and define the randomly-selected group coalition as including the respective variable group and all other variable groups that precede the respective variable group in the generated random ordering using mathematical analysis, which further narrows the abstract idea identified in the independent claim, which is directed to “Mathematical concepts.”
Dependent Claims 4, 13 and 20 are directed to further limiting generate a first synthetic data record comprising a mix of input variables from the given input data record and the sample historical data record; generate a second synthetic data record comprising an adjusted mix of input variables from the given input data record and the sample historical data record; determine a first score for the first synthetic data record and a second score for the second synthetic data record; and calculate a difference between the first score and the second score, wherein the difference is the iteration-specific contribution value-all of which are mathematical analysis/calculation, which further narrows the abstract idea identified in the independent claim, which is directed to “Mathematical concepts.”
Dependent Claims 5 and 14 are directed to further limiting identify a subset of input variables that are included in the randomly-selected group coalition and the randomly-selected variable coalition; for the identified subset of input variables, use values from the given input data record for the first synthetic data record; and for each other input variable, use values from the sample historical data record for the first synthetic data record; and for the identified subset of input variables, excluding the respective input variable, use values from the given input data record for the second synthetic data record; and for each other input variable, including the respective input variable, use values from the sample historical data record for the second synthetic data record - all of these steps encompass mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper., which further narrows the abstract idea identified in the independent claim, which is directed to “Mental process”.
Dependent Claims 6 and 15 are directed to further limiting determine an average of the iteration-specific contribution values for each iteration over the given number of iterations- which is mathematical analysis/calculation that further narrows the abstract idea identified in the independent claim, which is directed to “Mathematical concepts.”
Dependent Claims 7 and 16 are directed to further limiting while performing the given number of iterations for a first input variable of the model object, perform the given number of iterations for each other input variable of the model object- which is mathematical analysis/calculation that further narrows the abstract idea identified in the independent claim, which is directed to “Mathematical concepts.”
Dependent Claims 8 and 17 are directed to further limiting the number of iterations to a certain value- which is a matter of design choice and is considered an extra-solution activity.
Dependent Claims 9 and 18 are directed to further limiting identify a randomly-sampled historical data record from a set of historical data records- which encompass mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper, which further narrows the abstract idea identified in the independent claim, which is directed to “Mental process”.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not
identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and
the prior art are such that the claimed invention as a whole would have been obvious before the effective filing
date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains.
Patentability shall not be negated by the manner in which the invention was made.
7. Claims 1-4, 6-7, 9-13, 15-16 and 18-20 are rejected under 35 U.S.C. 103 as being obvious over Miroshnikov et al. hereafter Miroshnikov_1 (Pub. No.: US 2021/0383268 A1), in view of Miroshnikov et al. hereafter Miroshnikov_2 (Pub. No.: US 2021/0383275 A1).
Regarding Claim 1, Miroshnikov_1 disclose a computing platform comprising:
at least one processor; non-transitory computer-readable medium; and program instructions stored on the non-transitory computer-readable medium that are executable by the at least one processor such that the computing platform is configured to (Miroshnikov_1: [0014]: a non-transitory computer-readable media storing computer instructions is disclosed. The instructions, when executed by one or more processors, cause the one or more processors to perform the steps comprising:):
train a model object for a data science model using a machine learning process, wherein the model object is trained to (i) receive an input data record comprising a set of input variables and (ii) output a score for the input data record (Miroshnikov_1: [0031]-[0032]: FIG. 2 illustrates a system 200 for evaluating a trained ML model 150 … the ML model 150 defines a score function that is used in conjunction with a threshold to construct a classifier. The score function evaluates a set of input variables and generates a score (e.g., a scalar value) that is compared to the threshold value; Examiner’s Remark (ER): A machine learning model is trained in order to output a score.);
arrange the set of input variables into two or more variable groups based on dependencies between respective input variables, where each variable group comprises at least one input variable (Miroshnikov_1: [0060]: At step 308, one or more groups of input variables are identified that contribute to the bias of the model. In an embodiment, a clustering algorithm is used to divide the input variables in the input vector into groups. The clustering algorithm can be based on correlation between variables such that the input variables within a group are more strongly correlated with other variables in the group than any variables in other groups; Examiner’s Remark (ER): The input variables are arranged into a group based on correlation (i.e., dependencies));
identify a given input data record to be scored by the model object (Miroshnikov_1: [0066]: Once the groups of input variables are identified, a bias contribution value is calculated for each group of input variables; Examiner’s Remark (ER): The groups of input variables are identified to be scored);
for each respective input variable in each respective variable group of the model object, perform a given number of iterations of the following steps: identify a sample historical data record from a set of historical data records (Miroshnikov_1: [0069]: A covariance matrix for the input variables of the input vector is developed, based on an analysis of the training data set. Initially, all input variables are assigned to a single group. Primary eigenvectors of the covariance matrix are used to split the input variables in the group into two groups. The process is repeated recursively for each group until all groups include only a single input variable; Examiner’s Remark (ER): The training data set (i.e., sample historical data record) is analyzed (i.e., identified.));
use the given input data record, the sample historical data record, the randomly-selected group coalition, and the randomly-selected variable coalition to compute an iteration-specific contribution value for the respective input variable (Miroshnikov_1: [0066], [0069]-[0070]: Once the groups of input variables are identified, a bias contribution value is calculated for each group of input variables. … A covariance matrix for the input variables of the input vector is developed, based on an analysis of the training data set. … At step 404, a bias contribution value is calculated for each group of input variables. The bias contribution value is based on a quantile function related to a metric such as a grouped PDP metric or a SHAP metric. In an embodiment, the bias contribution value is based on a cumulative distribution function (CDF) for the score explainer function of each group of one or more input variables; Examiner’s Remark (ER): The set of input variables (i.e., input data record) are grouped, and used with the training set (i.e., sample historical data record), in order to calculate the contribution values. The contribution values are calculated based on a quantile function or cumulative distribution function, which are based on random variables);
Miroshnikov_1 do not explicitly disclose:
select a random group coalition comprising the respective variable group and zero or more other variable groups;
select a random variable coalition, within the respective variable group, comprising the respective input variable and zero or more other input variables;
and
for each respective input variable of the model object, aggregate the iteration specific contribution values calculated for each iteration and thereby determine an aggregated contribution value for the respective input variable.
Miroshnikov_2 teaches:
select a random group coalition comprising the respective variable group and zero or more other variable groups (Miroshnikov_2: [0050]-[0051]: At step 208, a subset of the training data set is selected. The subset of the training data set can be any random sample of M pairs of input vectors and corresponding target output vectors; Examiner’s Remark (ER): A random sample of M pairs (i.e., random group coalition) are selected);
select a random variable coalition, within the respective variable group, comprising the respective input variable and zero or more other input variables (Miroshnikov_2: [0055]-0056]: FIG. 3 illustrates input variables for an input vector completely separated in a hierarchical fashion based on a clustering algorithm, in accordance with some embodiments. Any clustering algorithm can be employed as long as the clustering algorithm is designed to group input variables based on dependencies such that the dependencies of input variables within a group are stronger than the dependencies of input variables between groups………As depicted in FIG. 3, the input vector includes a set of 40 features ( e.g., input variables). Examples of the
features include "MONTHS SINCE_LAST_GIFT" and "IN_HOUSE," which are random names given to features and not related to any specific data set; Examiner’s Remark (ER): random names given to features and not related to any specific data set (i.e. random variable coalition));
and
for each respective input variable of the model object, aggregate the iteration specific contribution values calculated for each iteration and thereby determine an aggregated contribution value for the respective input variable (Miroshnikov_2: [0078]-[0079]: Furthermore, given the summation operator, this process is repeated over all possible coalitions S given a random order of features joining the coalition. … It will be appreciated that, given the grouping of input variables by the clustering algorithm as a starting point and due to the additivity property of SHAP values, we can assign the SHAP value of a group of input variables to be the sum of the SHAP values of each variable in the group; Examiner’s Remark (ER): A SHAP value (i.e., iteration-specific contribution value) is calculated for each group of input variables, which sums (i.e., aggregate) each of the SHAP values of each variable in the group).
Miroshnikov_1 and Miroshnikov_2 are analogous art because they are from the same field of endeavor. They both relate to machine-learning.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the above system and method for mitigating bias in classification scores generated by machine learning models, as taught by Miroshnikov_1, and incorporating the use of utilizing grouped partial dependence plots and game-theoretic concepts and their extensions in the generation of adverse action reason codes, as taught by Miroshnikov_2.
One of ordinary skill in the art would have been motivated to do this modification in order to arrive at a computing platform that incorporates a random group selection, thereby improving the technique, as suggested by Miroshnikov_2 (Miroshnikov_2: [0033]).
Regarding Claim 2, the combinations of Miroshnikov_1 and Miroshnikov_2 teaches all the limitations of claim 1.
Miroshnikov_2 further disclose:
generate a random ordering of the two or more variable groups; and define the randomly-selected group coalition as including the respective variable group and all other variable groups that precede the respective variable group in the generated random ordering (Miroshnikov_2: [0078]-[0079]: given the summation operator, this process is repeated over all possible coalitions S given a random order of features joining the coalition. … given the grouping of input variables by the clustering algorithm as a starting point and due to the additivity property of SHAP values, we can assign the SHAP value of a group of input variables to be the sum of the SHAP values of each variable in the group; Examiner’s Remark (ER): The coalitions (i.e., variable groups) are given in a random order. The SHAP value is assigned to a group of input variables).
Regarding Claim 3, the combinations of Miroshnikov_1 and Miroshnikov_2 teaches all the limitations of claim 1.
Miroshnikov_2 further disclose:
generate a random ordering of the input variables in the respective variable group; and
define the randomly-selected variable coalition as including the respective input variable
and all other input variables that precede the respective input variable in the generated random
ordering (Miroshnikov_2: [0055]-0056]: FIG. 3 illustrates input variables for an input vector completely separated in a hierarchical fashion based on a clustering algorithm, in accordance with some embodiments. Any clustering algorithm can be employed as long as the clustering algorithm is designed to group input variables based on dependencies such that the dependencies of input variables within a group are stronger than the dependencies of input variables between groups………As depicted in FIG. 3, the input vector includes a set of 40 features ( e.g., input variables). Examples of the features include "MONTHS SINCE_LAST_GIFT" and "IN_HOUSE," which are random names given to features and not related to any specific data set; Examiner’s Remark (ER): random names given to features and not related to any specific data set (i.e. random variable coalition); Miroshnikov_2: [0078]-[0079]: given the summation operator, this process is repeated over all possible coalitions S given a random order of features joining the coalition. … given the grouping of input variables by the clustering algorithm as a starting point and due to the additivity property of SHAP values, we can assign the SHAP value of a group of input variables to be the sum of the SHAP values of each variable in the group; Examiner’s Remark (ER): The coalitions (i.e., variable groups) are given in a random order. The SHAP value is assigned to a group of input variables).
Motivation to combine Miroshnikov_1 and Miroshnikov_2 is same here as claim 1.
Regarding Claim 4, the combinations of Miroshnikov_1 and Miroshnikov_2 teaches all the limitations of claim 1.
Miroshnikov_2 further disclose:
generate a first synthetic data record comprising a mix of input variables from (i) the given
input data record and (ii) the sample historical data record; generate a second synthetic data record comprising an adjusted mix of input variables from (i) the given input data record and (ii) the sample historical data record; use the trained model object to determine a first score for the first synthetic data record and a second score for the second synthetic data record; and calculate a difference between the first score and the second score, wherein the difference is the iteration-specific contribution value (Miroshnikov_2: [0047], [0078]: At step 204, an ML model is trained based on the training data set. In an embodiment, the ML model can be trained by processing each input vector and then adjusting the parameters of the ML model to minimize a difference between the output vector generated by the ML model and the ground truth target output vector. Adjusting the parameters of the ML model can include using backpropagation with gradient descent or any other technically feasible algorithm for training the parameters of the model. … the marginal value given by a difference in the expected value of the model f given the sub-vector XS∪{i} and the expected value of the model f given the sub-vector XS; Examiner’s Remark (ER): The difference between two output scores may be determined based on the group of input variables, and training set (i.e., sample historical data record)).
Motivation to combine Miroshnikov_1 and Miroshnikov_2 is same here as claim 1.
Regarding Claim 6, the combinations of Miroshnikov_1 and Miroshnikov_2 teaches all the limitations of claim 1.
Miroshnikov_2 further disclose:
determine an average of the iteration-specific contribution values for each iteration over the given number of iterations (Miroshnikov_2: [0075]: The SHAP algorithm is based on the game-theoretic concept of the Shapley value, which takes into account all the different combinations between the feature of interest and the rest of the features in the input vector and produces a score (e.g., a scalar value) that represents the contribution of that feature value to the deviation of the model prediction for the specific instance of the input vector from the model's average prediction given the set of training data used to train the model; Examiner’s Remark (ER): The Shapley values (i.e., contribution values) are averaged).
Motivation to combine Miroshnikov_1 and Miroshnikov_2 is same here as claim 1.
Regarding Claim 7, the combinations of Miroshnikov_1 and Miroshnikov_2 teaches all the limitations of claim 1.
Miroshnikov_2 further disclose:
while performing the given number of iterations for a first input variable of the model
object, perform the given number of iterations for each other input variable of the model object (Miroshnikov_2: [0078]: given the summation operator, this process is repeated over all possible coalitions S given a random order of features joining the coalition. Examiner’s Remark (ER): The process is repeated for all the variable groups).
Motivation to combine Miroshnikov_1 and Miroshnikov_2 is same here as claim 1.
Regarding Claim 9, the combinations of Miroshnikov_1 and Miroshnikov_2 teaches all the limitations of claim 1.
Miroshnikov_2 further disclose:
identify a randomly-sampled historical data record from a set of historical data records (Miroshnikov_2: [0078]: At step 208, a subset of the training data set is selected. The subset of the training data set can be any random sample of M pairs of input vectors and corresponding target output vectors. Examiner’s Remark (ER): The training data (i.e., historical data records) are randomly sampled).
Motivation to combine Miroshnikov_1 and Miroshnikov_2 is same here as claim 1.
Regarding Claims 10-13, 15-16, and 18-20, the combinations of Miroshnikov_1 and Miroshnikov_2 teaches all of the limitations of claims 1-4, 6-7 and 9 in computing platform form rather than in computer readable medium and method form. Miroshnikov_1 also discloses a computer readable medium [0005], and a method [0005]. Therefore, the supporting rationale of the rejection to claims 1-4, 6-7 and 9 applies equally as well to those elements of claims 10-13, 15-16, and 18-20.
8. Claims 5 and 14 are rejected under 35 U.S.C. 103 as being obvious over Miroshnikov et al. hereafter Miroshnikov_1 (Pub. No.: US 2021/0383268 A1), in view of Miroshnikov et al. hereafter Miroshnikov_2 (Pub. No.: US 2021/0383275 A1), further in view of Torsten Schiemenz hereafter Schiemenz (Pub. No.: US 2019/0087744 A1).
Regarding Claim 5, the combinations of Miroshnikov_1 and Miroshnikov_2 teaches all the limitations of claim 4. The combinations further teach:
identify a subset of input variables that are included in the randomly-selected group
coalition and the randomly-selected variable coalition; for the identified subset of input variables, use values from the given input data record for the first synthetic data record; and
for each other input variable, use values from the sample historical data record for
the first synthetic data record; and wherein the program instructions that are executable by the at least one processor such that the computing platform is configured to generate the second synthetic data record comprising the adjusted mix of input variables comprise program instructions that are executable by the at least one processor such that the computing platform is configured to:
(Miroshnikov_1: [0066], [0069]-[0070]: Once the groups of input variables are identified, a bias contribution value is calculated for each group of input variables. … A covariance matrix for the input variables of the input vector is developed, based on an analysis of the training data set. … At step 404, a bias contribution value is calculated for each group of input variables. The bias contribution value is based on a quantile function related to a metric such as a grouped PDP metric or a SHAP metric. In an embodiment, the bias contribution value is based on a cumulative distribution function (CDF) for the score explainer function of each group of one or more input variables. Examiner’s Remark (ER): The set of input variables (i.e., input data record) are used with the training set (i.e., sample historical data record); Also, see Miroshnikov_2: [0050]-[0051]: At step 208, a subset of the training data set is selected. The subset of the training data set can be any random sample of M pairs of input vectors and corresponding target output vectors. Examiner’s Remark (ER): A random sample of M pairs (i.e., random group coalition) are selected).
However, the combination of Miroshnikov_1, and of Miroshnikov_2 do not explicitly teach:
for the identified subset of input variables, excluding input variables included in the respective group, use values from the given input data record for the second synthetic data record; and for each other input variable, including input variables included in the respective group, use values from the sample historical data record for the second synthetic data record.
Schiemenz teaches:
for the identified subset of input variables, excluding input variables included in the respective group, use values from the given input data record for the second synthetic data record; and for each other input variable, including input variables included in the respective group, use values from the sample historical data record for the second synthetic data record (Schiemenz: [0003], The machine-learning model includes a plurality of variables. Variables are iteratively removed from the machine-learning model, and for each iteration, the machine-learning model is applied with one or more variables removed from the dataset to generate a second output. For each iteration, the first and second outputs are compared. A subset of the removed variables having impact below a predetermined threshold on an output of the machine-learning model is determined based on the comparisons. An optimized machine-learning model that omits the subset of variables is applied to new data with the processing system to generate an output for the new data. Examiner’s Remark (ER): The variables are excluded in the iteration process).
Miroshnikov_1, Miroshnikov _2 and Schiemenz are analogous art because they are from the same field of endeavor. All of them relate to machine-learning.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the above system and method for mitigating bias in classification scores generated by machine learning models, as taught by the combinations of Miroshnikov_1 and Miroshnikov _2, and incorporating the teaching of automatic selection of variables for a machine-learning model, as taught by Schiemenz.
One of ordinary skill in the art would have been motivated to do this modification in order to arrive at a computing platform that provides prediction accuracy by minimizing variables, thereby improving the technique, as suggested by Schiemenz (Schiemenz: [0006]).
Regarding Claim 14, the combinations of Miroshnikov_1, Miroshnikov_2 and Schiemenz teaches all of the limitations of claim 5 in computing platform form rather than in computer readable medium form. Miroshnikov_1 also discloses a computer readable medium [0005]. Therefore, the supporting rationale of the rejection to claim 5 applies equally as well to those elements of claim 14.
9. Claims 8 and 17 are rejected under 35 U.S.C. 103 as being obvious over Miroshnikov et al. hereafter Miroshnikov_1 (Pub. No.: US 2021/0383268 A1), in view of Miroshnikov et al. hereafter Miroshnikov_2 (Pub. No.: US 2021/0383275 A1), further in view of Trenner et al. hereafter Trenner (Pub. No.: US 2011/0270782 A1).
Regarding Claim 8, the combinations of Miroshnikov_1 and Miroshnikov_2 teaches all the limitations of claim 8. However, the combinations fail to teach:
wherein the given number of iterations is 1,000 or more iterations.
Trenner teaches:
wherein the given number of iterations is 1,000 or more iterations (Trenner: [0164]: Determining whether there are more iterations may include determining whether more than a predetermined number of iterations (e.g., 100, 1000) iterations have been performed. Examiner’s Remark (ER): A thousand iterations may be performed).
Miroshnikov_1, Miroshnikov _2 and Trenner are analogous art because they are from the same field of endeavor. All of them relate to statistics.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the above system and method for mitigating bias in classification scores generated by machine learning models, as taught by the combinations of Miroshnikov_1 and Miroshnikov _2, and incorporating the teaching of systems and methods for determining investment strategies, as taught by Trenner.
One of ordinary skill in the art would have been motivated to do this modification for the purposes of determining investment strategies, as suggested by Trenner (Trenner: [0007]).
Regarding Claim 17, the combinations of Miroshnikov_1, Miroshnikov_2 and Trenner teaches all of the limitations of claim 8 in computing platform form rather than in computer readable medium form. Miroshnikov_1 also discloses a computer readable medium [0005]. Therefore, the supporting rationale of the rejection to claim 8 applies equally as well to those elements of claim 17.
Conclusion
10. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Miroshnikov et al. (Pub. No.: US 20210350272 A1) teaches a framework for interpreting machine learning models is proposed that utilizes interpretability methods to determine the contribution of groups of input variables to the output of the model. Input variables are grouped based on correlation with other input variables.
Kenthapadi et al. (Pub. No.: US 20200372304 A1) provide a system for quantifying machine learning model bias by obtaining a ranking of recommended candidates outputted by the machine learning model after the qualified candidates are inputted into the machine learning model.
Goldszmidt et al. (Pub. No.: US 20210224687 A1) conceptually presents systems and methods for triggering an update to a machine-learning model upon detecting that a distribution of particular (e.g., recently collected) input data set is sufficiently different from a distribution training input data set used to train the model.
Dodwell et al. (Pub. No.: US 20210174222 A1) defines operations that may include receiving, by a processing system, project data defining a proposed machine learning (ML) project of an entity and storing the project data in a project database with other project data for other projects..
11. Examiner’s Remarks: Examiner has cited particular columns and line numbers in the references applied to the claims above for the convenience of the applicant. Although the specified citations are representative of the teachings of the art and are applied to specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested from the applicant in preparing responses, to fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the Examiner. In the case of amending the claimed invention, Applicant is respectfully requested to indicate the portion(s) of the specification which dictate(s) the structure relied on for proper interpretation and also to verify and ascertain the metes and bounds of the claimed invention.
Correspondence Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IFTEKHAR A KHAN whose telephone number is (571)272-5699. The examiner can normally be reached on M-F from 9:00AM-6:00PM (CST). If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emerson Puente can be reached on (571)272-3652. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center and Private PAIR to authorized users only. Should you have questions about access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/IFTEKHAR A KHAN/Primary Examiner, Art Unit 2187
Prosecution Insights