Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/07/2026 has been entered.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 04/07/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Status of Claims
The amendments filed 04/07/2026 has been entered. The status of the claims is as follows:
Claims 1, 3-8, 11-15, 18-19, 21-25 remain pending in the application.
Claims 21-25 are new.
Claims 1, 3, 8, 15 are amended.
Response to Arguments
In reference to the rejection of claims under 35 USC 101:
Argument 1: Claims Do Not Recite an Abstract Idea:
Applicant asserts in Remarks pg. 2-4 that the Examiner erred in concluding that claim 1 recites an abstract idea. The claim as a whole is directed to a specific technological solution for a technical problem in machine learning systems, not to an abstract idea. The claimed features, taken together, describe a specific technological solution to a technical problem in machine learning systems. The claim is not directed to a mathematical algorithm or mental process in the abstract, but rather to a specific application of technology to improve the functioning of machine learning models by detecting and remediating bias. Applicant further states that the claims do not merely use a computer as a tool to perform an abstract process; rather, they improve the functioning of the computer system (the machine learning model) itself. The specification explains that machine learning models may exhibit bias in their predictions across different groups, which is a technical problem specific to machine learning systems. The claimed invention therefore provides a technological solution to this technical problem.
Response 1:
Applicant’s argument is not persuasive. Although the claim states that bias is detected and remediated, the claim merely recites the desired result of improving model fairness, i.e., decreasing or eliminating an output difference between 2 groups, without reciting a specific technological improvement to the machine learning model itself. The claimed steps use generic computing components to receive model/ data representations, compute an output difference, determine reasons using feature importance distributions, provide an output, and retrain the model using a remediation engine that adjusts features. The claim does not specify any particular model architecture, training algorithm, feature-adjustment technique, or computer improvement that changes how the ML system technically operates. Therefore, the claim is directed to analyzing data and applying the result to reduce bias, rather than a specific technological improvement to ML functionality.
Argument 2: Alleged Abstract Idea is Integrated into a Practical Application:
Applicant asserts on Remarks pg. 4-6 that even assuming, arguendo, that the claim recites a judicial exception, the claim as a whole integrates any such exception into a practical application. The MPEP requires consideration of the claim as a whole, "Accordingly, after determining that a claim recites a judicial exception in Step 2A Prong One, examiners should evaluate whether the claim as a whole integrates the recited judicial exception into a practical application of the exception in Step 2A Prong Two. A claim that integrates a judicial exception into a practical application will apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception." (MPEP § 2106.04(d), emphasis added). The Office's analysis for Step 2A Prong Two evaluated only individual parts of the claim in isolation, which is incorrect. The Examiner characterized individual limitations as "using a computer as a tool," "data gathering," or "data outputting" without considering how these limitations work together in the claim as a whole to provide a practical application. To determine if the claim recites a practical application, the Office must look at the claim as a whole, not just isolated sections of the claim. Therefore, the § 101 rejection is improper. Further, claim 1 is directed to the practical application of improving the fairness and accuracy of machine learning models by detecting, analyzing, and remediating bias in model predictions across different groups. This practical application addresses a real-world technical problem: that machine learning models may treat different groups of people differently in ways that are problematic from legal, ethical, and technical perspectives.
Response 2:
Applicant’s argument is not persuasive. The claim has been considered as a whole, but the recited steps collectively amount to receiving data/ model representations, comparing model outputs between 2 groups, identifying features contributing to the difference, outputting reasons, and broadly retraining the model to reduce bias. These steps describe the goal of detecting and remediating bias, but do not recite a specific technological solution for improving the operation of a computer of machine learning model. The “remediation engine” and “feature engineering sub-engine” are claimed only functionally, without any particular algorithm, model architecture, or concrete feature-adjustment technique. Thus, the claim merely applies the abstract idea using generic computing components to achieve a desired fairness result, and does not integrate the judicial exception into a practical application.
Argument 3: Claims Recite Additional Elements that amount to significantly more than the judicial exception.
Applicant asserts on Remarks pg. 6-7 that even assuming, arguendo, that the claim is directed to a judicial exception that is not integrated into a practical application, the claim still recites additional elements that amount to significantly more than the exception. The claim improves machine learning technology by providing a method to detect and eliminate bias. This is an improvement to the functioning of the computer itself (i.e., the machine learning model's operation), which constitutes "significantly more" under Step 2B. See MPEP § 2106.05(a).
Response 3:
Applicant’s argument is not persuasive. The additional elements, considered individually and as an ordered combination, do not amount to significantly more than the judicial exception. The claim merely uses a generic computing machine to analyze model outputs, compare groups, identify feature contributions, generate a broadly recited “remediation engine”, and retrain the model to achieve the desired result of reduced or eliminated bias. The recited “feature importance distributions”, “remediation engine”, and “feature engineering sub-engine” are described functionally and do not specify any unconventional algorithm, model architecture, or particular technical implementation that improves computer operation. Improving fairness, reliability, or legal compliance of model outputs is a desired result, not itself an inventive concept. Therefore, the claim does not recite additional elements that transform the abstract idea into patent-eligible subject matter.
In reference to the Rejection of Claims under 35 USC 103:
Applicant’s arguments, filed 12/01/2025, with respect to the rejection(s) of claim(s) under 35 U.S.C 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Scott Lundberg (“Explaining Quantitative Measures of Fairness”), Datta et al. (“Algorithmic Transparency via Quantitative Input Influence: Theory and Experiments with Learning Systems”) and Lundberg et al. (“Consistent Individualized Feature Attribution for Tree Ensembles”)
Claim Rejections - 35 USC § 112(a) – New Subject Matter
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim 23 is rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 23 recites “identifying a subset of features from the feature vector having contributions to the output difference greater than a threshold”. However, the originally filed specification does not provide support for comparing feature-level contributions to an output difference against a threshold or selecting features based on that comparison. Paragraph ¶[0033] of the specification only describes thresholds for terminating model training, such as an accuracy threshold, random-chance threshold, or performance plateau. These thresholds relate to model accuracy or training termination, not to feature contributions or output differences.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 3-8, 11-15, 18-19, 21-25 are rejected under U.S.C 101 for containing an abstract idea without significantly more.
Regarding claim 1:
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is a process.
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
computing, [at the computing machine], an output difference between an output of the machine learning model applied to the first data segment and an output of the machine learning model applied to the second data segment; This limitation is directed to mathematical calculation as it is calculating the output difference and paragraph ¶[0016] of the Instant Specification has defined this output difference as a statistical difference measurement (e.g., an arithmetic difference, a disparate impact ratio, a difference of means, and a Wasserstein distance) (see MPEP 2106.04(a)(2) l. C.)
determining, [using the computing machine], a set of reasons for the computed output difference between the first group and the second group based on a set of metrics defining distance between feature importance distributions, the set of reasons identifying a set of features driving a bias between the first group and the second group from a feature vector of the machine learning model along with a relative contribution of each feature to the computed output difference driving the bias between the first group and the second group; and - This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
at a computing machine– This limitation is directed to a computer merely used as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
receiving, [at a computing machine], a representation of a machine learning model, a representation of a first data segment associated with a first group, and a representation of a second data segment associated with a second group that is a complement of the first group; - This limitation is directed to insignificant extra-solution activity – mere data gathering (see MPEP 2106.05(g))
providing an output representing the set of reasons for the output difference between the first group and the second group. - This limitation is directed to insignificant extra-solution activity – mere data outputting (see MPEP 2106.05(g))
using the computing machine This limitation is directed to a computer merely used as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
generating, [using the computing machine], a remediation engine for the machine learning model to decrease a magnitude of the output difference between the first group and the second group, the remediation engine comprising a feature engineering sub-engine that adjusts features from the set of features based on the determined set of reasons; and. - Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
retraining the model using the remediation engine; - Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
the model providing outputs that eliminate bias between the first group and the second group Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No, there are no additional elements that amount to significantly more than the judicial exception. The additional elements are:
at a computing machine– This limitation is directed to a computer merely used as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
receiving, [at a computing machine], a representation of a machine learning model, a representation of a first data segment associated with a first group, and a representation of a second data segment associated with a second group that is a complement of the first group; - This limitation is directed to insignificant extra-solution activity – mere data gathering (see MPEP 2106.05(g))
providing an output representing the set of reasons for the output difference between the first group and the second group. - This limitation is directed to insignificant extra-solution activity – mere data outputting (see MPEP 2106.05(g))
using the computing machine This limitation is directed to a computer merely used as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
generating, [using the computing machine], a remediation engine for the machine learning model to decrease a magnitude of the output difference between the first group and the second group, the remediation engine comprising a feature engineering sub-engine that adjusts features from the set of features based on the determined set of reasons; and. - Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
retraining the model using the remediation engine; - Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
the model providing outputs that eliminate bias between the first group and the second group Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
Regarding claim 3,
Claim 3 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which includes an abstract idea (see rejection for claim 1). The additional limitations:
wherein [the remediation engine] selects features for adjustment based on a user input received at the computing machine. This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
the remediation engine – This limitation is directed to a computer merely used as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
Regarding claim 4,
Claim 4 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which includes an abstract idea (see rejection for claim 1). The additional limitations:
wherein the set of metrics comprises a metric based on a Difference of Means or a Wasserstein Distance. This claim merely recites a further limitation on the determining, using the computing machine, a set of reasons for the computed output difference based on a set of metrics defining distance between feature importance distributions, the set of reasons identifying a set of features from a feature vector of the machine learning model along with a relative contribution of each feature to the computed output difference from Claim 1 which was directed to an abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
Regarding claim 5,
Claim 5 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which includes an abstract idea (see rejection for claim 1). The additional limitations:
wherein the output difference comprises a statistical difference measurement between outputs for the first group and outputs for the second group. This claim merely recites a further limitation on the computing, at the computing machine, an output difference between an output of the machine learning model applied to the first data segment and an output of the machine learning model applied to the second data segment from Claim 1 which was directed to mathematical calculation (see MPEP 2106.04(a)(2) l. C.)
Regarding claim 6,
Claim 6 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 5 which includes an abstract idea (see rejection for claim 5). The additional limitations:
wherein the statistical difference measurement is one or more of: an arithmetic difference, a disparate impact ratio, a difference of means, and a Wasserstein distance. This claim merely recites a further limitation on the wherein the output difference comprises a statistical difference measurement from Claim 5 which was directed to mathematical calculation (see MPEP 2106.04(a)(2) l. C.)
Regarding claim 7,
Claim 7 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 5 which includes an abstract idea (see rejection for claim 5). The additional limitations:
wherein the statistical difference measurement comprises a difference of model outputs for the first group and outputs for the second group, the model outputs being one or more of: log-odds scores, probabilities of classification, and class outputs. This claim merely recites a further limitation on the wherein the output difference comprises a statistical difference measurement from Claim 5 which was directed to mathematical calculation (see MPEP 2106.04(a)(2) l. C.)
Regarding claim 8,
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is a process.
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
compute an output difference between an output of the machine learning model applied to the first data segment and an output of the machine learning model applied to the second data segment; This limitation is directed to mathematical calculation as it is calculating the output difference and paragraph ¶[0016] of the Instant Specification has defined this output difference as a statistical difference measurement (e.g., an arithmetic difference, a disparate impact ratio, a difference of means, and a Wasserstein distance) (see MPEP 2106.04(a)(2) l. C.)
determine a set of reasons for the computed output difference between the first group and the second group based on a set of metrics defining distance between feature importance distributions, the set of reasons identifying a set of features driving a bias between the first group and the second group from a feature vector of the machine learning model along with a relative contribution of each feature to the computed output difference driving the bias between the first group and the second group; and - This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
A non-transitory machine-readable medium storing instructions which, when executed by processing circuitry, cause the processing circuitry to This limitation is directed to a computer merely used as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
receive a representation of a machine learning model, a representation of a first data segment associated with a first group, and a representation of a second data segment associated with a second group that is a complement of the first group; - This limitation is directed to insignificant extra-solution activity – mere data gathering (see MPEP 2106.05(g))
provide an output representing the set of reasons for the output difference between the first group and the second group. - This limitation is directed to insignificant extra-solution activity – mere data outputting (see MPEP 2106.05(g))
generating, [using the computing machine], a remediation engine for the machine learning model to decrease a magnitude of the output difference between the first group and the second group, the remediation engine comprising a feature engineering sub-engine that adjusts features from the set of features based on the determined set of reasons; and. - Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
retraining the model using the remediation engine; - Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
the model providing outputs that eliminate bias between the first group and the second group Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No, there are no additional elements that amount to significantly more than the judicial exception. The additional elements are:
A non-transitory machine-readable medium storing instructions which, when executed by processing circuitry, cause the processing circuitry to This limitation is directed to a computer merely used as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
receive a representation of a machine learning model, a representation of a first data segment associated with a first group, and a representation of a second data segment associated with a second group that is a complement of the first group; - This limitation is directed to insignificant extra-solution activity – mere data gathering (see MPEP 2106.05(g))
provide an output representing the set of reasons for the output difference between the first group and the second group. - This limitation is directed to insignificant extra-solution activity – mere data outputting (see MPEP 2106.05(g))
generating, [using the computing machine], a remediation engine for the machine learning model to decrease a magnitude of the output difference between the first group and the second group, the remediation engine comprising a feature engineering sub-engine that adjusts features from the set of features based on the determined set of reasons; and. - Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
retraining the model using the remediation engine; - Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
the model providing outputs that eliminate bias between the first group and the second group Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
Regarding claim 11,
Claim 11 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 8 which includes an abstract idea (see rejection for claim 8). The additional limitations:
wherein the set of metrics comprises a metric based on a Difference of Means or a Wasserstein Distance. This claim merely recites a further limitation on the determinine a set of reasons for the computed output difference based on a set of metrics defining distance between feature importance distributions, the set of reasons identifying a set of features from a feature vector of the machine learning model along with a relative contribution of each feature to the computed output difference from Claim 8 which was directed to an abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
Regarding claim 12,
Claim 12 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 8 which includes an abstract idea (see rejection for claim 8). The additional limitations:
wherein the output difference comprises a statistical difference measurement between outputs for the first group and outputs for the second group. This claim merely recites a further limitation on the compute an output difference between an output of the machine learning model applied to the first data segment and an output of the machine learning model applied to the second data segment from Claim 8 which was directed to mathematical calculation (see MPEP 2106.04(a)(2) l. C.)
Regarding claim 13,
Claim 13 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 12 which includes an abstract idea (see rejection for claim 12). The additional limitations:
wherein the statistical difference measurement is one or more of: an arithmetic difference, a disparate impact ratio, a difference of means, and a Wasserstein distance. This claim merely recites a further limitation on the wherein the output difference comprises a statistical difference measurement from Claim 12 which was directed to mathematical calculation (see MPEP 2106.04(a)(2) l. C.)
Regarding claim 14,
Claim 14 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 12 which includes an abstract idea (see rejection for claim 12). The additional limitations:
wherein the statistical difference measurement comprises a difference of model outputs for the first group and outputs for the second group, the model outputs being one or more of: log-odds scores, probabilities of classification, and class outputs. This claim merely recites a further limitation on the wherein the output difference comprises a statistical difference measurement from Claim 12 which was directed to mathematical calculation (see MPEP 2106.04(a)(2) l. C.)
Regarding claim 15,
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is a process.
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
compute an output difference between an output of the machine learning model applied to the first data segment and an output of the machine learning model applied to the second data segment; This limitation is directed to mathematical calculation as it is calculating the output difference and paragraph ¶[0016] of the Instant Specification has defined this output difference as a statistical difference measurement (e.g., an arithmetic difference, a disparate impact ratio, a difference of means, and a Wasserstein distance) (see MPEP 2106.04(a)(2) l. C.)
determine a set of reasons for the computed output difference between the first group and the second group based on a set of metrics defining distance between feature importance distributions, the set of reasons identifying a set of features driving a bias between the first group and the second group from a feature vector of the machine learning model along with a relative contribution of each feature to the computed output difference driving the bias between the first group and the second group; and - This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
processing circuitry; and This limitation is directed to a computer merely used as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
a memory storing instructions which, when executed by the processing circuitry, cause the processing circuitry to This limitation is directed to a computer merely used as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
receive a representation of a machine learning model, a representation of a first data segment associated with a first group, and a representation of a second data segment associated with a second group that is a complement of the first group; - This limitation is directed to insignificant extra-solution activity – mere data gathering (see MPEP 2106.05(g))
provide an output representing the set of reasons for the output difference between the first group and the second group. - This limitation is directed to insignificant extra-solution activity – mere data outputting (see MPEP 2106.05(g))
generating, [using the computing machine], a remediation engine for the machine learning model to decrease a magnitude of the output difference between the first group and the second group, the remediation engine comprising a feature engineering sub-engine that adjusts features from the set of features based on the determined set of reasons; and. - Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
retraining the model using the remediation engine; - Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
the model providing outputs that eliminate bias between the first group and the second group Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No, there are no additional elements that amount to significantly more than the judicial exception. The additional elements are:
processing circuitry; and This limitation is directed to a computer merely used as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
a memory storing instructions which, when executed by the processing circuitry, cause the processing circuitry to This limitation is directed to a computer merely used as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
receive a representation of a machine learning model, a representation of a first data segment associated with a first group, and a representation of a second data segment associated with a second group that is a complement of the first group; - This limitation is directed to insignificant extra-solution activity – mere data gathering (see MPEP 2106.05(g))
provide an output representing the set of reasons for the output difference between the first group and the second group. - This limitation is directed to insignificant extra-solution activity – mere data outputting (see MPEP 2106.05(g))
generating, [using the computing machine], a remediation engine for the machine learning model to decrease a magnitude of the output difference between the first group and the second group, the remediation engine comprising a feature engineering sub-engine that adjusts features from the set of features based on the determined set of reasons; and. - Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
retraining the model using the remediation engine; - Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
the model providing outputs that eliminate bias between the first group and the second group Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
Regarding claim 18,
Claim 18 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 15 which includes an abstract idea (see rejection for claim 15). The additional limitations:
wherein the set of metrics comprises a metric based on a Difference of Means or a Wasserstein Distance. This claim merely recites a further limitation on the determine a set of reasons for the computed output difference based on a set of metrics defining distance between feature importance distributions, the set of reasons identifying a set of features from a feature vector of the machine learning model along with a relative contribution of each feature to the computed output difference from Claim 15 which was directed to an abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
Regarding claim 19,
Claim 19 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 15 which includes an abstract idea (see rejection for claim 15). The additional limitations:
wherein the output difference comprises a statistical difference measurement between outputs for the first group and outputs for the second group. This claim merely recites a further limitation on the compute an output difference between an output of the machine learning model applied to the first data segment and an output of the machine learning model applied to the second data segment from Claim 15 which was directed to mathematical calculation (see MPEP 2106.04(a)(2) l. C.)
Regarding claim 21,
Claim 21 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which includes an abstract idea (see rejection for claim 1). The additional limitations:
generating a visualization displaying the set of features ordered by the relative contribution to the computed output difference. Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
Regarding claim 22,
Claim 22 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which includes an abstract idea (see rejection for claim 1). The additional limitations:
computing a contribution to the output difference for each feature in the feature vector based on a difference between a feature importance distribution for the first group and a feature importance distribution for the second group; and - This limitation is directed to mathematical calculation (see MPEP 2106.04(a)(2) l. C.)
ordering the features according to contributions to the output difference. - This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
Regarding claim 23,
Claim 23 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which includes an abstract idea (see rejection for claim 1). The additional limitations:
identifying a subset of features from the feature vector having contributions to the output difference greater than a threshold; and This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
associating each feature in the subset with a Quantitative Input Influence representing contribution to the computed output difference This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
Regarding claim 24,
Claim 24 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which includes an abstract idea (see rejection for claim 1). The additional limitations:
generating an influence sensitivity plot for at least one feature identified in the set of reasons, the influence sensitivity plot providing a visual representation of how values of the at least one feature impacts the contribution to model outputs; and Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
presenting the influence sensitivity plot as part of the output representing the set of reasons. Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
Regarding claim 25,
Claim 25 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which includes an abstract idea (see rejection for claim 1). The additional limitations:
generating near clones of an input point from the first data segment and the second data segment by intervening on subsets of features; Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
computing predictions of the machine learning model on the near clones; This limitation is directed to mathematical calculation (see MPEP 2106.04(a)(2) l. C.)
computing feature importance scores using a co-operative game-theoretic index that determines an average marginal contribution of each feature to all possible subsets of features; This limitation is directed to mathematical calculation (see MPEP 2106.04(a)(2) l. C.)
identifying features with differences in feature importance distributions between the first group and the second group as reasons for the output difference This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 3-7, 21 are rejected under 35 U.S.C. 103 as being unpatentable over Julius Adebayo (“Fair ML: ToolBox for Diagnosing Bias in Predictive Modeling”) (hereafter referred to as “Adebayo”) in view of Jiang et al (“Wasserstein Fair Classification”) (hereafter referred to as “Jiang”) and further in view of Scott Lundberg (“Explaining Quantitative Measures of Fairness”)
Regarding Claim 1, Adebayo explicitly discloses:
receiving, at a computing machine, a representation of a machine learning model, a representation of a first data segment associated with a first group, and a representation of a second data segment associated with a second group that is a complement of the first group; (Adebayo, Pg. 32, ¶[1]: “For example, suppose we have a predictive model that is making loan decisions for different groups of people, if the probability of the model approving a loan for a female is less than 80 percent of the probability of the model approving a loan for a male, then, given an 80 percent threshold, this model exhibits disparate impact.”, Pg. 70, Section 6.2.1, Adebayo discloses a dataset with multiple input data attributes (Table 6.1) including income, customer age, asset, marital status, gender, employment status and education level that the bank uses in its models to calculate the credit limit and future probability of default on a loan
PNG
media_image1.png
355
322
media_image1.png
Greyscale
. Here, gender illustrates the first group of male and the second group is female, that is the complement of the first group, as claimed in the limitation above.)
computing, at the computing machine, an output difference between an output of the
machine learning model applied to the first data segment and an output of the machine learning model applied to the second data segment; (Adebayo, Page 83, ¶[1]: “We now define bias, bf as follows:
PNG
media_image2.png
56
502
media_image2.png
Greyscale
. Pr (f(g) = 1|g
∈
PC) corresponds to the probability of the bank giving an individual a loan given that they do not belong to the protected class, while Pr (f(g) = 1|g
∈
P) corresponds to the probability of the bank giving a loan to an individual given that such individual is a member of the protected class. The difference between these two probabilities is the bias of the model f. A predictive model f is said to achieve statistical parity given (G, P, D), if|bf (G, P, D)| <
ϵ
, where
ϵ
corresponds to an infinitesimal number. Statistical parity is achieved when the difference in outcome probabilities for individuals in a protected class, and individuals not in a protected class is infinitesimally small.”) [Examiner’s note: “computing an output difference between an output” is being interpreted as computing the bias of model f, which is also computing the difference between the probability of the bank giving an individual a loan given that they do not belong to the protected class (representation of first data segment) and the probability of the bank giving a loan to an individual given that such individual is a member of the protected class (representation of second data segment.)]
generating, using the computing machine, a remediation engine for the machine learning model to decrease a magnitude of the output difference between the first group and the second group, the remediation engine comprising a feature engineering sub-engine that adjusts features from the set of features based on the determined set of reasons; and retraining the model using the remediation engine, the model providing outputs that eliminate bias between the first group and the second group. (Adebayo, Page 38, ¶[1-2]: “More recently, a new class of studies has emerged. These studies seek to identify and correct bias throughout the predictive modeling process. In general these works can be broadly classified into three categories: data transformation, algorithm manipulation, and outcome manipulation methodologies. For data transformation techniques, the input data to the predictive model is transformed as a means of quantifying bias in the data. In [13], Kamiran and Calders present a method for transforming data labels to remove discrimination. With their proposed method, a Naive-Bayes classifier is trained on positive labels, then a set of highly ranked negatively-labeled items from the protected set are changed to achieve statistical parity of outcomes. The modified data generated is then used to learn a 'fairer ' model. Here, fairness refers to a more equitable distribution of positive outcomes among different classes.”) [Examiner’s note “a remediation engine” is being interpreted as a framework or mechanism that detects biases in model predictions and applies corrective measures to improves fairness, accuracy; “a feature engineering sub-engine” is being interpreted as the “data transformation technique” ]
Adebayo fails to disclose:
determining, using the computing machine, a set of reasons for the computed output difference between the first group and the second group based on a set of metrics defining distance between feature importance distributions,
the set of reasons identifying a set of features driving the bias between the first group and the second group from a feature vector of the machine learning model along with a relative contribution of each feature to the computed output difference; and
providing an output representing the set of reasons for the output difference between the first group and the second group.
However, Lundberg explicitly discloses:
determining, using the computing machine, a set of reasons for the computed output difference between the first group and the second group based on a set of metrics defining distance between feature importance distributions, (Lundberg, Pg. 4, Figure 1:
PNG
media_image3.png
492
1203
media_image3.png
Greyscale
“A simulation study on the effect of sex-specific reporting biases during credit scoring, and how SHAP values can be used to explain the resulting statistical parity differences. (A) On top is the mean credit worthiness of both men and women as predicted by a machine learning model. The difference between the two bar heights is the statistical parity difference (aka. demographic parity, one of the most common quantitative fairness metrics). On the bottom is the statistical parity difference among the SHAP values for each input feature to the model. The sum of the bars on the bottom equals the difference between the bars on the top. (B) The same analysis as done in (A), except that now the income of women is under-reported in the simulated data. This causes a moderately statistically significant disparity in the model’s output, and a huge disparity in the SHAP values for the reported income feature. (C) The same analysis as (B) except now the number of late payments is under-reported for women (which causes the model to underestimate their risk).”) [Examiner’s note: Fig.1 shows the statistical parity difference among SHAP values for each input feature, where the sum of the feature level bars equal to the overall difference between the model outputs for the 2 groups.]
providing an output representing the set of reasons for the output difference between the first group and the second group. . (Lundberg, Pg. 4, Figure 1:
PNG
media_image3.png
492
1203
media_image3.png
Greyscale
“A simulation study on the effect of sex-specific reporting biases during credit scoring, and how SHAP values can be used to explain the resulting statistical parity differences. (A) On top is the mean credit worthiness of both men and women as predicted by a machine learning model. The difference between the two bar heights is the statistical parity difference (aka. demographic parity, one of the most common quantitative fairness metrics). On the bottom is the statistical parity difference among the SHAP values for each input feature to the model. The sum of the bars on the bottom equals the difference between the bars on the top. (B) The same analysis as done in (A), except that now the income of women is under-reported in the simulated data. This causes a moderately statistically significant disparity in the model’s output, and a huge disparity in the SHAP values for the reported income feature. (C) The same analysis as (B) except now the number of late payments is under-reported for women (which causes the model to underestimate their risk).”) [Examiner’s note: Fig.1 shows the statistical parity difference among SHAP values for each input feature, where the sum of the feature level bars equal to the overall difference between the model outputs for the 2 groups.]
The combination of Adebayo and Lundberg are analogous art because they are in the same field of training time series data. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, having the teachings Adebayo and Lundberg before them, to modify the teachings of Adebayo to include the teachings of Lundberg providing a set of reasons for the output difference between 2 groups in order to explain why the model produced different outcomes for the groups, improve transparency of the model, and identify the features contributing to the disparity for further analysis or remediation.
However, Jiang explicitly discloses:
the set of reasons identifying a set of features driving the bias between the first group and the second group from a feature vector of the machine learning model along with a relative contribution of each feature to the computed output difference; and (Jiang, Page 1, Col. 2, Section 2: “Each datapoint (an, xn, yn) corresponds to information from an individual (or community): yn indicates a binary class, each element
a
i
n
of an corresponds to a different sensitive attribute, e.g. to the gender of the individual, and xn is a feature vector that, possibly together with an, can be used to form a prediction
y
^
n
∈
{0, 1} of the class yn.”) [Examiner’s note: the feature vector contains multiple attributes that contribute to the model’s prediction (
y
^
n), and the highlight suggests that these features, along with an, influence the computed outcome.]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Adebayo and Jiang. Adebayo teaches a toolbox for diagnosing bias in predictive modeling to address the question “how can an analyst determine the relative significance of the inputs to a black-box predictive model in order to assess the model’s fairness”. Jiang teaches an approach to fair classification that enforces independence between the classifier outputs and sensitive information by minimizing Wasserstein distances. One of ordinary skill would have motivation to combine Adebayo and Jiang because Wasserstein distance is particular useful in detecting subtle biases in features and outcome distributions, even when they overlap. By applying Wasserstein distance, we can quantify the extent of bias in model predictions, identify features contributing to disparities.
Regarding Claim 3, the combination of Adebayo, Jiang, and Lundberg explicitly discloses all the limitations of claim 2 (as shown in the rejections above).
Adebayo in view of Jiang and Lundberg further discloses:
wherein the remediation engine selects features for adjustment based on a user input received at the computing machine. (Adebayo, Page 51, ¶[1]: “As part of the module, users can specify whether they want to use the simple linear transformation procedure or the more general non-linear basis expansion procedure. Further, users can specify the sets of functions that they wish to include for transforming the input data.”) [Examiner’s note: “the remediation engine” is being interpreted as the data transformation technique as explained above]
Regarding Claim 4, the combination of Adebayo, Jiang, and Lundberg explicitly discloses all the limitations of claim 1 (as shown in the rejections above).
Adebayo in view of Jiang and Lundberg further discloses:
wherein the set of metrics comprises a metric based on a Difference of Means or a Wasserstein Distance. (Jiang, Page 1, Col. 2, ¶[2]: “In this paper, we propose an approach that overcomes these limitations by imposing independence constraints directly on the model outputs. This is achieved through enforcing small Wasserstein distances between the distributions of the model outputs corresponding to groups of individuals with different sensitive attributes.”)
Regarding Claim 5, the combination of Adebayo, Jiang, and Lundberg explicitly discloses all the limitations of claim 1 (as shown in the rejections above).
Adebayo in view of Jiang and Lundberg further discloses:
wherein the output difference comprises a statistical difference measurement between outputs for the first group and outputs for the second group. (Adebayo, Page 83, ¶[1-3]: “We now define bias, bf as follows:
PNG
media_image2.png
56
502
media_image2.png
Greyscale
. Pr (f(g) = 1|g
∈
PC) corresponds to the probability of the bank giving an individual a loan given that they do not belong to the protected class, while Pr (f(g) = 1|g
∈
P) corresponds to the probability of the bank giving a loan to an individual given that such individual is a member of the protected class. The difference between these two probabilities is the bias of the model f. A predictive model f is said to achieve statistical parity given (G, P, D), if|bf (G, P, D)| <
ϵ
, where
ϵ
corresponds to an infinitesimal number. Statistical parity is achieved when the difference in outcome probabilities for individuals in a protected class, and individuals not in a protected class is infinitesimally small”) [Examiner’s note: “a statistical difference measurement” is being interpreted as the “the difference between the 2 outcome probabilities”]
Regarding Claim 6, the combination of Adebayo, Jiang, and Lundberg explicitly discloses all the limitations of claim 5 (as shown in the rejections above).
Adebayo in view of Jiang and Lundberg further discloses:
wherein the statistical difference measurement is one or more of: an arithmetic difference, a disparate impact ratio, a difference of means, and a Wasserstein distance. (Adebayo, Page 83, ¶[1-3]: ““We now define bias, bf as follows:
PNG
media_image2.png
56
502
media_image2.png
Greyscale
. Pr (f(g) = 1|g
∈
PC) corresponds to the probability of the bank giving an individual a loan given that they do not belong to the protected class, while Pr (f(g) = 1|g
∈
P) corresponds to the probability of the bank giving a loan to an individual given that such individual is a member of the protected class. The difference between these two probabilities is the bias of the model f. A predictive model f is said to achieve statistical parity given (G, P, D), if|bf (G, P, D)| <
ϵ
, where
ϵ
corresponds to an infinitesimal number. Statistical parity is achieved when the difference in outcome probabilities for individuals in a protected class, and individuals not in a protected class is infinitesimally small”) [Examiner’s note: “an arithmetic difference” here is being interpreted as the bias
PNG
media_image2.png
56
502
media_image2.png
Greyscale
, because it measures the distance between two outcome probabilities.]
Regarding Claim 7, the combination of Adebayo, Jiang, and Lundberg explicitly discloses all the limitations of claim 5 (as shown in the rejections above).
Adebayo in view of Jiang and Lundberg further discloses:
wherein the statistical difference measurement comprises a difference of model outputs for the first group and outputs for the second group, the model outputs being one or more of: log-odds scores, probabilities of classification, and class outputs. (Adebayo, Page 83, ¶[1-2]: ““We now define bias, bf as follows:
PNG
media_image2.png
56
502
media_image2.png
Greyscale
. Pr (f(g) = 1|g
∈
PC) corresponds to the probability of the bank giving an individual a loan given that they do not belong to the protected class, while Pr (f(g) = 1|g
∈
P) corresponds to the probability of the bank giving a loan to an individual given that such individual is a member of the protected class. The difference between these two probabilities is the bias of the model f. ) [Examiner’s note: the highlight indicates the model output being probabilities of classification e.g., probability of the bank giving an individual a loan given that they do not belong to the protected class and probability of the bank giving a loan to an individual given that such individual is a member of the protected class]
Regarding Claim 21, the combination of Adebayo, Jiang, and Lundberg explicitly discloses all the limitations of claim 1 (as shown in the rejections above).
Adebayo in view of Jiang and Lundberg further discloses:
wherein providing the output representing the set of reasons further comprises: generating a visualization displaying the set of features ordered by the relative contribution to the computed output difference. (Lundberg, Pg. 4, Figure 1:
PNG
media_image3.png
492
1203
media_image3.png
Greyscale
“A simulation study on the effect of sex-specific reporting biases during credit scoring, and how SHAP values can be used to explain the resulting statistical parity differences. (A) On top is the mean credit worthiness of both men and women as predicted by a machine learning model. The difference between the two bar heights is the statistical parity difference (aka. demographic parity, one of the most common quantitative fairness metrics). On the bottom is the statistical parity difference among the SHAP values for each input feature to the model. The sum of the bars on the bottom equals the difference between the bars on the top. (B) The same analysis as done in (A), except that now the income of women is under-reported in the simulated data. This causes a moderately statistically significant disparity in the model’s output, and a huge disparity in the SHAP values for the reported income feature. (C) The same analysis as (B) except now the number of late payments is under-reported for women (which causes the model to underestimate their risk).”) [Examiner’s note: Fig.1 shows the statistical parity difference among SHAP values for each input feature, where the sum of the feature level bars equal to the overall difference between the model outputs for the 2 groups.]
Claim(s) 8, 11-15, 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Julius Adebayo (“Fair ML: ToolBox for Diagnosing Bias in Predictive Modeling”) (hereafter referred to as “Adebayo”) in view of Jiang et al (“Wasserstein Fair Classification”) (hereafter referred to as “Jiang”), Scott Lundberg and in further view of Bhide et al (US 2020/0184350 A1) (hereafter referred to as “Bhide”)
Regarding claim 8, Adebayo explicitly discloses:
receiving, at a computing machine, a representation of a machine learning model, a representation of a first data segment associated with a first group, and a representation of a second data segment associated with a second group that is a complement of the first group; (Adebayo, Pg. 32, ¶[1]: “For example, suppose we have a predictive model that is making loan decisions for different groups of people, if the probability of the model approving a loan for a female is less than 80 percent of the probability of the model approving a loan for a male, then, given an 80 percent threshold, this model exhibits disparate impact.”, Pg. 70, Section 6.2.1, Adebayo discloses a dataset with multiple input data attributes (Table 6.1) including income, customer age, asset, marital status, gender, employment status and education level that the bank uses in its models to calculate the credit limit and future probability of default on a loan
PNG
media_image1.png
355
322
media_image1.png
Greyscale
. Here, gender illustrates the first group of male and the second group is female, that is the complement of the first group, as claimed in the limitation above.)
compute an output difference between an output of the machine learning model applied to the first data segment and an output of the machine learning model applied to the second data segment; (Adebayo, Page 83, ¶[1]: “We now define bias, bf as follows:
PNG
media_image2.png
56
502
media_image2.png
Greyscale
. Pr (f(g) = 1|g
∈
PC) corresponds to the probability of the bank giving an individual a loan given that they do not belong to the protected class, while Pr (f(g) = 1|g
∈
P) corresponds to the probability of the bank giving a loan to an individual given that such individual is a member of the protected class. The difference between these two probabilities is the bias of the model f. A predictive model f is said to achieve statistical parity given (G, P, D), if|bf (G, P, D)| <
ϵ
, where
ϵ
corresponds to an infinitesimal number. Statistical parity is achieved when the difference in outcome probabilities for individuals in a protected class, and individuals not in a protected class is infinitesimally small.”) [Examiner’s note: “computing an output difference between an output” is being interpreted as computing the bias of model f, which is also computing the difference between the probability of the bank giving an individual a loan given that they do not belong to the protected class (representation of first data segment) and the probability of the bank giving a loan to an individual given that such individual is a member of the protected class (representation of second data segment.)]
generating, using the computing machine, a remediation engine for the machine learning model to decrease a magnitude of the output difference between the first group and the second group, the remediation engine comprising a feature engineering sub-engine that adjusts features from the set of features based on the determined set of reasons; and retraining the model using the remediation engine, the model providing outputs that eliminate bias between the first group and the second group. (Adebayo, Page 38, ¶[1-2]: “More recently, a new class of studies has emerged. These studies seek to identify and correct bias throughout the predictive modeling process. In general these works can be broadly classified into three categories: data transformation, algorithm manipulation, and outcome manipulation methodologies. For data transformation techniques, the input data to the predictive model is transformed as a means of quantifying bias in the data. In [13], Kamiran and Calders present a method for transforming data labels to remove discrimination. With their proposed method, a Naive-Bayes classifier is trained on positive labels, then a set of highly ranked negatively-labeled items from the protected set are changed to achieve statistical parity of outcomes. The modified data generated is then used to learn a 'fairer ' model. Here, fairness refers to a more equitable distribution of positive outcomes among different classes.”) [Examiner’s note “a remediation engine” is being interpreted as a framework or mechanism that detects biases in model predictions and applies corrective measures to improves fairness, accuracy; “a feature engineering sub-engine” is being interpreted as the “data transformation technique” ]
Adebayo fails to disclose:
A non-transitory machine-readable medium storing instructions which, when executed by processing circuitry, cause the processing circuitry to
determining, using the computing machine, a set of reasons for the computed output difference between the first group and the second group based on a set of metrics defining distance between feature importance distributions,
the set of reasons identifying a set of features driving the bias between the first group and the second group from a feature vector of the machine learning model along with a relative contribution of each feature to the computed output difference; and
providing an output representing the set of reasons for the output difference between the first group and the second group.
However, Lundberg explicitly discloses:
determining, using the computing machine, a set of reasons for the computed output difference between the first group and the second group based on a set of metrics defining distance between feature importance distributions, (Lundberg, Pg. 4, Figure 1:
PNG
media_image3.png
492
1203
media_image3.png
Greyscale
“A simulation study on the effect of sex-specific reporting biases during credit scoring, and how SHAP values can be used to explain the resulting statistical parity differences. (A) On top is the mean credit worthiness of both men and women as predicted by a machine learning model. The difference between the two bar heights is the statistical parity difference (aka. demographic parity, one of the most common quantitative fairness metrics). On the bottom is the statistical parity difference among the SHAP values for each input feature to the model. The sum of the bars on the bottom equals the difference between the bars on the top. (B) The same analysis as done in (A), except that now the income of women is under-reported in the simulated data. This causes a moderately statistically significant disparity in the model’s output, and a huge disparity in the SHAP values for the reported income feature. (C) The same analysis as (B) except now the number of late payments is under-reported for women (which causes the model to underestimate their risk).”) [Examiner’s note: Fig.1 shows the statistical parity difference among SHAP values for each input feature, where the sum of the feature level bars equal to the overall difference between the model outputs for the 2 groups.]
providing an output representing the set of reasons for the output difference between the first group and the second group. . (Lundberg, Pg. 4, Figure 1:
PNG
media_image3.png
492
1203
media_image3.png
Greyscale
“A simulation study on the effect of sex-specific reporting biases during credit scoring, and how SHAP values can be used to explain the resulting statistical parity differences. (A) On top is the mean credit worthiness of both men and women as predicted by a machine learning model. The difference between the two bar heights is the statistical parity difference (aka. demographic parity, one of the most common quantitative fairness metrics). On the bottom is the statistical parity difference among the SHAP values for each input feature to the model. The sum of the bars on the bottom equals the difference between the bars on the top. (B) The same analysis as done in (A), except that now the income of women is under-reported in the simulated data. This causes a moderately statistically significant disparity in the model’s output, and a huge disparity in the SHAP values for the reported income feature. (C) The same analysis as (B) except now the number of late payments is under-reported for women (which causes the model to underestimate their risk).”) [Examiner’s note: Fig.1 shows the statistical parity difference among SHAP values for each input feature, where the sum of the feature level bars equal to the overall difference between the model outputs for the 2 groups.]
The combination of Adebayo and Lundberg are analogous art because they are in the same field of training time series data. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, having the teachings Adebayo and Lundberg before them, to modify the teachings of Adebayo to include the teachings of Lundberg providing a set of reasons for the output difference between 2 groups in order to explain why the model produced different outcomes for the groups, improve transparency of the model, and identify the features contributing to the disparity for further analysis or remediation.
However, Bhide explicitly discloses:
A non-transitory machine-readable medium storing instructions which, when executed by processing circuitry, cause the processing circuitry to: (Bhide, ¶[0103]: “The computer program product may include a computer readable storage medium ( or media) having computer readable program instructions thereon for causing a processor to carry, out aspects of the present invention.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Adebayo and Bhide. Adebayo teaches a toolbox for diagnosing bias in predictive modeling to address the question “how can an analyst determine the relative significance of the inputs to a black-box predictive model in order to assess the model’s fairness”. Bhide teaches an individual bias detector used to prioritize data samples in a bias mitigation algorithm aiming to improve the group fairness measure of disparate impact. One of ordinary skill would have motivation to combine Adebayo and Bhide because MPEP 2143 sets forth the Supreme Court rationales for obviousness including: (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results; (E): “Obvious to try” choosing from a finite number of identified, predictable solutions, with a reasonable expectation of success; (F) Known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of the ordinary skill in the art.
However, Jiang explicitly discloses:
the set of reasons identifying a set of features driving the bias between the first group and the second group from a feature vector of the machine learning model along with a relative contribution of each feature to the computed output difference; and (Jiang, Page 1, Col. 2, Section 2: “Each datapoint (an, xn, yn) corresponds to information from an individual (or community): yn indicates a binary class, each element
a
i
n
of an corresponds to a different sensitive attribute, e.g. to the gender of the individual, and xn is a feature vector that, possibly together with an, can be used to form a prediction
y
^
n
∈
{0, 1} of the class yn.”) [Examiner’s note: the feature vector contains multiple attributes that contribute to the model’s prediction (
y
^
n), and the highlight suggests that these features, along with an, influence the computed outcome.]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Adebayo and Jiang. Adebayo teaches a toolbox for diagnosing bias in predictive modeling to address the question “how can an analyst determine the relative significance of the inputs to a black-box predictive model in order to assess the model’s fairness”. Jiang teaches an approach to fair classification that enforces independence between the classifier outputs and sensitive information by minimizing Wasserstein distances. One of ordinary skill would have motivation to combine Adebayo and Jiang because Wasserstein distance is particular useful in detecting subtle biases in features and outcome distributions, even when they overlap. By applying Wasserstein distance, we can quantify the extent of bias in model predictions, identify features contributing to disparities.
Regarding Claim 11, the combination of Adebayo, Bhide, Lundberg and Jiang explicitly discloses all the limitations of claim 8 (as shown in the rejections above).
Adebayo in view of Jiang, Lundberg and Bhide further discloses:
wherein the set of metrics comprises a metric based on a Difference of Means or a Wasserstein Distance. (Jiang, Page 1, Col. 2, ¶[2]: “In this paper, we propose an approach that overcomes these limitations by imposing independence constraints directly on the model outputs. This is achieved through enforcing small Wasserstein distances between the distributions of the model outputs corresponding to groups of individuals with different sensitive attributes.”)
Regarding Claim 12, the combination of Adebayo, Lundberg, Bhide and Jiang explicitly discloses all the limitations of claim 8 (as shown in the rejections above).
Adebayo in view of Jiang, Lundberg and Bhide further discloses:
wherein the output difference comprises a statistical difference measurement between outputs for the first group and outputs for the second group. (Adebayo, Page 83, ¶[1-3]: “We now define bias, bf as follows:
PNG
media_image2.png
56
502
media_image2.png
Greyscale
. Pr (f(g) = 1|g
∈
PC) corresponds to the probability of the bank giving an individual a loan given that they do not belong to the protected class, while Pr (f(g) = 1|g
∈
P) corresponds to the probability of the bank giving a loan to an individual given that such individual is a member of the protected class. The difference between these two probabilities is the bias of the model f. A predictive model f is said to achieve statistical parity given (G, P, D), if|bf (G, P, D)| <
ϵ
, where
ϵ
corresponds to an infinitesimal number. Statistical parity is achieved when the difference in outcome probabilities for individuals in a protected class, and individuals not in a protected class is infinitesimally small”) [Examiner’s note: “a statistical difference measurement” is being interpreted as the “the difference between the 2 outcome probabilities”]
Regarding Claim 13, the combination of Adebayo, Bhide, Lundberg and Jiang explicitly discloses all the limitations of claim 12 (as shown in the rejections above).
Adebayo in view of Jiang, Lundberg and Bhide further discloses:
wherein the statistical difference measurement is one or more of: an arithmetic difference, a disparate impact ratio, a difference of means, and a Wasserstein distance. (Adebayo, Page 83, ¶[1-3]: “We now define bias, bf as follows:
PNG
media_image2.png
56
502
media_image2.png
Greyscale
. Pr (f(g) = 1|g
∈
PC) corresponds to the probability of the bank giving an individual a loan given that they do not belong to the protected class, while Pr (f(g) = 1|g
∈
P) corresponds to the probability of the bank giving a loan to an individual given that such individual is a member of the protected class. The difference between these two probabilities is the bias of the model f. A predictive model f is said to achieve statistical parity given (G, P, D), if|bf (G, P, D)| <
ϵ
, where
ϵ
corresponds to an infinitesimal number. Statistical parity is achieved when the difference in outcome probabilities for individuals in a protected class, and individuals not in a protected class is infinitesimally small”) [Examiner’s note: “an arithmetic difference” here is being interpreted as the bias
PNG
media_image2.png
56
502
media_image2.png
Greyscale
, because it measures the distance between two outcome probabilities.]
Regarding Claim 14, the combination of Adebayo, Bhide, Lundberg and Jiang explicitly discloses all the limitations of claim 12 (as shown in the rejections above).
Adebayo in view of Jiang, Lundberg and Bhide further discloses:
wherein the statistical difference measurement comprises a difference of model outputs for the first group and outputs for the second group, the model outputs being one or more of: log-odds scores, probabilities of classification, and class outputs. (Adebayo, Page 83, ¶[1-2]: ““We now define bias, bf as follows:
PNG
media_image2.png
56
502
media_image2.png
Greyscale
. Pr (f(g) = 1|g
∈
PC) corresponds to the probability of the bank giving an individual a loan given that they do not belong to the protected class, while Pr (f(g) = 1|g
∈
P) corresponds to the probability of the bank giving a loan to an individual given that such individual is a member of the protected class. The difference between these two probabilities is the bias of the model f. ) [Examiner’s note: the highlight indicates the model output being probabilities of classification e.g., probability of the bank giving an individual a loan given that they do not belong to the protected class and probability of the bank giving a loan to an individual given that such individual is a member of the protected class]
Regarding claim 15, Adebayo explicitly discloses:
receive a representation of a machine learning model, a representation of a machine learning model, a representation of a first data segment associated with a first group, and a representation of a second data segment associated with a second group that is a complement of the first group; (Adebayo, Pg. 32, ¶[1]: “For example, suppose we have a predictive model that is making loan decisions for different groups of people, if the probability of the model approving a loan for a female is less than 80 percent of the probability of the model approving a loan for a male, then, given an 80 percent threshold, this model exhibits disparate impact.”, Pg. 70, Section 6.2.1, Adebayo discloses a dataset with multiple input data attributes (Table 6.1) including income, customer age, asset, marital status, gender, employment status and education level that the bank uses in its models to calculate the credit limit and future probability of default on a loan
PNG
media_image1.png
355
322
media_image1.png
Greyscale
. Here, gender illustrates the first group of male and the second group is female, that is the complement of the first group, as claimed in the limitation above.)
compute an output difference between an output of the machine learning model applied to the first data segment and an output of the machine learning model applied to the second data segment; (Adebayo, Page 83, ¶[1]: “We now define bias, bf as follows:
PNG
media_image2.png
56
502
media_image2.png
Greyscale
. Pr (f(g) = 1|g
∈
PC) corresponds to the probability of the bank giving an individual a loan given that they do not belong to the protected class, while Pr (f(g) = 1|g
∈
P) corresponds to the probability of the bank giving a loan to an individual given that such individual is a member of the protected class. The difference between these two probabilities is the bias of the model f. A predictive model f is said to achieve statistical parity given (G, P, D), if|bf (G, P, D)| <
ϵ
, where
ϵ
corresponds to an infinitesimal number. Statistical parity is achieved when the difference in outcome probabilities for individuals in a protected class, and individuals not in a protected class is infinitesimally small.”) [Examiner’s note: “computing an output difference between an output” is being interpreted as computing the bias of model f, which is also computing the difference between the probability of the bank giving an individual a loan given that they do not belong to the protected class (representation of first data segment) and the probability of the bank giving a loan to an individual given that such individual is a member of the protected class (representation of second data segment.)]
generating, using the computing machine, a remediation engine for the machine learning model to decrease a magnitude of the output difference between the first group and the second group, the remediation engine comprising a feature engineering sub-engine that adjusts features from the set of features based on the determined set of reasons; and retraining the model using the remediation engine, the model providing outputs that eliminate bias between the first group and the second group. (Adebayo, Page 38, ¶[1-2]: “More recently, a new class of studies has emerged. These studies seek to identify and correct bias throughout the predictive modeling process. In general these works can be broadly classified into three categories: data transformation, algorithm manipulation, and outcome manipulation methodologies. For data transformation techniques, the input data to the predictive model is transformed as a means of quantifying bias in the data. In [13], Kamiran and Calders present a method for transforming data labels to remove discrimination. With their proposed method, a Naive-Bayes classifier is trained on positive labels, then a set of highly ranked negatively-labeled items from the protected set are changed to achieve statistical parity of outcomes. The modified data generated is then used to learn a 'fairer ' model. Here, fairness refers to a more equitable distribution of positive outcomes among different classes.”) [Examiner’s note “a remediation engine” is being interpreted as a framework or mechanism that detects biases in model predictions and applies corrective measures to improves fairness, accuracy; “a feature engineering sub-engine” is being interpreted as the “data transformation technique” ]
Adebayo fails to disclose:
processing circuitry; and a memory storing instructions which, when executed by the processing circuitry, cause the processing circuitry to:
determining, using the computing machine, a set of reasons for the computed output difference between the first group and the second group based on a set of metrics defining distance between feature importance distributions,
the set of reasons identifying a set of features driving the bias between the first group and the second group from a feature vector of the machine learning model along with a relative contribution of each feature to the computed output difference; and
providing an output representing the set of reasons for the output difference between the first group and the second group.
However, Lundberg explicitly discloses:
determining, using the computing machine, a set of reasons for the computed output difference between the first group and the second group based on a set of metrics defining distance between feature importance distributions, (Lundberg, Pg. 4, Figure 1:
PNG
media_image3.png
492
1203
media_image3.png
Greyscale
“A simulation study on the effect of sex-specific reporting biases during credit scoring, and how SHAP values can be used to explain the resulting statistical parity differences. (A) On top is the mean credit worthiness of both men and women as predicted by a machine learning model. The difference between the two bar heights is the statistical parity difference (aka. demographic parity, one of the most common quantitative fairness metrics). On the bottom is the statistical parity difference among the SHAP values for each input feature to the model. The sum of the bars on the bottom equals the difference between the bars on the top. (B) The same analysis as done in (A), except that now the income of women is under-reported in the simulated data. This causes a moderately statistically significant disparity in the model’s output, and a huge disparity in the SHAP values for the reported income feature. (C) The same analysis as (B) except now the number of late payments is under-reported for women (which causes the model to underestimate their risk).”) [Examiner’s note: Fig.1 shows the statistical parity difference among SHAP values for each input feature, where the sum of the feature level bars equal to the overall difference between the model outputs for the 2 groups.]
providing an output representing the set of reasons for the output difference between the first group and the second group. (Lundberg, Pg. 4, Figure 1:
PNG
media_image3.png
492
1203
media_image3.png
Greyscale
“A simulation study on the effect of sex-specific reporting biases during credit scoring, and how SHAP values can be used to explain the resulting statistical parity differences. (A) On top is the mean credit worthiness of both men and women as predicted by a machine learning model. The difference between the two bar heights is the statistical parity difference (aka. demographic parity, one of the most common quantitative fairness metrics). On the bottom is the statistical parity difference among the SHAP values for each input feature to the model. The sum of the bars on the bottom equals the difference between the bars on the top. (B) The same analysis as done in (A), except that now the income of women is under-reported in the simulated data. This causes a moderately statistically significant disparity in the model’s output, and a huge disparity in the SHAP values for the reported income feature. (C) The same analysis as (B) except now the number of late payments is under-reported for women (which causes the model to underestimate their risk).”) [Examiner’s note: Fig.1 shows the statistical parity difference among SHAP values for each input feature, where the sum of the feature level bars equal to the overall difference between the model outputs for the 2 groups.]
The combination of Adebayo and Lundberg are analogous art because they are in the same field of training time series data. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, having the teachings Adebayo and Lundberg before them, to modify the teachings of Adebayo to include the teachings of Lundberg providing a set of reasons for the output difference between 2 groups in order to explain why the model produced different outcomes for the groups, improve transparency of the model, and identify the features contributing to the disparity for further analysis or remediation.
However, Bhide explicitly discloses:
processing circuitry; and a memory storing instructions which, when executed by the processing circuitry, cause the processing circuitry to: (Bhide, ¶[0103]: “The computer program product may include a computer readable storage medium ( or media) having computer readable program instructions thereon for causing a processor to carry, out aspects of the present invention.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Adebayo and Bhide. Adebayo teaches a toolbox for diagnosing bias in predictive modeling to address the question “how can an analyst determine the relative significance of the inputs to a black-box predictive model in order to assess the model’s fairness”. Bhide teaches an individual bias detector used to prioritize data samples in a bias mitigation algorithm aiming to improve the group fairness measure of disparate impact. One of ordinary skill would have motivation to combine Adebayo and Bhide because MPEP 2143 sets forth the Supreme Court rationales for obviousness including: (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results; (E): “Obvious to try” choosing from a finite number of identified, predictable solutions, with a reasonable expectation of success; (F) Known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of the ordinary skill in the art.
However, Jiang explicitly discloses:
the set of reasons identifying a set of features driving the bias between the first group and the second group from a feature vector of the machine learning model along with a relative contribution of each feature to the computed output difference; and (Jiang, Page 1, Col. 2, Section 2: “Each datapoint (an, xn, yn) corresponds to information from an individual (or community): yn indicates a binary class, each element
a
i
n
of an corresponds to a different sensitive attribute, e.g. to the gender of the individual, and xn is a feature vector that, possibly together with an, can be used to form a prediction
y
^
n
∈
{0, 1} of the class yn.”) [Examiner’s note: the feature vector contains multiple attributes that contribute to the model’s prediction (
y
^
n), and the highlight suggests that these features, along with an, influence the computed outcome.]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Adebayo and Jiang. Adebayo teaches a toolbox for diagnosing bias in predictive modeling to address the question “how can an analyst determine the relative significance of the inputs to a black-box predictive model in order to assess the model’s fairness”. Jiang teaches an approach to fair classification that enforces independence between the classifier outputs and sensitive information by minimizing Wasserstein distances. One of ordinary skill would have motivation to combine Adebayo and Jiang because Wasserstein distance is particular useful in detecting subtle biases in features and outcome distributions, even when they overlap. By applying Wasserstein distance, we can quantify the extent of bias in model predictions, identify features contributing to disparities.
Regarding Claim 18, the combination of Adebayo, Lundberg, Bhide and Jiang explicitly discloses all the limitations of claim 15 (as shown in the rejections above).
Adebayo in view of Jiang, Lundberg and Bhide further discloses:
wherein the set of metrics comprises a metric based on a Difference of Means or a Wasserstein Distance. (Jiang, Page 1, Col. 2, ¶[2]: “In this paper, we propose an approach that overcomes these limitations by imposing independence constraints directly on the model outputs. This is achieved through enforcing small Wasserstein distances between the distributions of the model outputs corresponding to groups of individuals with different sensitive attributes.”)
Regarding Claim 19, the combination of Adebayo, Bhide, Lundberg and Jiang explicitly discloses all the limitations of claim 15 (as shown in the rejections above).
Adebayo in view of Jiang, Lundberg and Bhide further discloses:
wherein the output difference comprises a statistical difference measurement between outputs for the first group and outputs for the second group. (Adebayo, Page 83, ¶[1-3]: “We now define bias, bf as follows:
PNG
media_image2.png
56
502
media_image2.png
Greyscale
. Pr (f(g) = 1|g
∈
PC) corresponds to the probability of the bank giving an individual a loan given that they do not belong to the protected class, while Pr (f(g) = 1|g
∈
P) corresponds to the probability of the bank giving a loan to an individual given that such individual is a member of the protected class. The difference between these two probabilities is the bias of the model f. A predictive model f is said to achieve statistical parity given (G, P, D), if|bf (G, P, D)| <
ϵ
, where
ϵ
corresponds to an infinitesimal number. Statistical parity is achieved when the difference in outcome probabilities for individuals in a protected class, and individuals not in a protected class is infinitesimally small”) [Examiner’s note: “a statistical difference measurement” is being interpreted as the “the difference between the 2 outcome probabilities”]
Claim(s) 22, 24 is rejected under 35 U.S.C. 103 as being unpatentable over Julius Adebayo (“Fair ML: ToolBox for Diagnosing Bias in Predictive Modeling”) (hereafter referred to as “Adebayo”) in view of Jiang et al (“Wasserstein Fair Classification”) (hereafter referred to as “Jiang”), Scott Lundberg and in further view of Lundberg et al. (“Consistent Individualized Feature Attribution for Tree Ensembles”) (hereafter referred to as “Lundberg2019”)
Regarding Claim 22, the combination of Adebayo, Jiang, and Lundberg explicitly discloses all the limitations of claim 1 (as shown in the rejections above).
Adebayo in view of Jiang and Lundberg further discloses:
wherein determining the set of reasons further comprises: computing a contribution to the output difference for each feature in the feature vector based on a difference between a feature importance distribution for the first group and a feature importance distribution for the second group; and (Lundberg, Pg. 2, Col. 1, Section Applying Explainable AI Methods to Quantitative Fairness Metrics, ¶[1]: “These methods explain the output of a model as a sum of effects attributable to each input feature. If this decomposition is exact then for a model f applied to a set of inputs x you can represent the model output as a sum of feature impacts
PNG
media_image4.png
61
237
media_image4.png
Greyscale
where Фi (f, x) is the impact feature xi has on the model’s output. Note that Ф0(f) is a constant bias term that does not depend on the current input x.”, Pg. 2, Col. 2, ¶[1-2]: “Since quantitative fairness metrics are computed by measuring expected differences in model outputs between specific groups of samples, we can also apply these metrics to the "partial model outputs" represented by the attribution values Фi (f, x).”, Pg. 3, Col. 1, ¶[4]: “When we introduce an under-reporting bias for women’s income into the simulation we see a moderately significant statistical parity difference appear in the model’s output (Figure 1B top). If this were a real application, this statistical parity difference might trigger an in-depth analysis of the model to determine what might be causing the disparity. While this investigation is challenging given just a single statistical parity difference value, it is much easier given the per-feature statistical parity decomposition based on SHAP (Figure 1B bottom), where there is a clearly significant bias coming from the reported income feature.”)
Adebayo in view of Jiang and Lundberg fails to discloses:
ordering the features according to contributions to the output difference.
However, Lundberg2019 explicitly discloses:
ordering the features according to contributions to the output difference. (Lundberg2019, Pg. 7, Col. 2, Section 5.5.1: “SHAP summary plots leverage individualized feature attributions to convey all these aspects of a feature’s importance while remaining visually concise (Figure 8). Features are first sorted by their global impact
PNG
media_image5.png
24
75
media_image5.png
Greyscale
, then dots representing the SHAP values
Ф
i
(
j
)
are plotted horizontally, stacking vertically when they run out of space.”, Pg. 8, Figure 8:
PNG
media_image6.png
435
458
media_image6.png
Greyscale
, Pg. 8, Col. 1, ¶[1]: “In Figure 8 we see (unsurprisingly) that age at baseline is the most important risk factor for death over the next 20 years. The density of the age plot shows how common different ages are in the dataset, and the coloring shows a smooth increase in the model’s output (a log odds ratio) as age increases.”)
The combination of Adebayo and Lundberg2019 are analogous art because they are in the same field of training time series data. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, having the teachings Adebayo and Lundberg2019 before them, to modify the teachings of Adebayo to include the teachings of Lundberg2019 ordering the features according to contributions to the output difference because ranking the feature allows the most influential causes of the disparity to be identified more efficiently. Presenting the features in order of contribution improves interpretability of the model, helps the user focus on the features that most strongly explain the difference between the 2 groups, and supports further analysis or remediation of the detected disparity.
Regarding Claim 24, the combination of Adebayo, Jiang, and Lundberg explicitly discloses all the limitations of claim 1 (as shown in the rejections above).
Adebayo in view of Jiang and Lundberg fails to disclose:
wherein providing the output representing the set of reasons further comprises: generating an influence sensitivity plot for at least one feature identified in the set of reasons, the influence sensitivity plot providing a visual representation of how values of the at least one feature impacts the contribution to model outputs; and
presenting the influence sensitivity plot as part of the output representing the set of reasons.
However, Lundberg2019 explicitly discloses:
wherein providing the output representing the set of reasons further comprises: generating an influence sensitivity plot for at least one feature identified in the set of reasons, the influence sensitivity plot providing a visual representation of how values of the at least one feature impacts the contribution to model outputs; and (Lundberg2019, Pg. 7, Col. 2, Section 5.5.1: “SHAP Summary Plots. SHAP summary plots leverage individualized feature attributions to convey all these aspects of a feature’s importance while remaining visually concise (Figure 8)… Each dot is colored by the value of that feature, from low (blue) to high (red). If the impact of the feature on the model’s output varies smoothly as its value changes then this coloring will also have a smooth gradation.”, Pg. 8, FIG.8:
PNG
media_image6.png
435
458
media_image6.png
Greyscale
)
presenting the influence sensitivity plot as part of the output representing the set of reasons. (Lundberg2019, Pg. 8, Figure 8: “SHAP summary plot of a 14 feature XGBoost survival model on 20 year mortality followup data from NHANES I [18]. The higher the SHAP value of a feature, the higher your log odds of death in this Cox hazards model. Every individual in the dataset is run through the model and a dot is created for each feature attribution value, so one person gets one dot on each feature’s line. Dot’s are colored by the feature’s value for that person and pile up vertically to show density.”
PNG
media_image6.png
435
458
media_image6.png
Greyscale
)
Claim(s) 23, 25 is rejected under 35 U.S.C. 103 as being unpatentable over Julius Adebayo (“Fair ML: ToolBox for Diagnosing Bias in Predictive Modeling”) (hereafter referred to as “Adebayo”) in view of Jiang et al (“Wasserstein Fair Classification”) (hereafter referred to as “Jiang”), Scott Lundberg and in further view of Lundberg et al. (“Consistent Individualized Feature Attribution for Tree Ensembles”) (hereafter referred to as “Lundberg2019”) and Datta et al. (“Algorithmic Transparency via Quantitative Input Influence: Theory and Experiments with Learning Systems”)
Regarding Claim 23, the combination of Adebayo, Jiang, and Lundberg explicitly discloses all the limitations of claim 1 (as shown in the rejections above).
Adebayo in view of Jiang and Lundberg further discloses:
wherein determining the set of reasons comprises: identifying a subset of features from the feature vector having contributions to the output difference greater than a threshold; and (Lundberg, Pg. 3, Col. 1, ¶[3]: “As a baseline experiment we refrain from introducing any sex-specific reporting errors. This results in no significant statistical parity difference between the credit score of men and women (Figure 1A top). When we decompose the statistical parity difference using the SHAP feature attribution method [4] we also see no significant feature-level differences (Figure 1A bottom).”, Pg. 3, Col. 1, ¶[4]: “When we introduce an under-reporting bias for women’s income into the simulation we see a moderately significant statistical parity difference appear in the model’s output (Figure 1B top).”) [Examiner’s note: Lundberg decomposes the statistical parity between men and women into per-feature SHAP-based components, and then identifies the reported-income feature as having a clearly significant feature-level disparity, while other features do not show significant feature-level differences.]
Adebayo in view of Jiang and Lundberg fails to disclose:
associating each feature in the subset with a Quantitative Input Influence representing contribution to the computed output difference.
However, Datta explicitly discloses:
associating each feature in the subset with a Quantitative Input Influence representing contribution to the computed output difference. (Datta, Pg. 599, Col. 1, ¶[3]: “We formalize transparency reports by introducing a family of Quantitative Input Influence (QII) measures that capture the degree of influence of inputs on outputs of the system.”, Pg. 601, Col. 1, Definition 1: For a quantity of interest QA(・), and an input i, the Quantitative Input Influence of i on QA(・) is defined to be
PNG
media_image7.png
33
301
media_image7.png
Greyscale
)
The combination of Adebayo and Datta are analogous art because they are in the same field of training time series data. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, having the teachings Adebayo and Datta before them, to modify the teachings of Adebayo to include the teachings of Datta associating each feature in the subset with a Quantitative Input Influence representing contribution to the computed output difference because assigning a numerical influence value allows the system to measure and compare how strongly each feature contributes to the disparity between the 2 groups.
Regarding Claim 25, the combination of Adebayo, Jiang, and Lundberg explicitly discloses all the limitations of claim 1 (as shown in the rejections above).
Adebayo in view of Jiang and Lundberg further discloses:
identifying features with differences in feature importance distributions between the first group and the second group as reasons for the output difference. (Lundberg, Pg. 4, Fig. 1: “Figure 1: A simulation study on the effect of sex-specific reporting biases during credit scoring, and how SHAP values can be used to explain the resulting statistical parity differences. (A) On top is the mean credit worthiness of both men and women as predicted by a machine learning model. The difference between the two bar heights is the statistical parity difference (aka. demographic parity, one of the most common quantitative fairness metrics). On the bottom is the statistical parity difference among the SHAP values for each input feature to the model. The sum of the bars on the bottom equals the difference between the bars on the top. (B) The same analysis as done in (A), except that now the income of women is under-reported in the simulated data. This causes a moderately statistically significant disparity in the model’s output, and a huge disparity in the SHAP values for the reported income feature. (C) The same analysis as (B) except now the number of late payments is under-reported for women (which causes the model to underestimate their risk).”
PNG
media_image3.png
492
1203
media_image3.png
Greyscale
)
Adebayo in view of Jiang and Lundberg fails to disclose:
wherein determining the set of reasons further comprises: generating near clones of an input point from the first data segment and the second data segment by intervening on subsets of features;
computing predictions of the machine learning model on the near clones;
computing feature importance scores using a co-operative game-theoretic index that determines an average marginal contribution of each feature to all possible subsets of features; and
However, Datta explicitly discloses:
wherein determining the set of reasons further comprises: generating near clones of an input point from the first data segment and the second data segment by intervening on subsets of features; (Datta, Pg. 602, Col. 2, Section III, ¶[2]: “Indeed, changes to outcome are more likely to occur if we intervene on sets of features. In order to get a better understanding of the influence of a feature i ∈ N, we should measure its effect when coupled with interventions on other features… when intervening on S, we draw the states of i ∈ S based on the joint distribution of the states of features in S, πS(uS), as defined in Equation (1).”)
computing predictions of the machine learning model on the near clones; (Datta, Pg. 600, Col. 2, ¶[3]: “We are also given a probability distribution π on X, where π(x) is the probability of the input vector x. We can define a marginal probability of a set of inputs S in the standard way as follows:
PNG
media_image8.png
57
361
media_image8.png
Greyscale
”)
computing feature importance scores using a co-operative game-theoretic index that determines an average marginal contribution of each feature to all possible subsets of features; and (Datta, Pg. 603, Col. 1, Section A: “In this section, we discuss how measures from the theory of cooperative games define measures for aggregating marginal influence… With this intuition in mind, one can naturally study influence measures using cooperative game theory, and in particular, prevalent influence measures in cooperative games such as the Shapley value, Banzhaf index and others. These measures can be thought of as influence aggregation methods, which, given an influence measure v : 2N → R, output a vector φ ∈ Rn, whose i-th coordinate corresponds in some natural way to the aggregate influence, or aggregate causal effect, of feature i.”)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMY TRAN whose telephone number is (571)270-0693. The examiner can normally be reached Monday - Friday 7:30 am - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AMY TRAN/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights