DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in response to submission of application on 07/25/2024.
Claims 1-20 are presented for examination.
Priority
Applicant’s claim for the benefit of a prior-filed application 63/579,450 filed on 08/29/2023 is acknowledged and admitted. Receipt is acknowledged of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record in the file.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 10/21/2024 and 09/17/2026 is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
The Drawings filed on 7/25/2024 are acceptable for examination purposes.
Specification
The Specification filed on 07/25/2024 is acceptable for examination purposes.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis of the claims will follow the 2019 Revised Patent Subject Matter Eligibility Guidelines (“2019 PEG”).
Step 1: Is the claim directed at one of the four statutory categories?
Claims 1-8 are directed to A computer-implemented method, which falls within the statutory category of a process.
Claims 8-14 are directed to A computer program product comprising a non-transient machine-readable medium, which falls within the statutory category of an article of manufacture.
Claims 15-20 are directed to A system, which falls within the statutory category of a machine.
Therefore, claims 1-20 are directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter. (MPEP § 2106.03)
Independent Claims
Step 2A Prong One: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, independent claims 1, 8, & 15 recites an abstract idea in the form of mental processes. A mental process is a process that “can be performed in the human mind, or by a human using a pen and paper” (MPEP § 2106.04(a)(2)(III), paragraph 1). Examples of mental processes include “observations, evaluations, judgments, and opinions” (MPEP § 2106.04(a)(2)(III), paragraph 2).
The following limitations of claims 1, 8, & 15 are mental processes:
processing, by at least one processor, a training data set, wherein the training data set comprises a plurality of training examples, and wherein each of the plurality of training examples is associated with multiple class labels and a group membership label; [Processing training data as claimed, reviewing data and associating it with specific labels is a mental process involving evaluation, observations and judgement.]
wherein random weights are assigned to multiple performance objectives for training the set of candidate classifiers, and wherein the multiple performance objectives correspond to the multiple class labels; [Assigning random weights to performance objectives for training is a mental process that could be practically performed by a human mind using a pen and paper.]
assessing, by the at least one processor, each classifier of the set of candidate classifiers to generate multiple performance measurements for each of the set of candidate classifiers, wherein each of the multiple performance measurements is associated with each of the multiple performance objectives, respectively; [Assessing is observation and judgement, generating performance measurements based on these assessments would further involve judgement and opinions, therefore this limitation is also considered a mental process.]
generating, by the at least one processor, a tradeoff table presenting performances of each classifier of the set of candidate classifiers, based at least in part on the multiple performance measurements; and [Generating a tradeoff table based on classifiers and measurements is a mental process that can be practically performed with aid of pen and paper, using observations, evaluations, and judgement.]
Therefore, the independent claims recite a judicial exception
Step 2A Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. The judicial exception recited in the above discussed claims is not integrated into a practical application.
training, by the at least one processor, a set of candidate classifiers [This is mere recitation that a judicial exception is to be performed using generic computer equipment running general class of computer algorithms in their ordinary capacity. (MPEP 2106.05(f))]
providing, by the at least one processor via a display, a visualization of the tradeoff table with dynamic interactive user experience to illustrate corresponding performances across different corresponding first set of performance objectives of the multiple performance objectives, wherein the dynamic interactive user experience allows a user to adjust a level of focus on a first set of the multiple performance objectives. [This is mere recitation that a judicial exception is to be performed using generic computer equipment running general class of computer algorithms in their ordinary capacity. (MPEP 2106.05(f))]
Claim 1 recites the additional limitation of A computer-implemented method for generating a classifier [This is mere recitation that a judicial exception is to be performed using generic computer equipment running general class of computer algorithms in their ordinary capacity. (MPEP 2106.05(f))]
Claim 8 recites the additional limitation of A computer program product comprising a non-transient machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations [This is mere recitation that a judicial exception is to be performed using generic computer equipment running general class of computer algorithms in their ordinary capacity. (MPEP 2106.05(f))]
Claim 15 recites the additional limitation of A system comprising: a programmable processor; and a non-transient machine-readable medium storing instructions that, when executed by the processor, cause the programmable processor to perform operations comprising [This is mere recitation that a judicial exception is to be performed using generic computer equipment running general class of computer algorithms in their ordinary capacity. (MPEP 2106.05(f))]
Therefore, under MPEP § 2106.04(d), the additional elements of the claims do not integrate the judicial exception into a practical application.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. The claims do not recite additional elements that are sufficient for the claims to amount to significantly more than the judicial exception.
Additional elements that are mere instructions to apply an exception do not constitute significantly more than a judicial exception under MPEP § 2106.05(I)(A). Since the additional elements in the independent claims are all mere instructions to apply an exception, they do not constitute significantly more than a judicial exception.
Therefore, the additional elements identified in the Step 2A Prong Two analysis do not constitute significantly more than a judicial exception.
Dependent Claims
The remaining dependent claims being rejected do not recite additional elements, whether considered individually or in combination, that are sufficient to integrate the judicial exception into a practical application or amount to significantly more than the judicial exception.
Claims 2, 9, & 16:
Step 2A, Prong 1:
wherein the randomized Multidivergence objective is a weighted linear combination of divergence associated with the multiple performance objectives. [Determining a multidivergence objective as a weighted linear combination is a process involving calculations that can reasonably be performed with the human mind with aid of pen and paper, therefore this limitation is also a mental process: observation, evaluation, judgement, something that can be done with the human mind, or with the aid of pen and paper (MPEP 2106.04(a)(2)(III))]
Step 2A, Prong 2 and Step 2B:
training the set of candidate classifiers to maximize a randomized Multi divergence objective for each choice of the random weights assigned to the multiple performance objectives. [This is mere recitation that a judicial exception is to be performed using generic computer equipment running general class of computer algorithms in their ordinary capacity. (MPEP 2106.05(f)). Additional elements that are mere instructions to apply an exception do not constitute significantly more than a judicial exception under MPEP § 2106.05(I)(A).]
Claims 3, 10, & 17:
Step 2A, Prong 1:
There are no additional abstract idea limitations.
Step 2A, Prong 2 and Step 2B:
training the set of candidate classifiers to minimize a first group score distance between mean scores across different groups [This is mere recitation that a judicial exception is to be performed using generic computer equipment running general class of computer algorithms in their ordinary capacity. (MPEP 2106.05(f)). Additional elements that are mere instructions to apply an exception do not constitute significantly more than a judicial exception under MPEP § 2106.05(I)(A).]
Claims 4, 11, & 18:
Step 2A, Prong 1:
There are no additional abstract idea limitations.
Step 2A, Prong 2 and Step 2B:
training the set of candidate classifiers to minimize a second group score distance between standard deviations across different groups. [This is mere recitation that a judicial exception is to be performed using generic computer equipment running general class of computer algorithms in their ordinary capacity. (MPEP 2106.05(f)) Additional elements that are mere instructions to apply an exception do not constitute significantly more than a judicial exception under MPEP § 2106.05(I)(A).]
Claims 5, 12, & 19:
Step 2A, Prong 1:
adjusting a first group fairness coefficient associated with the first group score distance to balance between the performance objectives and mean score distribution across different groups; and adjusting a second group fairness coefficient associated with the second group score distance to balance between the performance objectives and standard deviation distribution across different groups. [Adjusting a coefficients to balance performance objectives and mean score distributions involves experimentation, evaluation, observations and judgement, therefore this further limitation is also a mental process: observation, evaluation, judgement, something that can be done with the human mind, or with the aid of pen and paper (MPEP 2106.04(a)(2)(III))]
Step 2A, Prong 2 and Step 2B:
There are no additional elements recited, as such the claim does not provide a practical application and is not considered to be significantly more.
Claims 6, 13, & 20:
Step 2A, Prong 1:
wherein the first group fairness coefficient comprises a positive mean score coefficient and a negative mean score coefficient, wherein the positive mean score coefficient aids in reducing differences in mean scores for positive outcomes across different groups, and the negative mean score coefficient aids in reducing differences in mean scores for negative outcomes across different groups [These further limitations merely further define the mental process recited in the parent claim and are therefore considered to be part of the mental process of the parent claim.]
wherein the second group fairness coefficient comprises a positive standard deviation coefficient and a negative standard deviation coefficient, wherein the positive standard deviation coefficient aids in reducing differences in standard deviations for positive outcomes across different groups, and the negative standard deviation coefficient aids in reducing differences in standard deviations for negative outcomes across different groups. [These further limitations merely further define the mental process recited in the parent claim and are therefore considered to be part of the mental process of the parent claim.]
Step 2A, Prong 2 and Step 2B:
There are no additional elements recited, as such the claim does not provide a practical application and is not considered to be significantly more.
Claim 7 & 14:
Step 2A, Prong 1:
wherein the tradeoff table further presents a feature influence metric and a fairness metric, wherein the fairness metric includes the first group score distance between mean scores across different groups and the second group score distance between standard deviations across different groups. [These further limitations merely further define the mental process recited in the parent claim and are therefore considered to be part of the mental process of the parent claim.]
Step 2A, Prong 2 and Step 2B:
There are no additional elements recited, as such the claim does not provide a practical application and is not considered to be significantly more.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1, 2, 8, 9, 15 & 16 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over the corresponding claims of U.S. Application No. 18/784,747. Although the claims at issue are not identical, they are not patentably distinct from each other for the reasons outlined in the table below.
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
This application – 18/784,752
Co-pending application – 18/784,747
1. A computer-implemented method for generating a classifier, comprising:
processing, by at least one processor, a training data set, wherein the training data set comprises a plurality of training examples, and wherein each of the plurality of training examples is associated with multiple class labels and a group membership label;
training, by the at least one processor, a set of candidate classifiers, wherein random weights are assigned to multiple performance objectives for training the set of candidate classifiers, and wherein the multiple performance objectives correspond to the multiple class labels;
assessing, by the at least one processor, each classifier of the set of candidate classifiers to generate multiple performance measurements for each of the set of candidate classifiers, wherein each of the multiple performance measurements is associated with each of the multiple performance objectives, respectively;
generating, by the at least one processor, a tradeoff table presenting performances of each classifier of the set of candidate classifiers, based at least in part on the multiple performance measurements; and
providing, by the at least one processor via a display, a visualization of the tradeoff table with dynamic interactive user experience to illustrate corresponding performances across different corresponding first set of performance objectives of the multiple performance objectives, wherein the dynamic interactive user experience allows a user to adjust a level of focus on a first set of the multiple performance objectives.
Claim 1. A computer-implemented method for generating a classifier, comprising:
processing, by at least one processor, a training data set, wherein the training data set comprises a plurality of training examples, and wherein each of the plurality of training examples is associated with multiple class labels;
training, by the at least one processor, a set of candidate classifiers, wherein random weights are assigned to multiple training objectives for training the set of candidate classifiers, and wherein the multiple training objectives corresponds to the multiple class labels;
assessing, by the at least one processor, each classifier of the set of candidate classifiers to generate multiple performance measurements for each of the set of candidate classifiers, wherein each of the multiple performance measurements is associated with each of the multiple training objectives, respectively;
generating, by the at least one processor, a tradeoff table presenting performances of each classifier of the set of candidate classifiers, based at least in part on the multiple performance measurements; and
providing, by the at least one processor via a display, a visualization of the tradeoff table with dynamic interactive user experience to illustrate corresponding performances across different corresponding objectives of the multiple training objectives, wherein the dynamic interactive user experience allows a user to adjust a level of focus on a first set of the multiple training objectives.
Claim 7. The method of claim 1, wherein training the set of candidate classifiers further comprises incorporating a fairness objective into the multiple training objectives, wherein the fairness objective minimizes classifier outcome distribution disparities across different groups.
As illustrated in the table above, every limitation in this application has a corresponding equivalent or more specific limitation in the ‘747 application. Thus, the ‘747 application anticipates this claim.
Independent claims 8 & 15 recite an analogous computer program product and an analogous system, respectively.
The correspondence in dependent claims is illustrated in the table below.
This application – 18/784,752
Co-pending application – 18/784,747
2. The method of claim 1, wherein the training the set of candidate classifiers further comprises: training the set of candidate classifiers to maximize a randomized Multi divergence objective for each choice of the random weights assigned to the multiple performance objectives, wherein the randomized Multidivergence objective is a weighted linear combination of divergence associated with the multiple performance objectives.
3. The method of claim 1, wherein the training the set of candidate classifiers further comprises: training the set of candidate classifiers to maximize a randomized Multi divergence objective for each choice of the random weights assigned to the multiple training objectives, wherein the randomized Multidivergence objective is a weighted linear combination of divergence associated with the multiple training objectives.
As illustrated in the table above, every claim limitation in this application has a corresponding equivalent or more specific limitation in the ‘747 application. Thus, the ‘747 application anticipates this claim.
Dependent claims 9 & 16 recite an analogous computer program product and an analogous system, respectively.
The following references are relied upon for the art rejection set forth below:
Lin et al. Pareto Multi-Task Learning, December 2019, hereinafter Lin
Hickey et al. US Pub. No. US 2021/0248503 A1, August 2021, hereinafter Hickey
Wexler et al. The What-If Tool: Interactive Probing of Machine Learning Models, January 2020, hereinafter Wexler
Zhong et al. Learning Fair Classifiers via Min-Max F-divergence Regularization, June 2023, hereinafter Zhong
Chen et al. Towards Threshold invariant Fair Classification, June 2020, hereinafter Chen
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 8, & 15 are rejected under 35 U.S.C. 103 as being unpatentable over Lin in view of Hickey in further view of Wexler.
Regarding Claim 1, Lin discloses a computer-implemented method for generating a classifier, comprising: [Lin: Abstract] “Multi-task learning is a powerful method… we generalize this idea and propose a novel Pareto multi-task learning algorithm (Pareto MTL) to find a set of well-distributed Pareto solutions which can represent different trade-offs among different tasks.”
processing, […], a training data set, wherein the training data set comprises a plurality of training examples, and wherein each of the plurality of training examples is associated with multiple class labels and […]; [Lin: §6.1] “To construct the MultiMNIST dataset, we randomly pick two images with different digits from the original MNIST dataset [32], and then combine these two images into a new one…For each dataset, we have a two objective MTL problem to classify the item on the top-left (task 1) and to classify the item on the bottom-right (task 2)” Lin discloses constructing a dataset wherein each training example includes two separately classified items associated with two class labels, one for each classification task.
training, […], a set of candidate classifiers, wherein random weights are assigned to multiple performance objectives for training the set of candidate classifiers, and wherein the multiple performance objectives correspond to the multiple class labels; [Lin: Supplementary Material §B] “We also compare Pareto MIL with the linear scalarization method with random weight research. As shown in sub-figure (b), Pareto MTL's solutions dominate nearly all random search's solutions” [Lin: §3.2] “Linear scalarization… approach for solving multi-task learning problems… uses a linear weighted sum method to combine the losses of all tasks into a single surrogate loss… where
w
i
is the weight for the
i
-th task.” Lin discloses generating multiple classifier solutions using randomly selected weight vectors applied to multiple task objectives. As discussed in the previous limitation, respective task objectives correspond to the two classification tasks, and therefore to the multiple class labels associated with each training example.
assessing, […], each classifier of the set of candidate classifiers to generate multiple performance measurements for each of the set of candidate classifiers, wherein each of the multiple performance measurements is associated with each of the multiple performance objectives, respectively; [Lin: Supplementary Material §B] “Pareto MTL with 25 preference vectors can represent the different trade-offs among three tasks much better. MTL practitioners can easily select their preferred solutions from the obtained results.” [Lin: §6.1, Fig. 4] “The results for three experiments with Task1&2 accuracy: our proposed Pareto MTL can successfully find a set of well-distributed solutions with different trade-offs for all experiments” Figure 4 plots each candidate solution according to its respective Task 1 and Task 2 accuracy. Lin therefore discloses assessing each candidate solution using a respective performance measurement for each task objective.
generating, […] a tradeoff table presenting performances of each classifier of the set of candidate classifiers, based at least in part on the multiple performance measurements; and [Lin: Supplementary Material §B, Table 1] “The gender accuracy, race accuracy, and age L1-loss obtained by different algorithms. The best results are highlighted. Pareto MTL can find widely distributed solutions with diverse trade-offs.” Lin discloses generating a tradeoff table, Table 1, that presents performances or accuracies of the candidate classifiers, with multiple performance measurements or accuracies accounted for.
Lin does not expressly disclose the methods being performed by the at least one processor, however Hickey does teach this limitation. [Hickey: ¶0292] “an embodiment of a computer suitable for performing the method steps” [Hickey: ¶0293] “Computer 1400 comprises central control unit 1402 comprising CPU 1404 and memory unit 1406.” Hickey discloses performing the machine learning method operations using at least one processor. Lin further does not expressly disclose the plurality of training examples being associated with a group membership label, however Hickey does teach this additional limitation.
[Hickey: ¶0077] “The protected characteristic represents a generic property that defines a grouping of training data set 200 into a privileged group and an un-privileged group... and the presence of a characteristic is indicated through a binary indicator variable for each data point in the training data set.” Hickey discloses associating each training example with an indicator, or label, identifying its membership in a corresponding group.
Lin and Hickey are analogous art because they are from the same field of endeavor, specifically training machine learning classifiers using optimization objectives.
It would have been obvious to a person having ordinary skill in the art (PHOSITA, before the effective filing date of the claimed invention, to implement Lin’s classifier training framework on Hickey’s processor based computer system and incorporate Hickey’s protected-group information and fairness training technique to reduce group dependent bias while maintaining classifier performance.
The suggestion/motivation for doing so would have been provided by Hickey itself, which teaches that [Hickey: ¶0088] “Reducing the influence of the protected characteristic… reduces the bias of the machine learning model… Therefore, reducing the influence of the protected characteristic on model configuration 116 improves the fairness of the machine learning model.” Hickey further teaches [Hickey: ¶0101] “the training process… is adapted in order to reduce the influence of the protected characteristic within the model configuration… whilst maintaining the overall performance…. Not to such an extent as to substantially reduce the predictive power, or accuracy, of the machine learning model.”. A PHOSITA would have incorporated combined Hickey’s known fairness training technique within Lin’s known classifier training framework, combining prior-art elements according to known methods to obtain the predictable results of reduced group dependent bias while substantially maintaining classifier predictive performance. (MPEP § 2143.01(A)).
The combination of Lin and Hickey does not expressly disclose providing, by the at least one processor via a display, a visualization of the tradeoff table with dynamic interactive user experience to illustrate corresponding performances across different corresponding first set of performance objectives of the multiple performance objectives, wherein the dynamic interactive user experience allows a user to adjust a level of focus on a first set of the multiple performance objectives. However, Wexler in combination with Lin does teach this limitation. Lin teaches differing trade-off preferences among the multiple task objectives . [Lin: Supplementary Material §B] “It is possible to propose learning-based Pareto MTL for dynamically adjusting the preference vectors to incorporate the MTL practitioner's preference, and to guide the solutions search to their preferred smaller subspace for a MIL problem.” Lin teaches dynamically adjusting the relative preference or focus applied to selected portions of the multiple objective solution space. Wexler teaches a dynamic interactive visualization in which users can manipulate controls and view corresponding changes in performance information. [Wexler: §4.3] “users can interact with configuration controls … and view the resulting performance table on the right side.” Wexler further teaches that these controls and thresholds can be modified and [Wexler: §4.3.3] “the performance table and visualization instantly update to reflect resulting changes in performance.” Wexler teaches an interactive user interface that displays performance dynamically when the user makes adjustments. Applying these interactive controls and responsive visualization to Lin’s dynamically adjustable preference vectors would allow a user to interactively adjust the level of focus applied to selected performance objectives and view the corresponding tradeoff results.
Lin and Wexler are analogous art because they are from the same field of endeavor, specifically machine-learning model evaluation and selection. Wexler is also reasonably pertinent to the problem faced by Lin of comparing multiple classifier solutions having different performance tradeoffs.
It would have been obvious to a PHOSITA, before the effective filing date of the claimed invention, to present a visualization of Lin’s dynamically adjustable task preference framework using Wexler’s interactive performance visualization and controls so that a user would be able to adjust the relative focus among selected performance objectives and compare the different performance tradeoffs.
The suggestion/motivation for doing so would have been provided by Lin and Wexler themselves. Lin teaches that [Lin: Abstract] “Practitioners can easily select their preferred solution from these Pareto solutions, or use different trade-off solutions for different situations” and further teaches [Lin: §1] “practitioners can easily select their preferred solution(s) among the set of obtained Pareto optimal solutions with different trade-offs, rather than exhaustively searching for a set of proper weights for all tasks.” Wexler teaches [Wexler: § Abstract] “the What-If Tool, an open-source application that allows practitioners to probe, visualize, and analyze ML systems, with minimal coding. The What-If Tool lets practitioners test performance in hypothetical situations, analyze tile importance of different data features, and visualize model behavior across multiple models”. A PHOSITA would have applied Wexler’s interactive controls and responsive visualization to Lin’s adjustable preference framework to facilitate efficient selection and comparison of preferred task performance tradeoffs, combining prior-art elements according to known methods to obtain the predictable result allowing a user to interactively adjust the relative focus among performance objectives and visualize the corresponding classifier performance tradeoffs. (MPEP § 2143.01(A)).
Claim 8 is substantially similar in scope and spirit as claim 1, differing only in reciting A computer program product comprising a non-transient machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising: the recited operations – a computer implemented form that Hickey discloses [Hickey: ¶0014] “an additional aspect of the invention there is provided a non-transitory computer readable medium comprising one or more programs, the one or more programs comprising instructions which when executed by one or more processors of an electronic device cause the electronic device to perform the methods of any one of the claims”. Therefore, the rejections of claim 1 is applied accordingly.
Claim 15 is substantially similar in scope and spirit as claim 1, differing only in reciting A system comprising: a programmable processor; and a non-transient machine-readable medium storing instructions that, when executed by the processor, cause the programmable processor to perform operations comprising: the recited operations – a computer implemented form that Hickey discloses [Hickey: ¶0293] “Computer 1400 comprises central control unit 1402 comprising CPU 1404 and memory unit 1406. CPU 1404 is communicatively coupled to memory unit” [Hickey: ¶0014] “an additional aspect of the invention there is provided a non-transitory computer readable medium comprising one or more programs, the one or more programs comprising instructions which when executed by one or more processors of an electronic device cause the electronic device to perform the methods of any one of the claims”. Therefore, the rejections of claim 1 is applied accordingly.
Claims 2, 3, 9, 10, 16, & 17 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Lin, Hickey, and Wexler in further view of Zhong.
Regarding Claim 2, the combination of Lin, Hickey, and Wexler disclose the method according to claim 1, wherein the training the set of candidate classifiers further comprises: training the set of candidate classifiers to maximize a randomized Multidivergence objective for each choice of the random weights assigned to the multiple performance objectives, […] [Lin: Supplementary Material §B] “We also compare Pareto MIL with the linear scalarization method with random weight research. As shown in sub-figure (b), Pareto MTL's solutions dominate nearly all random search's solutions” [Lin: §3.1] “A MTL problem involves a set of m correlated tasks with a loss vector… A MTL algorithm is to optimize all tasks simultaneously” [Lin: §3.2] “Linear scalarization… approach for solving multi-task learning problems… uses a linear weighted sum method to combine the losses of all tasks into a single surrogate loss
min
θ
L
(
θ
)
=
∑
i
=
1
m
w
i
L
i
(
θ
)
where
w
i
is the weight for the
i
-th task.” Lin discloses a scalarized MTL objective for a weight vector, wherein the per-task losses are combined using the assigned weights. As discussed with respect to claim 1, the respective tasks correspond to the multiple classification objectives associated with the multiple class labels. Lin further discloses random weight search to obtain different solutions corresponding to different choices of task weights.
The combination of Lin, Hickey, and Wexler does not expressly disclose wherein the randomized Multidivergence objective is a weighted linear combination of divergence associated with the multiple performance objectives. However, Zhong does teach this limitation. [Zhong: §III] “the fairness constrained learning can be formulated as the following optimization problem (denoted by OPT):
min
w
1
N
∑
n
=
1
N
L
w
(
y
^
n
;
y
n
)
+
λ
∑
i
≠
j
D
f
w
(
π
i
|
|
π
j
)
, where
D
f
w
(
π
i
|
|
π
j
)
is the F-divergence between group
i
and group
j
for the classifier parameterized by
w
;” Zhong discloses training a classifier using an objective incorporating multiple F-divergence terms. Applying Zhong’s divergence-based objective terms within Lin’s weighted multi-objective framework would result in a weighted linear combination of divergence terms associated with the respective performance objectives.
Lin and Zhong are analogous art because they are from the same field of endeavor, specifically optimization-based training of machine learning classifiers. Zhong is reasonably pertinent to the problem of selecting objective functions suitable for gradient-based classifier training.
It would have been obvious to a PHOSITA, before the effective filing date of the claimed invention, to apply Zhong’s F-divergence regularization technique in Lin’s classifier training optimization to improve fairness while maintaining classification performance.
The suggestion/motivation for doing so would have been provided by Zhong itself, which teaches [Zhong: Abstract] “a novel min-max F-divergence regularization framework for learning fair classification models while preserving high accuracy” and further teaches that F-divergence are [Zhong: Abstract] “widely applicable in practical gradient-based training methods.” Zhong additionally discloses that it’s [Zhong: Abstract] “F-divergence based framework achieves state-of-the-art performance with respect to the trade-off between accuracy and fairness.” A PHOSITA would have applied Zhong’s known F-divergence regularization technique in Lin’s classifier training, combining prior-art elements according to known methods to obtain the predictable result of improving fairness of the trained classifiers while preserving classification accuracy (MPEP § 2143.01(A))
Regarding Claim 3, the combination of Lin, Hickey, Wexler, and Zhong disclose the method according to claim 2, wherein training the set of candidate classifiers further comprises: training the set of candidate classifiers to minimize a first group score distance between mean scores across different groups. [Hickey: ¶0019] “the estimated relationship is determined between a first group measure and a second group measure.” [Hickey: ¶0020] “optimising the second objective minimises the difference between the first group measure and the second group measure thereby to reduce the influence of the protected characteristic.” [Hickey: ¶0145] “the expectation … corresponds to the average score taken over the plurality of predicted scores 412 given the presence of the protected characteristic” [Hickey: ¶0149] “Minimisation of second objective 408, as defined in Equation (6), will minimise the difference between predictions produced by the data points within training data set…” Hickey discloses minimizing, during classifier optimization, the difference between the average predicted scores of two different groups, thereby minimizing a group score distance between mean scores across the groups. The rational to combine Hickey and Lin is the same as the parent claim 1.
Claims 9 & 10 is substantially similar in scope and spirit as claims 2 & 3, respectively, differing only in reciting A computer program product comprising a non-transient machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising: the recited operations – a computer implemented form that Hickey discloses [Hickey: ¶0014] “an additional aspect of the invention there is provided a non-transitory computer readable medium comprising one or more programs, the one or more programs comprising instructions which when executed by one or more processors of an electronic device cause the electronic device to perform the methods of any one of the claims”. Therefore, the rejections of claims 2 & 3 are applied accordingly.
Claim 16 & 17 are substantially similar in scope and spirit as claims 2 & 3, respectively, differing only in reciting A system comprising: a programmable processor; and a non-transient machine-readable medium storing instructions that, when executed by the processor, cause the programmable processor to perform operations comprising: the recited operations – a computer implemented form that Hickey discloses [Hickey: ¶0293] “Computer 1400 comprises central control unit 1402 comprising CPU 1404 and memory unit 1406. CPU 1404 is communicatively coupled to memory unit” [Hickey: ¶0014] “an additional aspect of the invention there is provided a non-transitory computer readable medium comprising one or more programs, the one or more programs comprising instructions which when executed by one or more processors of an electronic device cause the electronic device to perform the methods of any one of the claims”. Therefore, the rejections of claims 2 & 3 are applied accordingly.
Claims 4-7, 11-14, & 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Lin, Hickey, Wexler, and Zhong in further view of Chen.
Regarding Claim 4, the combination of Lin, Hickey, Wexler and Zhong disclose the method according to Claim 3, but does not expressly disclose, wherein training the set of candidate classifiers further comprises: training the set of candidate classifiers to minimize a second group score distance between standard deviations across different groups. However, Chen teaches comparing the dispersion of classifier scores across protected groups during classifier training. [Chen: §4.1, Eq. (15)] “Given two groups 𝒟0 and 𝒟1… the mean and variance are sufficient to characterize the distributions” and “estimate means and variances of the distributions raw score over 𝒟0 and 𝒟0 using maximum likelihood estimate” and determines a respective means
μ
^
i
and variance
σ
^
i
2
of the raw classifier scores for each group. [Chen: §4.1, Eq. (17)] Chen defines “the distance between two risk distributions r(𝒟0) and r(𝒟1)” and notes that “the distance is non-negative and approaches zero when the means and variances from 𝒟0 and 𝒟1 are equal.” [Chen: §4.2, Eq. (20)] Chen defines the logistic-regression loss as the classification loss plus the fairness regularizer and notes “Since (20) is differentiable, we can use gradient-based methods to minimize (20) with respect to
w
and
b
.” Chen discloses minimizing, during classifier training, a group score distribution distance that includes a comparison of the group score variances. Because Chen represents those variances as
σ
^
i
2
, with corresponding standard deviations
σ
^
i
of the group scores. Expressing the compared group dispersion in terms of the corresponding standard deviations rather than their squared values would predictably preserve the comparison of group score dispersion.
Chen and Lin are analogous art because they are from the same field of endeavor, specifically training and evaluating machine-learning classifiers.
It would have been obvious to a PHOSITA, before the effective filing date of the claimed invention, to apply Chen’s group score distribution fairness regularization in the classifier training framework of Lin to improve fairness across protected groups over changes in the classifier decision threshold.
The suggestion/motivation for doing so would have been provided by Chen itself, which teaches [Chen: Abstract] “models may… lead to unfair predictions in certain sense among the population groups of interest, where the grouping is based on such sensitive attributes as race and gender” and that [Chen: Abstract] “the “fair” model trained with these fairness definitions is threshold sensitive, i.e., the condition of fairness may no longer hold true when tuning the decision threshold.” Chen discloses it’s proposed threshold invariant fairness [Chen: Abstract] “is effective to alleviate the threshold sensitivity in machine learning models designed to achieve fairness.” A PHOSITA would have applied Chen’s fairness regularization in the known classifier training framework, combining prior-art elements according to known methods to obtain the predictable result of reducing threshold sensitivity while maintaining equitable classifier performance across protected groups. (MPEP § 2143.01(A))
Regarding Claim 5, the combination of Lin, Hickey, Wexler, Zhong, and Chen disclose the method according to claim 4, further comprising: adjusting a first group fairness coefficient associated with the first group score distance to balance between the performance objectives and mean score distribution across different groups; As discussed with respect to claim 3, [Hickey: ¶¶0019, 0020, 0145, 0149] Hickey’s fairness objective minimizes differences between the mean predicted scores of the respective groups. Hickey further teaches applying balancing terms to the performance and fairness objectives during joint optimization. [Hickey: ¶0040] “optimising the current configuration of the machine learning model comprises the steps of applying a first balancing term to the first objective, and applying a second balancing term to the second objective.” [Hickey: ¶0114] “first balancing term 420 and second balancing term 422 are configured to control the trade-off between first objective 406 and second objective 408 during joint optimisation 416. Beneficially, this allows for fine tuning and greater control between the influences of first objective 406 and second objective 408 on joint optimisation 416.” Hickey discloses adjusting a fairness coefficient associated with the mean-score group distance to balance the mean-score fairness objective against classifier performance objective.
and adjusting a second group fairness coefficient associated with the second group score distance to balance between the performance objectives and standard deviation distribution across different groups. As discussed with respect to claim 4, [Chen: §§ 4.1-4.2, Eq. 15, 17, 20] Chen’s fairness objective is directed to differences in the dispersion of classifier scores across protected groups. Chen further teaches weighting the fairness regularizer relative to the classification performance loss. [Chen: §4.2 Eq 20] “
L
L
R
w
,
b
=
1
D
∑
(
X
,
Y
)
∈
D
L
c
e
s
X
,
Y
+
η
(
E
f
D
where
L
c
e
⋅
denotes the cross-entropy loss of the risk score and the label, and
η
is a positive hyperparameter.” Chen teaches a coefficient
η
associated with the fairness regularizer for controlling its contribution relative to the classification performance objective. Lin further teaches [Lin: §3.2] “Linear scalarization… uses a linear weighted sum method to combine the losses of all tasks into a single surrogate loss… where
w
i
is the weight for the
i
-th task.” Lin teaches the use of respective coefficients for controlling the contributions of individual objectives within a combined training objective. Chen’s positive hyperparameter
η
, in view of Lin’s objective-weighting framework, teaches the claimed second group fairness coefficient associated with the second group score distance for balancing the dispersion based fairness objective against the performance objectives.
Regarding Claim 6, the combination of Lin, Hickey, Wexler, Zhong, and Chen disclose the method according to claim 5, wherein the first group fairness coefficient comprises a positive mean score coefficient and a negative mean score coefficient, wherein the positive mean score coefficient aids in reducing differences in mean scores for positive outcomes across different groups, and the negative mean score coefficient aids in reducing differences in mean scores for negative outcomes across different groups, Chen teaches separately evaluating group score distributions for positive and negative outcomes. [Chen: §4.1] “For TIEO case, 𝒟 is splited into four groups: negative and positive samples with negative protected attribute, 𝒟0,n and 𝒟0,p, respectively, and negative and positive samples with positive protected attribute, 𝒟1,n and 𝒟1,p, respectively. The regularizer for this case
E
f
,
E
O
is defined as
E
f
,
E
O
D
=
∑
y
∈
{
n
,
p
}
d
r
D
0
,
y
,
r
D
1
,
y
.
” As discussed with respect to claim 4, Chen further teaches that the respective group-score distributions are characterized by their means and variances and that the distribution distance approaches zero when the corresponding group means and variances are equal. Chen therefore teaches separately reducing differences between group mean scores for positive outcomes and for negative outcomes. Hickey further teaches separately accounting for group mean-score differences associated with positive and negative outcomes. [Hickey: ¶0165, Eq (11)] “
L
2
Y
^
,
Z
,
Y
=
∑
y
∈
{
0,1
}
(
E
[
Y
^
Y
=
y
,
Z
=
1
-
E
Y
^
Y
=
y
,
Z
=
0
2
” Hickey sums group mean-score differences separately for y=1 and y=0. Hickey further teaches separately weighting fairness terms with separate weighting units [Hickey: ¶0176] “for fine tuning and greater control between the influences of first regulariser 604 and second regulariser 606.” [Lin: §3.2] “Linear scalarization… uses a linear weighted sum method to combine the losses of all tasks into a single surrogate loss… where
w
i
is the weight for the
i
-th task.” Lin further teaches assigning respective weights to individual objectives within a combined training objective. In view of Hickey’s separate weighting of fairness terms and Lin’s respective objective weighting framework, applying respective weights to Chen’s separately evaluated positive and negative outcome mean-score components would provide respective positive and negative mean-score coefficients, comprising the first group fairness coefficient as claimed.
It would have been obvious to a PHOSITA, before the effective filing date of the claimed invention, to apply Hickey’s known technique of separately weighting fairness terms to Chen’s separately evaluated positive and negative outcome fairness components in order to provide finer control over the respective influences of those components during classifier optimization. Hickey expressly teaches that separate weighting terms provide “fine tuning and greater control” [Hickey: ¶0176] over the influence of the respective fairness objectives. A PHOSITA would have recognized that applying this known weighting technique to Chen’s positive and negative outcome fairness components would predictably permit independent control of their respective contributions during optimization. (MPEP § 2143.01(C))
wherein the second group fairness coefficient comprises a positive standard deviation coefficient and a negative standard deviation coefficient, wherein the positive standard deviation coefficient aids in reducing differences in standard deviations for positive outcomes across different groups, and the negative standard deviation coefficient aids in reducing differences in standard deviations for negative outcomes across different groups. As disclosed in the previous limitation, Chen separately evaluates the protected group score distributions for positive and negative outcomes and that as these distributions are characterized by their means
μ
^
i
and variance
σ
^
i
2
[see Chen: §4.1]. Chen therefore teaches separately reducing differences in group-score dispersion for positive outcomes and negative outcomes. Chen further separates the positive and the negative outcome distribution distance terms. [Chen: §4.1, Eq (19)] Chen defines the regularizer as “
E
f
,
E
O
D
=
∑
y
∈
{
n
,
p
}
d
r
D
0
,
y
,
r
D
1
,
y
”, a sum of respective distance terms, separately evaluating the protected group score distributions for negative and positive outcomes. As disclosed with respect to claim 4, Chen’s variances
σ
^
i
2
correspond to squared standard deviations
σ
i
, and expressing the compared dispersion in terms of the corresponding standard deviations would predictably preserve the comparison of group-score dispersion. Hickey further teaches separately weighting fairness terms with separate weighting units [Hickey: ¶0176] “for fine tuning and greater control between the influences of first regulariser 604 and second regulariser 606.” Lin further teaches, as discussed above, assigning respective weights to individual objectives within a combined training objective. In view of Hickey’s separate weighting of fairness terms and Lin’s respective objective weighting framework, applying respective weights to Chen’s separately evaluated positive and negative outcome dispersion terms would provide respective positive and negative standard-deviation coefficients, comprising the second group fairness coefficient as claimed. The rationale for combining Hickey and Chen is the same as for the previous limitation.
Regarding Claim 7, the combination of Lin, Hickey, Wexler, Zhong, and Chen disclose the method according to claim 4, wherein the tradeoff table further presents a feature influence metric and a fairness metric, [Wexler: Abstract] “The What-If Tool lets practitioners test performance in hypothetical situations, analyze tile importance of different data features, and visualize model behavior across multiple models and subsets of input data. It also lets practitioners measure systems according to multiple ML fairness metrics.” Wexler further teaches [Wexler 4.3] “tools for analyzing performance, available under a “Performance + Fairness” tab… users can interact with configuration controls on the left side panel to specify ground truth features. select features to slice by, and set cost ratios for performance measures, and view the resulting performance table”. Wexler discloses presenting feature-influence information and fairness metrics in an interactive model-evaluation interface associated with a performance table.
wherein the fairness metric includes the first group score distance between mean scores across different groups and the second group score distance between standard deviations across different groups. As discussed with respect to claims 3 and 4, Hickey teaches the first group score distance based on differences between mean classifier scores across protected groups, and Chen teaches the second group score distance based on differences in group-score dispersion, expressed in terms of respective variances/standard deviations. Wexler teaches presenting feature influence information and fairness metrics in an interactive model evaluation interface associated with a performance table. Wexler further teaches [Wexler: §3.1 N4] “Complementary views at… feature-, …and outcome- levels are particularly insightful when comparing the performance of multiple models”. Lin also teaches a tradeoff table that presents the values of respective training objectives for multiple candidate solutions [Lin: Supplementary Material §B, Table 1]. It would have been obvious to a PHOSITA, to include the feature influence and previously established group fairness measures in Lin’s tradeoff table to facilitate comparison of candidate classifiers using complementary feature- and outcome- level information, with the result of providing the claimed feature influence metric and fairness metric, including the first and second group score distances. (MPEP § 2143.041(A)).
Claims 11-14 is substantially similar in scope and spirit as claims 4-7, respectively, differing only in reciting A computer program product comprising a non-transient machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising: the recited operations – a computer implemented form that Hickey discloses [Hickey: ¶0014] “an additional aspect of the invention there is provided a non-transitory computer readable medium comprising one or more programs, the one or more programs comprising instructions which when executed by one or more processors of an electronic device cause the electronic device to perform the methods of any one of the claims”. Therefore, the rejections of claims 4-7 are applied accordingly.
Claim 18-20 are substantially similar in scope and spirit as claims 4-6, respectively, differing only in reciting A system comprising: a programmable processor; and a non-transient machine-readable medium storing instructions that, when executed by the processor, cause the programmable processor to perform operations comprising: the recited operations – a computer implemented form that Hickey discloses [Hickey: ¶0293] “Computer 1400 comprises central control unit 1402 comprising CPU 1404 and memory unit 1406. CPU 1404 is communicatively coupled to memory unit” [Hickey: ¶0014] “an additional aspect of the invention there is provided a non-transitory computer readable medium comprising one or more programs, the one or more programs comprising instructions which when executed by one or more processors of an electronic device cause the electronic device to perform the methods of any one of the claims”. Therefore, the rejections of claims 4-6 are applied accordingly.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Wei et al. US Pub No. US 2021/0374581 A1, Dec 2021
Fair classification using group membership information and classifier score transformation to reduce group based disparities
Wei et al. Optimized Score transformation for Consistent Fair Classification, Oct 2021
Published paper providing additional details of method and experiments for the fair classification approach described in Wei US 2021/0374581
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GRANT F FITCH whose telephone number is (571)270-0621. The examiner can normally be reached Monday-Thursday 7-3.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached at (571) 270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/G.F.F./Examiner, Art Unit 2124
/MIRANDA M HUANG/Supervisory Patent Examiner, Art Unit 2124
