Notice of Pre-AIA or AIA Status
This Non-Final communication is in response to application 18/536,274 filed on 12/12/2023. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
101 Subject Matter Eligibility Analysis
Step 1: Claims 1-20 are within the four statutory (a process, machine, manufacture or composition of matter.) Claims 1-8 are a process and claims 9-20 are a machine.
With respect to claim 1:
Step 2A Prong 1: The claim recites an abstract idea enumerated in the 2019 PEG.
generating risk scores for each of said candidate trained machine learning models over each of said evaluation dimensions; (This is an abstract idea of a "Mental Process." The "generating" step under its broadest reasonable interpretation, covers concepts that can be practically performed by a human using a pen and paper.)
determining correlations between said evaluation dimensions based, at least in part, on said generated risk scores; and (This is an abstract idea of a "Mental Process." The "determining" step under its broadest reasonable interpretation, covers concepts that can be practically performed in the human mind. The determination could be made manually by an individual.)
performing an optimization calculation to identify a subset of said set of candidate trained machine learning models, wherein each of the candidate trained machine learning models in the subset optimizes an overall risk measure over all of said evaluation dimensions, wherein the optimization calculation is based, at least in part, on said determined correlations. (This is an abstract idea of a "Mental Process." The "optimization calculation" step under its broadest reasonable interpretation, covers concepts that can be practically performed by a human using a pen and paper.)
Step 2A Prong 2: The judicial exception is not integrated into a practical application
Additional elements:
receiving a set of candidate trained machine learning models and a set of evaluation dimensions; (this limitation amounts to adding insignificant extra-solution activity to the judicial exception).
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly
more than the judicial exception
The additional element adds insignificant extra-solution activity to the judicial exception and cannot provide an inventive concept. Storing and retrieving information in memory is directed to a well understood routine conventional activity of data transmission (MPEP 2106.05(d)(II)(iv)).
Therefore, claim 1 is ineligible.
With respect to claim 2:
Step 2A Prong 1: claim 2, which incorporates the rejection of claim 1, does not recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
said evaluation dimensions are selected from the group consisting of: accuracy, bias, environmental impact, fairness, toxicity, privacy, legality, accountability, transparency, explainability, predictive performance, uncertainty, interpretability, robustness, training efficiency, memory efficiency, computational efficiency, and security. (this limitation amounts to adding insignificant extra-solution activity to the judicial exception).
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional element adds insignificant extra-solution activity to the judicial exception and cannot provide an inventive concept. Storing and retrieving information in memory is directed to a well understood routine conventional activity of data transmission (MPEP 2106.05(d)(II)(iv)).
Therefore, claim 2 is ineligible.
With respect to claim 3:
Step 2A Prong 1: claim 3, which incorporates the rejection of claim 1, does not recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
said candidate trained machine learning models are predictive machine learning models. (This amounts to no more than mere instructions to “apply” the exception using a generic computer component.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional element is recited in a generic level and they represent generic computer components to apply the abstract idea. Mere instructions to apply an exception cannot provide an inventive concept (MPEP 2106.05(f)).
Therefore, claim 3 is ineligible.
With respect to claim 4:
Step 2A Prong 1: claim 4, which incorporates the rejection of claim 1, recites an additional abstract idea:
said optimization calculation is a multi-objective optimization which optimizes over all of said evaluation dimensions, wherein an optimal solution represents at least one trade-off between two or more conflicting evaluation dimensions of said evaluation dimensions. (This is an abstract idea of a "Mental Process." The "optimization calculation" step under its broadest reasonable interpretation, covers concepts that can be practically performed by a human using a pen and paper.)
Step 2A Prong 2: claim 4 does not recite any additional elements and thus cannot be integrated into a practical application.
Step 2B: claim 4 does not recite an additional element.
Therefore, claim 4 is ineligible.
With respect to claim 5:
Step 2A Prong 1: claim 5, which incorporates the rejection of claim 1, does not recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
receiving user selections associated with a series of comparisons between pairs of said machine learning models in said subset, with respect to risk scores associated with two of said evaluation dimensions which are determined to be negatively correlated. (this limitation amounts to adding insignificant extra-solution activity to the judicial exception).
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional element adds insignificant extra-solution activity to the judicial exception and cannot provide an inventive concept. Storing and retrieving information in memory is directed to a well understood routine conventional activity of data transmission (MPEP 2106.05(d)(II)(iv)).
Therefore, claim 5 is ineligible.
With respect to claim 6:
Step 2A Prong 1: claim 6, which incorporates the rejection of claim 5, recites an additional abstract idea:
ranking said candidate trained machine learning models in said subset, based, at least in part, on said user selections over said series of comparisons. (This is an abstract idea of a "Mental Process." The "ranking" step under its broadest reasonable interpretation, covers concepts that can be practically performed in the human mind. The ranking could be made manually by an individual.)
Step 2A Prong 2: claim 6 does not recite any additional elements and thus cannot be integrated into a practical application.
Step 2B: claim 6 does not recite an additional element.
Therefore, claim 6 is ineligible.
With respect to claim 7:
Step 2A Prong 1: claim 7, which incorporates the rejection of claim 1, does not recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
training multiple machine learning models, to produce said set of candidate trained machine learning models. (This amounts to no more than mere instructions to “apply” the exception using a generic computer component.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional element is recited in a generic level and they represent generic computer components to apply the abstract idea. Mere instructions to apply an exception cannot provide an inventive concept (MPEP 2106.05(f)).
Therefore, claim 7 is ineligible.
With respect to claim 8:
Step 2A Prong 1: claim 8, which incorporates the rejection of claim 1, does not recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
risk scores are generated by inferencing said trained machine learning models over one or more provided datasets, to obtain prediction results. (This amounts to no more than mere instructions to “apply” the exception using a generic computer component.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional element is recited in a generic level and they represent generic computer components to apply the abstract idea. Mere instructions to apply an exception cannot provide an inventive concept (MPEP 2106.05(f)).
Therefore, claim 8 is ineligible.
With respect to claim 9:
The claim recites similar limitations as corresponding to claim 1. Therefore, the same subject matter analysis that was utilized for claim 1, as described above, is equally applicable to claim 9. Therefore, claim 9 is ineligible.
With respect to claim 10:
The claim recites similar limitations as corresponding to claim 2. Therefore, the same subject matter analysis that was utilized for claim 2, as described above, is equally applicable to claim 10. Therefore, claim 10 is ineligible.
With respect to claim 11:
The claim recites similar limitations as corresponding to claim 3. Therefore, the same subject matter analysis that was utilized for claim 3, as described above, is equally applicable to claim 11. Therefore, claim 11 is ineligible.
With respect to claim 12:
The claim recites similar limitations as corresponding to claim 4. Therefore, the same subject matter analysis that was utilized for claim 4, as described above, is equally applicable to claim 12. Therefore, claim 12 is ineligible.
With respect to claim 13:
The claim recites similar limitations as corresponding to claim 5. Therefore, the same subject matter analysis that was utilized for claim 5, as described above, is equally applicable to claim 13. Therefore, claim 13 is ineligible.
With respect to claim 14:
The claim recites similar limitations as corresponding to claim 6. Therefore, the same subject matter analysis that was utilized for claim 6, as described above, is equally applicable to claim 14. Therefore, claim 14 is ineligible.
With respect to claim 15:
The claim recites similar limitations as corresponding to claim 7. Therefore, the same subject matter analysis that was utilized for claim 7, as described above, is equally applicable to claim 15. Therefore, claim 15 is ineligible.
With respect to claim 16:
The claim recites similar limitations as corresponding to claim 8. Therefore, the same subject matter analysis that was utilized for claim 8, as described above, is equally applicable to claim 16. Therefore, claim 16 is ineligible.
With respect to claim 17:
The claim recites similar limitations as corresponding to claim 1. Therefore, the same subject matter analysis that was utilized for claim 1, as described above, is equally applicable to claim 17. Therefore, claim 17 is ineligible.
With respect to claim 18:
The claim recites similar limitations as corresponding to claim 2. Therefore, the same subject matter analysis that was utilized for claim 2, as described above, is equally applicable to claim 18. Therefore, claim 18 is ineligible.
With respect to claim 19:
The claim recites similar limitations as corresponding to claims 5 and 6. Therefore, the same subject matter analysis that was utilized for claims 5 and 6, as described above, is equally applicable to claim 19. Therefore, claim 19 is ineligible.
With respect to claim 20:
The claim recites similar limitations as corresponding to claim 7. Therefore, the same subject matter analysis that was utilized for claim 7, as described above, is equally applicable to claim 20. Therefore, claim 20 is ineligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4 7-12, 15-18 & 20 are rejected under 35 U.S.C. 103 as being unpatentable over by Saxena (US 2022/0180146 A1) in view of Fritsche (NPL: ‘Capturing Relationships in Multi-objective optimization’ (2016)).
Regarding claim 1, Saxena teaches:
A computer-implemented method comprising: ([0077] “ A system, computer program product, and method are disclosed and described herein directed toward performing multi-objective automated machine learning to optimize a plurality of objectives.”)
receiving a set of candidate trained machine learning models and a set of evaluation dimensions; ([0087] “Further, one or more data transformers and ML models are collected 512 (where the ML models and the transformers are labeled 432 and 434, respectively, in FIG. 4) and a ML pipeline search space 436 is built 514 through populating the ML pipeline search space 436 with the textual names of the ML models 432 and transformers 434 that can be used for creating the ML pipelines. “ and [0086] “In addition, in some embodiments, a set of objectives 508 to be attained, sometimes referred to as objectives 508 of interest (shown as 440 in FIG. 4), are input into the engine 504 through the GUI 506. Typically, the objectives 508 will be to either minimize or maximize the respective outcomes. In some embodiments, custom objectives such as robustness and fairness measures of a result are used. Also, in some embodiments, domain-specific custom objectives may be used.” Where objectives equate to evaluation dimensions.
generating risk scores for each of said candidate trained machine learning models over each of said evaluation dimensions; ([0090] “In some embodiments, an adaptive scheme 522 is used for dynamically, adaptively, and objectively computing engine-generated objective weights 524 based on the returned objective values (discussed further herein). In general, the adaptive scheme 522 internally determines the initial set of weights which are then dynamically updated to generate adaptive weights based on the objective values. The initial weights and the subsequent adaptively-generated weights (discussed further herein) are jointly referred to as adaptive weights 524. In some embodiments, the initial weights are determined through generating a coarse grid in an n-dimensional space. In some embodiments, the initial weights may be determined based on the best and worst case values of the “individual” objective functions f(x) that are optimized one at a time, i.e., not on a weighted sum. Accordingly, the adaptive weights 524 for each of the objectives 508 may be automatically system-generated, and any mechanism for initial weight selection that enables operation of the multi-objective joint optimization engine 504 may be used.”)
determining correlations between said evaluation dimensions based, at least in part, on said generated risk scores; and
performing an optimization calculation to identify a subset of said set of candidate trained machine learning models, wherein each of the candidate trained machine learning models in the subset optimizes an overall risk measure over all of said evaluation dimensions, wherein the optimization calculation is based, at least in part, on said determined correlations. ([0099] “The single objective joint optimization module 526 will use the optimizer 528 to determine the exact algorithms used for T.sub.1, T.sub.2, T.sub.3 and M out of the available choices. Similarly, the optimizer 528 will determine the exact hyperparameters hp.sub.1, hp.sub.2, hp.sub.3 and hp.sub.4 used for these transformers 434 and ML models 432 through hyperparameter optimization (HPO), inside the single objective joint optimization module 526” and “[0101] “an output 534 of the iterative process executed by the multi-objective joint optimization engine 504 includes the objective values 532 and a set of pipelines 536 as the Pareto-optimal solutions 602.” And [0102] “The Pareto-optimal solutions 602 are presented to the user as an output 538 that includes the Pareto-front 604 with the corresponding ML pipelines 460 such that the user may interact with the output 538 to either select the most appropriate ML pipeline 460 that optimizes the objectives, possibly reinitiate the process, or further refine the results (discussed further below).”)
Saxena does not teach:
determining correlations between said evaluation dimensions based, at least in part, on said generated risk scores; and
However, Fritsche does:
determining correlations between said evaluation dimensions based, at least in part, on said generated risk scores; and (Page 2 left column “The main contributions presented in the paper are the following: 1) We investigate the efficiency of different correlation measures for the task of capturing the relationships between the components of MOPs.” See also section A. Correlation measures and Fig. 1 both also on page 2. )
Saxena and Fritsche are considered analogous art to the claimed invention because they are in the same field of endeavor being multi-objective optimization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the multi-objective automated machine learning of Saxena with the objective correlation of Fritsche. One would want to do this to understand the relationship between multiple objectives and how they influence model selection (Fritsche introduction).
Regarding claim 2, Saxena in view of Fritsche teaches claim 1 as outlined above. Saxena further teaches:
said evaluation dimensions are selected from the group consisting of: accuracy, bias, environmental impact, fairness, toxicity, privacy, legality, accountability, transparency, explainability, predictive performance, uncertainty, interpretability, robustness, training efficiency, memory efficiency, computational efficiency, and security. ([0078] “In addition, one or more standard evaluation metrics may be used, such as, and without limitation, accuracy, precision, recall, false positive rate (FPR), Matthews correlation coefficient (MCC), and area under receiver operating characteristics curve (AUROC).” And [0086] “In some embodiments, custom objectives such as robustness and fairness measures of a result are used.” And [0070] “Other examples of objectives of interest include robustness of computing functions, computing efficiency, time to generate a prediction, and particular user- and/or domain-specific objectives.”))
Regarding claim 3, Saxena in view of Fritsche teaches claim 1 as outlined above. Saxena further teaches:
said candidate trained machine learning models are predictive machine learning models. ([0075] “The ML models may be one or more of a classifier or a regressor to generate the respective predictions”)
Regarding claim 4, Saxena in view of Fritsche teaches claim 1 as outlined above. Saxena further teaches:
said optimization calculation is a multi-objective optimization which optimizes over all of said evaluation dimensions, wherein an optimal solution represents at least one trade-off between two or more conflicting evaluation dimensions of said evaluation dimensions. ([0099] “The single objective joint optimization module 526 will use the optimizer 528 to determine the exact algorithms used for T.sub.1, T.sub.2, T.sub.3 and M out of the available choices. Similarly, the optimizer 528 will determine the exact hyperparameters hp.sub.1, hp.sub.2, hp.sub.3 and hp.sub.4 used for these transformers 434 and ML models 432 through hyperparameter optimization (HPO), inside the single objective joint optimization module 526” and “[0101] “an output 534 of the iterative process executed by the multi-objective joint optimization engine 504 includes the objective values 532 and a set of pipelines 536 as the Pareto-optimal solutions 602.” And [0102] “The Pareto-optimal solutions 602 are presented to the user as an output 538 that includes the Pareto-front 604 with the corresponding ML pipelines 460 such that the user may interact with the output 538 to either select the most appropriate ML pipeline 460 that optimizes the objectives, possibly reinitiate the process, or further refine the results (discussed further below).”)
Regarding claim 7, Saxena in view of Fritsche teaches claim 1 as outlined above. Saxena further teaches:
training multiple machine learning models, to produce said set of candidate trained machine learning models. ([0086] “In such instances, the respective ML model 432 is initially trained on some training dataset, then the objective function f(x) is evaluated”)
Regarding claim 8, Saxena in view of Fritsche teaches claim 1 as outlined above. Saxena further teaches:
said risk scores are generated by inferencing said trained machine learning models over one or more provided datasets, to obtain prediction results. ; ([0090] “In some embodiments, an adaptive scheme 522 is used for dynamically, adaptively, and objectively computing engine-generated objective weights 524 based on the returned objective values (discussed further herein). In general, the adaptive scheme 522 internally determines the initial set of weights which are then dynamically updated to generate adaptive weights based on the objective values. The initial weights and the subsequent adaptively-generated weights (discussed further herein) are jointly referred to as adaptive weights 524. In some embodiments, the initial weights are determined through generating a coarse grid in an n-dimensional space. In some embodiments, the initial weights may be determined based on the best and worst case values of the “individual” objective functions f(x) that are optimized one at a time, i.e., not on a weighted sum. Accordingly, the adaptive weights 524 for each of the objectives 508 may be automatically system-generated, and any mechanism for initial weight selection that enables operation of the multi-objective joint optimization engine 504 may be used.”)
Regarding claim 9, Saxena teaches:
A system comprising: at least one hardware processor; and a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by said at least one hardware processor to: ([0077] “A system, computer program product, and method are disclosed and described herein directed toward performing multi-objective automated machine learning to optimize a plurality of objectives.”)
receiving a set of candidate trained machine learning models and a set of evaluation dimensions; ([0087] “Further, one or more data transformers and ML models are collected 512 (where the ML models and the transformers are labeled 432 and 434, respectively, in FIG. 4) and a ML pipeline search space 436 is built 514 through populating the ML pipeline search space 436 with the textual names of the ML models 432 and transformers 434 that can be used for creating the ML pipelines. “ and [0086] “In addition, in some embodiments, a set of objectives 508 to be attained, sometimes referred to as objectives 508 of interest (shown as 440 in FIG. 4), are input into the engine 504 through the GUI 506. Typically, the objectives 508 will be to either minimize or maximize the respective outcomes. In some embodiments, custom objectives such as robustness and fairness measures of a result are used. Also, in some embodiments, domain-specific custom objectives may be used.” Where objectives equate to evaluation dimensions.
generating risk scores for each of said candidate trained machine learning models over each of said evaluation dimensions; ([0090] “In some embodiments, an adaptive scheme 522 is used for dynamically, adaptively, and objectively computing engine-generated objective weights 524 based on the returned objective values (discussed further herein). In general, the adaptive scheme 522 internally determines the initial set of weights which are then dynamically updated to generate adaptive weights based on the objective values. The initial weights and the subsequent adaptively-generated weights (discussed further herein) are jointly referred to as adaptive weights 524. In some embodiments, the initial weights are determined through generating a coarse grid in an n-dimensional space. In some embodiments, the initial weights may be determined based on the best and worst case values of the “individual” objective functions f(x) that are optimized one at a time, i.e., not on a weighted sum. Accordingly, the adaptive weights 524 for each of the objectives 508 may be automatically system-generated, and any mechanism for initial weight selection that enables operation of the multi-objective joint optimization engine 504 may be used.”)
determining correlations between said evaluation dimensions based, at least in part, on said generated risk scores; and
performing an optimization calculation to identify a subset of said set of candidate trained machine learning models, wherein each of the candidate trained machine learning models in the subset optimizes an overall risk measure over all of said evaluation dimensions, wherein the optimization calculation is based, at least in part, on said determined correlations. ([0099] “The single objective joint optimization module 526 will use the optimizer 528 to determine the exact algorithms used for T.sub.1, T.sub.2, T.sub.3 and M out of the available choices. Similarly, the optimizer 528 will determine the exact hyperparameters hp.sub.1, hp.sub.2, hp.sub.3 and hp.sub.4 used for these transformers 434 and ML models 432 through hyperparameter optimization (HPO), inside the single objective joint optimization module 526” and “[0101] “an output 534 of the iterative process executed by the multi-objective joint optimization engine 504 includes the objective values 532 and a set of pipelines 536 as the Pareto-optimal solutions 602.” And [0102] “The Pareto-optimal solutions 602 are presented to the user as an output 538 that includes the Pareto-front 604 with the corresponding ML pipelines 460 such that the user may interact with the output 538 to either select the most appropriate ML pipeline 460 that optimizes the objectives, possibly reinitiate the process, or further refine the results (discussed further below).”)
Saxena does not teach:
determining correlations between said evaluation dimensions based, at least in part, on said generated risk scores; and
However, Fritsche does:
determining correlations between said evaluation dimensions based, at least in part, on said generated risk scores; and (Page 2 left column “The main contributions presented in the paper are the following: 1) We investigate the efficiency of different correlation measures for the task of capturing the relationships between the components of MOPs.” See also section A. Correlation measures and Fig. 1 both also on page 2. )
Saxena and Fritsche are considered analogous art to the claimed invention because they are in the same field of endeavor being multi-objective optimization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the multi-objective automated machine learning of Saxena with the objective correlation of Fritsche. One would want to do this to understand the relationship between multiple objectives and how they influence model selection (Fritsche introduction).
Regarding claim 10, Saxena in view of Fritsche teaches claim 9 as outlined above. Saxena further teaches:
said evaluation dimensions are selected from the group consisting of: accuracy, bias, environmental impact, fairness, toxicity, privacy, legality, accountability, transparency, explainability, predictive performance, uncertainty, interpretability, robustness, training efficiency, memory efficiency, computational efficiency, and security. ([0078] “In addition, one or more standard evaluation metrics may be used, such as, and without limitation, accuracy, precision, recall, false positive rate (FPR), Matthews correlation coefficient (MCC), and area under receiver operating characteristics curve (AUROC).” And [0086] “In some embodiments, custom objectives such as robustness and fairness measures of a result are used.” And [0070] “Other examples of objectives of interest include robustness of computing functions, computing efficiency, time to generate a prediction, and particular user- and/or domain-specific objectives.”))
Regarding claim 11, Saxena in view of Fritsche teaches claim 9 as outlined above. Saxena further teaches:
said candidate trained machine learning models are predictive machine learning models. ([0075] “The ML models may be one or more of a classifier or a regressor to generate the respective predictions”)
Regarding claim 12, Saxena in view of Fritsche teaches claim 9 as outlined above. Saxena further teaches:
said optimization calculation is a multi-objective optimization which optimizes over all of said evaluation dimensions, wherein an optimal solution represents at least one trade-off between two or more conflicting evaluation dimensions of said evaluation dimensions. ([0099] “The single objective joint optimization module 526 will use the optimizer 528 to determine the exact algorithms used for T.sub.1, T.sub.2, T.sub.3 and M out of the available choices. Similarly, the optimizer 528 will determine the exact hyperparameters hp.sub.1, hp.sub.2, hp.sub.3 and hp.sub.4 used for these transformers 434 and ML models 432 through hyperparameter optimization (HPO), inside the single objective joint optimization module 526” and “[0101] “an output 534 of the iterative process executed by the multi-objective joint optimization engine 504 includes the objective values 532 and a set of pipelines 536 as the Pareto-optimal solutions 602.” And [0102] “The Pareto-optimal solutions 602 are presented to the user as an output 538 that includes the Pareto-front 604 with the corresponding ML pipelines 460 such that the user may interact with the output 538 to either select the most appropriate ML pipeline 460 that optimizes the objectives, possibly reinitiate the process, or further refine the results (discussed further below).”)
Regarding claim 15, Saxena in view of Fritsche teaches claim 9 as outlined above. Saxena further teaches:
training multiple machine learning models, to produce said set of candidate trained machine learning models. ([0086] “In such instances, the respective ML model 432 is initially trained on some training dataset, then the objective function f(x) is evaluated”)
Regarding claim 16, Saxena in view of Fritsche teaches claim 9 as outlined above. Saxena further teaches:
said risk scores are generated by inferencing said trained machine learning models over one or more provided datasets, to obtain prediction results. ; ([0090] “In some embodiments, an adaptive scheme 522 is used for dynamically, adaptively, and objectively computing engine-generated objective weights 524 based on the returned objective values (discussed further herein). In general, the adaptive scheme 522 internally determines the initial set of weights which are then dynamically updated to generate adaptive weights based on the objective values. The initial weights and the subsequent adaptively-generated weights (discussed further herein) are jointly referred to as adaptive weights 524. In some embodiments, the initial weights are determined through generating a coarse grid in an n-dimensional space. In some embodiments, the initial weights may be determined based on the best and worst case values of the “individual” objective functions f(x) that are optimized one at a time, i.e., not on a weighted sum. Accordingly, the adaptive weights 524 for each of the objectives 508 may be automatically system-generated, and any mechanism for initial weight selection that enables operation of the multi-objective joint optimization engine 504 may be used.”)
Regarding claim 17, Saxena teaches:
A computer program product comprising a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by at least one hardware processor to: ([0077] “A system, computer program product, and method are disclosed and described herein directed toward performing multi-objective automated machine learning to optimize a plurality of objectives.”)
receiving a set of candidate trained machine learning models and a set of evaluation dimensions; ([0087] “Further, one or more data transformers and ML models are collected 512 (where the ML models and the transformers are labeled 432 and 434, respectively, in FIG. 4) and a ML pipeline search space 436 is built 514 through populating the ML pipeline search space 436 with the textual names of the ML models 432 and transformers 434 that can be used for creating the ML pipelines. “ and [0086] “In addition, in some embodiments, a set of objectives 508 to be attained, sometimes referred to as objectives 508 of interest (shown as 440 in FIG. 4), are input into the engine 504 through the GUI 506. Typically, the objectives 508 will be to either minimize or maximize the respective outcomes. In some embodiments, custom objectives such as robustness and fairness measures of a result are used. Also, in some embodiments, domain-specific custom objectives may be used.” Where objectives equate to evaluation dimensions.
generating risk scores for each of said candidate trained machine learning models over each of said evaluation dimensions; ([0090] “In some embodiments, an adaptive scheme 522 is used for dynamically, adaptively, and objectively computing engine-generated objective weights 524 based on the returned objective values (discussed further herein). In general, the adaptive scheme 522 internally determines the initial set of weights which are then dynamically updated to generate adaptive weights based on the objective values. The initial weights and the subsequent adaptively-generated weights (discussed further herein) are jointly referred to as adaptive weights 524. In some embodiments, the initial weights are determined through generating a coarse grid in an n-dimensional space. In some embodiments, the initial weights may be determined based on the best and worst case values of the “individual” objective functions f(x) that are optimized one at a time, i.e., not on a weighted sum. Accordingly, the adaptive weights 524 for each of the objectives 508 may be automatically system-generated, and any mechanism for initial weight selection that enables operation of the multi-objective joint optimization engine 504 may be used.”)
determining correlations between said evaluation dimensions based, at least in part, on said generated risk scores; and
performing an optimization calculation to identify a subset of said set of candidate trained machine learning models, wherein each of the candidate trained machine learning models in the subset optimizes an overall risk measure over all of said evaluation dimensions, wherein the optimization calculation is based, at least in part, on said determined correlations. ([0099] “The single objective joint optimization module 526 will use the optimizer 528 to determine the exact algorithms used for T.sub.1, T.sub.2, T.sub.3 and M out of the available choices. Similarly, the optimizer 528 will determine the exact hyperparameters hp.sub.1, hp.sub.2, hp.sub.3 and hp.sub.4 used for these transformers 434 and ML models 432 through hyperparameter optimization (HPO), inside the single objective joint optimization module 526” and “[0101] “an output 534 of the iterative process executed by the multi-objective joint optimization engine 504 includes the objective values 532 and a set of pipelines 536 as the Pareto-optimal solutions 602.” And [0102] “The Pareto-optimal solutions 602 are presented to the user as an output 538 that includes the Pareto-front 604 with the corresponding ML pipelines 460 such that the user may interact with the output 538 to either select the most appropriate ML pipeline 460 that optimizes the objectives, possibly reinitiate the process, or further refine the results (discussed further below).”)
Saxena does not teach:
determining correlations between said evaluation dimensions based, at least in part, on said generated risk scores; and
However, Fritsche does:
determining correlations between said evaluation dimensions based, at least in part, on said generated risk scores; and (Page 2 left column “The main contributions presented in the paper are the following: 1) We investigate the efficiency of different correlation measures for the task of capturing the relationships between the components of MOPs.” See also section A. Correlation measures and Fig. 1 both also on page 2. )
Saxena and Fritsche are considered analogous art to the claimed invention because they are in the same field of endeavor being multi-objective optimization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the multi-objective automated machine learning of Saxena with the objective correlation of Fritsche. One would want to do this to understand the relationship between multiple objectives and how they influence model selection (Fritsche introduction).
Regarding claim 18, Saxena in view of Fritsche teaches claim 17 as outlined above. Saxena further teaches:
said evaluation dimensions are selected from the group consisting of: accuracy, bias, environmental impact, fairness, toxicity, privacy, legality, accountability, transparency, explainability, predictive performance, uncertainty, interpretability, robustness, training efficiency, memory efficiency, computational efficiency, and security. ([0078] “In addition, one or more standard evaluation metrics may be used, such as, and without limitation, accuracy, precision, recall, false positive rate (FPR), Matthews correlation coefficient (MCC), and area under receiver operating characteristics curve (AUROC).” And [0086] “In some embodiments, custom objectives such as robustness and fairness measures of a result are used.” And [0070] “Other examples of objectives of interest include robustness of computing functions, computing efficiency, time to generate a prediction, and particular user- and/or domain-specific objectives.”))
Regarding claim 20, Saxena in view of Fritsche teaches claim 17 as outlined above. Saxena further teaches:
training multiple machine learning models, to produce said set of candidate trained machine learning models. ([0086] “In such instances, the respective ML model 432 is initially trained on some training dataset, then the objective function f(x) is evaluated”)
Claims 5-6, 13-14, & 19 are rejected under 35 U.S.C. 103 as being unpatentable over by Saxena in view of Fritsche and Zhang (NPL: ‘Joint Optimization of AI Fairness and Utility: A Human-Centered Approach (Accessed through applicant’s IDS)).
Regarding claim 5, Saxena in view of Fritsche teaches claim 1 as outlined above. Neither of them teach:
receiving user selections associated with a series of comparisons between pairs of said machine learning models in said subset, with respect to risk scores associated with two of said evaluation dimensions which are determined to be negatively correlated.
However, Zhang does:
receiving user selections associated with a series of comparisons between pairs of said machine learning models in said subset, with respect to risk scores associated with two of said evaluation dimensions which are determined to be negatively correlated. (Page 3 Left column “For each pair, the PM is asked which one is better or more important, and rate the level of importance on a one to nine ratio scale. These ratings are then organized into a matrix, and the maximum eigenvalue of the matrix is used as weightings for the criteria or scores for the options. These eigenvalues in effect combines the result of direct pairwise comparisons and the indirect comparisons that they imply. This process also yields a consistency measure which can be used to check whether the PM provided incompatible pairwise ratio scores. As a result, AHP provides more consistent and reliable weights through one unified process” where PM is a policy maker [user]).
Saxena, Fritsche and Zhang are considered analogous art to the claimed invention because they are in the same field of endeavor being multi-objective optimization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the multi-objective automated machine learning of Saxena with the objective correlation of Fritsche and with the user comparison selection of Zhang. One would want to do this to provide more consistent and reliable weights (Zhang page 3 left column)
Regarding claim 6, Saxena in view of Fritsche and Zhang teaches claim 5 as outlined above. Zhang further teaches:
ranking said candidate trained machine learning models in said subset, based, at least in part, on said user selections over said series of comparisons. (End of page 4 start of page 5 “Unlike traditional usage of AHP that requires the PM to rank both the criteria and the alternative options on each criterion, we only ask the PM to rank the criteria. This is because: a) we have infinite amount of options—infinite possible threshold settings for the privileged and unprivileged groups; and b) we do not need the PM to rank each option on each criterion since they can be ranked directly by our independence, separation, and utility metrics. Therefore, as long as we have the PM’s weight for each metric, we can combine the three metrics into one single metric and use automatic optimization to find the best model threshold setting.”)
Regarding claim 13, Saxena in view of Fritsche teaches claim 9 as outlined above. Neither of them teach:
receiving user selections associated with a series of comparisons between pairs of said machine learning models in said subset, with respect to risk scores associated with two of said evaluation dimensions which are determined to be negatively correlated.
However, Zhang does:
receiving user selections associated with a series of comparisons between pairs of said machine learning models in said subset, with respect to risk scores associated with two of said evaluation dimensions which are determined to be negatively correlated. (Page 3 Left column “For each pair, the PM is asked which one is better or more important, and rate the level of importance on a one to nine ratio scale. These ratings are then organized into a matrix, and the maximum eigenvalue of the matrix is used as weightings for the criteria or scores for the options. These eigenvalues in effect combines the result of direct pairwise comparisons and the indirect comparisons that they imply. This process also yields a consistency measure which can be used to check whether the PM provided incompatible pairwise ratio scores. As a result, AHP provides more consistent and reliable weights through one unified process” where PM is a policy maker [user]).
Saxena, Fritsche and Zhang are considered analogous art to the claimed invention because they are in the same field of endeavor being multi-objective optimization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the multi-objective automated machine learning of Saxena with the objective correlation of Fritsche and with the user comparison selection of Zhang. One would want to do this to provide more consistent and reliable weights (Zhang page 3 left column)
Regarding claim 14, Saxena in view of Fritsche and Zhang teaches claim 13 as outlined above. Zhang further teaches:
ranking said candidate trained machine learning models in said subset, based, at least in part, on said user selections over said series of comparisons. (End of page 4 start of page 5 “Unlike traditional usage of AHP that requires the PM to rank both the criteria and the alternative options on each criterion, we only ask the PM to rank the criteria. This is because: a) we have infinite amount of options—infinite possible threshold settings for the privileged and unprivileged groups; and b) we do not need the PM to rank each option on each criterion since they can be ranked directly by our independence, separation, and utility metrics. Therefore, as long as we have the PM’s weight for each metric, we can combine the three metrics into one single metric and use automatic optimization to find the best model threshold setting.”)
Regarding claim 19, Saxena in view of Fritsche teaches claim 17 as outlined above. Neither of them teach:
receive user selections associated with a series of comparisons between pairs of said machine learning models in said subset, with respect to risk scores associated with two of said evaluation dimensions which are determined to be negatively correlated, and to rank said candidate trained machine learning models in said subset, based, at least in part, on said user selections over said series of comparisons.
However, Zhang does:
receive user selections associated with a series of comparisons between pairs of said machine learning models in said subset, with respect to risk scores associated with two of said evaluation dimensions which are determined to be negatively correlated, and to rank said candidate trained machine learning models in said subset, based, at least in part, on said user selections over said series of comparisons. (Page 3 Left column “For each pair, the PM is asked which one is better or more important, and rate the level of importance on a one to nine ratio scale. These ratings are then organized into a matrix, and the maximum eigenvalue of the matrix is used as weightings for the criteria or scores for the options. These eigenvalues in effect combines the result of direct pairwise comparisons and the indirect comparisons that they imply. This process also yields a consistency measure which can be used to check whether the PM provided incompatible pairwise ratio scores. As a result, AHP provides more consistent and reliable weights through one unified process” where PM is a policy maker [user]. Also End of page 4 start of page 5 “Unlike traditional usage of AHP that requires the PM to rank both the criteria and the alternative options on each criterion, we only ask the PM to rank the criteria. This is because: a) we have infinite amount of options—infinite possible threshold settings for the privileged and unprivileged groups; and b) we do not need the PM to rank each option on each criterion since they can be ranked directly by our independence, separation, and utility metrics. Therefore, as long as we have the PM’s weight for each metric, we can combine the three metrics into one single metric and use automatic optimization to find the best model threshold setting.”).
Saxena, Fritsche and Zhang are considered analogous art to the claimed invention because they are in the same field of endeavor being multi-objective optimization. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the multi-objective automated machine learning of Saxena with the objective correlation of Fritsche and with the user comparison selection of Zhang. One would want to do this to provide more consistent and reliable weights (Zhang page 3 left column)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL P GRUSZKA whose telephone number is (571)272-5259. The examiner can normally be reached M-F 9:00 AM - 6:00 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached at (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL GRUSZKA/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights