Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
Acknowledgment is made of the Information Disclosure Statement dated 10/06/2025, and 04/16/2026 All of the cited references have been considered.
Response to Arguments
Applicant’s arguments filed 01/06/2026 on pages 8-11 of Remarks regarding the rejection under 35 U.S.C. 101 with respect to claims 1-20 have been fully considered but they are not persuasive. See updated rejection below.
Beginning on page 9 of Remarks, Applicant asserts that under 101 Step 2A Prong One the claims are not directed to an abstract idea because the human mind is not capable of training a machine learning model using high ranked features. MPEP 2106.04(a)(2)(III)(c) talks about mental processes on a generic computer. Also, see MPEP 2106.04(d) and 2106.05(f). The above mentioned sections of the MPEP set forth that a claim may recite a mental process even with the use of a generic computer. As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a machine-learning based model (e.g., by using these elements as tools).
On pages 10-12 of Remarks, Applicant asserts that under 101 Step 2A Prong Two and 2B that the claims are directed to a practical application. Applicant further submits that feedback assisted penalty-based machine learning that reduces variance in model accuracy, including higher-order feature-selection techniques which “discover combinations of features” such as “higher-order interactions between larger numbers (e.g., three or more) of features” fall within a technical field. However, Examiner respectfully disagrees. See MPEP 2106.04(d) and 2106.05(f). The above mentioned sections of the MPEP set forth that a claim may recite a mental process even with the use of a generic computer. Specifically, they amount to mere instructions to apply the exception using a machine learning model and a computer system (e.g., by using these elements as tools). Improving an abstract element is not sufficient to integrate into a practical application. See updated rejection below.
Applicant’s arguments on pages 13-15 of Remarks regarding the rejection under 35 U.S.C. 103 with respect to claims 1-20 have been fully considered but are moot. New reference Heinze‑Deml has been incorporated below to teach the newly presented limitations.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1,
Claim 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 1 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“processing, [by the computer system,] based on a variance in accuracy of the machine learning model across the different test datasets the training dataset using an optimization model to generate a second set of ranking values for the plurality of features and”
“determining higher-order interactions between features in the plurality of features, including at least a relevancy between pairs of the plurality of features and the set of labels for the plurality of data samples and a redundancy between groups of three or more of the plurality of features; and”
“applying a penalty value to the objective function in response to determining that the variance in the accuracy of the machine learning model across the different test datasets exceeds a variance threshold”
“updating, [by the computer system,] the training dataset based on the second set of ranking values for the plurality of features; and”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., processing the training dataset, determining, applying, updating). The above limitations in the context of this claim encompass, inter alia, processing the training dataset, determining higher-order interactions, applying a penalty value, generating ranking values and updating the training dataset (corresponding to mental processes which can be done mentally or by pen and paper).
“wherein the processing the training dataset to generate the second set of ranking values includes: using quantum annealing to determine a minimization of an objective function utilized in the optimization model, and wherein the minimization of the objective function corresponds to an output vector indicating the second set of ranking values for the plurality of features.”
As drafted, under their broadest reasonable interpretation, cover mathematical concepts (including mathematical relationships, mathematical formulas or equations, or mathematical calculations, e.g., quantum annealing). The above limitations in the context of this claim encompass, inter alia, quantum annealing and determining a minimization of an objective function (corresponding to mathematical concepts) as also shown by the formula in paragraph [0049] of the Specification.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“by a computer system,”
“executing, by a computer system, a machine learning model on different test datasets, wherein the machine learning model is trained based on a training dataset that includes:”
“training, by the computer system, the machine learning model based on the updated training dataset.”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a computer system and a machine learning model (e.g., by using these elements as tools).
The limitations:
“accessing, by a computer system, a training dataset that includes:”
“a plurality of data samples that include data values for a plurality of features; and”
“a set of labels corresponding to the plurality of data samples; and”
As drafted, amount to insignificant extra-solution activities, which do not integrate a
judicial exception into a practical application. For example, the additional elements of
"accessing a training dataset" amount to mere data gathering and data storage, respectively, which are insignificant extra-solution activities that do not integrate a judicial exception into a
practical application. See MPEP 2106.05(g).
Step 2B Analysis: The claim does not include additional elements that are sufficient to
amount to significantly more than the judicial exception.
The limitations:
“by a computer system,”
“executing, by a computer system, a machine learning model on different test datasets, wherein the machine learning model is trained based on a training dataset that includes:”
“training, by the computer system, the machine learning model based on the updated training dataset.”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a computer system and a machine learning model (e.g., by using these elements as tools).
As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are insignificant extra-solution activities or mere instructions to apply an exception. (i.e., the additional element describes a unit for applying the abstract ideas). Insignificant extra-solution activities and mere instructions to apply an exception cannot provide an inventive concept. Moreover, receiving, communicating, and storing data are insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP 2106.05(d)(II) ("The courts have recognized the following computer functions as well-understood, routine, and conventional functions ... iv. Storing and retrieving information in memory") (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)).
The claim is not patent eligible.
Regarding Claim 2,
Claim 2 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 2 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“based on the ranking values, selecting, from the plurality of features, a subset of features to include in a reduced feature set.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., selecting). The above limitations in the context of this claim encompass, inter alia, selecting a subset of features (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 1.
Step 2B Analysis: This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 3,
Claim 3 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 3 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“comparing a particular one of the ranking values, for the particular feature, to a particular threshold value; and”
“in response to the particular ranking value not meeting the particular threshold value, excluding the particular feature from the reduced feature set.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., comparing). The above limitations in the context of this claim encompass, inter alia, comparing ranking values and excluding the particular feature (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 1.
Step 2B Analysis: This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 4,
Claim 4 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 4 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“evaluates a measure of mutual information between groups of two or more features and the set of labels for the plurality of data samples; and”
“evaluates a measure of conditional mutual information between a first feature and the set of labels for the plurality of features provided that a group of two or more other features are selected for inclusion in the reduced feature set.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., evaluating a measure). The above limitations in the context of this claim encompass, inter alia, evaluating a measure of mutual information (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 1.
Step 2B Analysis: This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 5,
Claim 5 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 5 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the objective function utilized in the QUBO model utilizes performance feedback information corresponding to machine learning models that are trained based on candidate feature sets”
As drafted, under their broadest reasonable interpretation, cover mathematical concepts (including mathematical relationships, mathematical formulas or equations, or mathematical calculations, e.g., objective function). The above limitations in the context of this claim encompass, inter alia, quantum annealing and determining a minimization of an objective function (corresponding to mathematical concepts), as also shown by the formula in paragraph [0049] of the Specification.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“wherein the optimization model is an ensemble model, and”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using an optimization model (e.g., by using these elements as tools).
Step 2B Analysis: This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“wherein the optimization model is an ensemble model, and”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using an optimization model (e.g., by using these elements as tools).
The claim is not patent eligible.
Regarding Claim 6,
Claim 6 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 6 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the ranking values for the plurality of features are continuous values provided within a particular range, and wherein a magnitude of a given one of the ranking values indicates a relative ranking of a corresponding one of the plurality of features.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., ranking values). The above limitations in the context of this claim encompass, inter alia, ranking values (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 1.
Step 2B Analysis: This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 7,
Claim 7 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 7 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the updated training dataset that includes data values for a subset of features that are included in a reduced feature set, wherein the updated training dataset does not include second data values for one or more of the plurality of features that are not included in the reduced feature set; and”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., updating). The above limitations in the context of this claim encompass, inter alia, updating the training set (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 1.
Step 2B Analysis: This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 8,
Claim 8 recites a computer readable medium for performing steps similar of claim 1 and is rejected with the same rationale, mutatis mutandis, in view of the following additional elements, considered individually and as an ordered combination with the additional elements identified above, failing to integrate the abstract idea into a practical application or amount to significantly more than the abstract idea:
“a non-transitory, computer-readable medium having instructions stored thereon that are executable by a computer system to perform operations comprising:”
This is a recitation of generic computer components to be used in performing the abstract idea, which does not integrate the abstract idea into a practical application or amount to significantly more than the abstract idea. See MPEP 2106.05(f).
Regarding Claim 9,
Claim 9 recites a computer-readable medium for performing steps substantially similar to those of claim 2 and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 10,
Claim 10 recites a computer-readable medium for performing steps substantially similar to those of claim 3 and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 11,
Claim 11 recites a computer-readable medium for performing steps substantially similar to those of claim 5 and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 12,
Claim 12 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 12 is directed to a computer-readable medium, i.e., a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the objective function utilized in the QUBO model utilizes performance feedback information corresponding to machine learning models that are trained based on candidate feature sets.”
As drafted, under their broadest reasonable interpretation, cover mathematical concepts (including mathematical relationships, mathematical formulas or equations, or mathematical calculations, e.g., objective function). The above limitations in the context of this claim encompass, inter alia, quantum annealing and determining a minimization of an objective function (corresponding to mathematical concepts) as also shown by the formula in paragraph [0049] of the Specification.
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 8.
Step 2B Analysis: This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 13,
Claim 13 recites a computer-readable medium for performing steps substantially similar to those of claim 7 and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 14,
Claim 14 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 14 is directed to a computer-readable medium, i.e., a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the ranking values for the plurality of features are integer values provided within a particular range, and wherein a magnitude of a given one of the ranking values indicates a relative ranking of a corresponding one of the plurality of features.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., ranking values). The above limitations in the context of this claim encompass, inter alia, ranking values (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 8.
Step 2B Analysis: This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 15,
Claim 15 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 15 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“performing, [by the computer system,] a feature selection operation to select, from the plurality of features, a reduced feature set for the training dataset, wherein the feature selection operation includes:”
“processing the training dataset using a [feature-ranking-based optimization model] to generate a ranking of the plurality of features;”
“subsequent to the feature selection operation, updating, [by the computer system,] the training dataset to remove data values for a subset of the plurality of features that are not included in the reduced feature set; and”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., selecting). The above limitations in the context of this claim encompass, inter alia, selecting, processing, generating, updating and removing values (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“by the computer system,”
“[processing the training dataset using a] feature-ranking-based optimization model [to generate a ranking of the plurality of features;]”
“training, by the computer system, a machine learning model based on the updated training dataset.”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a computer system, an optimization model and a machine learning model (e.g., by using these elements as tools).
The limitations:
“accessing, [by a computer system,] a training dataset that includes a plurality of data samples, wherein a given one of the plurality of data samples includes data values for a plurality of features;”
As drafted, amount to insignificant extra-solution activities, which do not integrate a judicial exception into a practical application. For example, the additional elements of
"accessing a training dataset" amount to mere data gathering and data storage, respectively, which are insignificant extra-solution activities that do not integrate a judicial exception into a
practical application. See MPEP 2106.05(g).
Step 2B Analysis: The claim does not include additional elements that are sufficient to
amount to significantly more than the judicial exception.
The limitations:
“by the computer system,”
“[processing the training dataset using a] feature-ranking-based optimization model [to generate a ranking of the plurality of features;]”
“training, by the computer system, a machine learning model based on the updated training dataset.”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a computer system, an optimization model and a machine learning model (e.g., by using these elements as tools).
As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are insignificant extra-solution activities or mere instructions to apply an exception. (i.e., the additional element describes a computer system for applying the abstract ideas). Insignificant extra-solution activities and mere instructions to apply an exception cannot provide an inventive concept. Moreover, receiving, communicating, and storing data are insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP 2106.05(d)(II) ("The courts have recognized the following computer functions as well understood, routine, and conventional functions ... iv. Storing and retrieving information in memory") (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)).
The claim is not patent eligible.
Regarding Claim 16,
Claim 16 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 16 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the ranking of the plurality of features is indicated using an output vector that includes a plurality of ranking values indicating a relative importance of the plurality of features.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., ranking). The above limitations in the context of this claim encompass, inter alia, ranking features (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 15.
Step 2B Analysis: This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 17,
Claim 17 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 17 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: Please see the corresponding analysis of Claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“wherein the feature-ranking-based optimization model is a QUBO model”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using an optimization model (e.g., by using these elements as tools).
Step 2B Analysis: This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“wherein the feature-ranking-based optimization model is a QUBO model”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using an optimization model (e.g., by using these elements as tools).
The claim is not patent eligible.
Regarding Claim 18,
Claim 18 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 18 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the plurality of weighting values for the plurality of features are continuous values provided within a particular range.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., weighting values). The above limitations in the context of this claim encompass, inter alia, weighting values (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 15.
Step 2B Analysis: This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 19,
Claim 19 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 19 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the weighting values for the plurality of features are integer values provided within a particular range”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., weighting values). The above limitations in the context of this claim encompass, inter alia, weighting values (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 15.
Step 2B Analysis: This claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 20,
Claim 20 recites a method for performing steps substantially similar to those of claim 4 and is rejected with the same rationale, mutatis mutandis.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4, 6, 7, 8, 9, 11, 13, 14, 15, 16, 17, 18, 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Milne et al. (Optimal feature selection in credit scoring and classification using a quantum annealer); hereinafter Milne in view of Karnagel et al. (US20200327357A1); hereinafter Karnagel in view of Mandal et al. (An Approach of Feature Subset Selection Using Simulated Quantum Annealing); hereinafter Mandal in view of Vinh et al. (Can high-order dependencies improve mutual information based feature selection?); hereinafter Vinh and in view of Wang et al. (A multi-objective evolutionary algorithm for feature selection based on mutual information with a new redundancy measure); hereinafter Wang in further view of Heinze‑Deml et al. (Conditional variance penalties and domain shift robustness); hereinafter Heinze‑Deml
Claim 1 is rejected over Milne, Karnagel, Mandal, Vinh, Wang and Heinze-Deml.
Regarding claim 1, Milne teaches a method, comprising: executing, by a computer system, a machine learning model on different test datasets, wherein the machine learning model is trained based on a training dataset that includes: (Milne [page 1, Abstract, line 3]: “Quadratic optimization scales exponentially with the number of features, but a QUBO implementation on a quantum annealer has the potential to be faster than classical solvers”; and page 8, “Testing and scoring were performed using the StratifiedShuffleSplit cross-validation class from scikit-learn. Given the feature matrix, this class returns sets of row indices that can be used to divide the matrix rows into a training set and a test set. The separation is done in folds, with the number of folds set by an argument. For example, a shuffle
and split with 5 folds will take 80% of the matrix for the training set”; Note: The quantum annealer is a computer system that performs the method and each fold is a different test set.)
a plurality of data samples that include data values for a plurality of features; and
a set of labels corresponding to the plurality of data samples; and (Milne [page 6, section 5, line 1 ]: “The German Credit Data under consideration was originally published in 1994 by Hans Hofmann at the Institute for Statistics and Econometrics the University of Hamburg. It has been studied extensively (UCI MLR op. cit., Huang op. cit.). The data consists of 20 features (a plurality of features) (7 numerical, 13 categorical) and a binary classification (a set of labels corresponding to the data samples) (good credit or bad credit). There are 1000 rows (plurality of data samples), of which 700 are ‘‘good’’ and 300 are ‘‘bad’’.)
Milne does not teach processing, by the computer system, based on a variance in accuracy of the machine learning model across the different test datasets the training dataset using an optimization model to generate a second set of ranking values for the plurality of features,
However, Karnagel teaches processing, by the computer system, based on a variance in accuracy of the machine learning model across the different test datasets the training dataset using an optimization model to generate a second set of ranking values for the plurality of features, (Karnagel [0119]: “predicted optimal feature amounts by the multiple regressors are (e.g. arithmetic or geometric) averaged into a single predicted amount, and/or outlier (e.g. excessive variance) constituent predictions are discarded (i.e. ignored). In an embodiment, the average is weighted by accuracy, or a most accurate regressor is selected.”; [0117]: “there are multiple (e.g. discrepant) rankings of the features by different ranking functions. In an embodiment, each ranking contributes a same amount of landmarks. For example instead of one ranking with four landmarks, there may be twelve landmarks, with each of three rankings contributing four landmarks based on a same four distinct percentages.”; and [0118]: “a ranking identifier is also encoded into a landmark tuple. In an embodiment, each ranking has its own predictive regressor that is trained only with landmark tuples of that ranking. In an embodiment, a final feature subset is a (e.g. truncated) union of feature subsets predicted by the many regressors.”)
It would have been obvious before the effective filing date to combine the plurality of features of Milne and the feature ranking of Karnagel for efficient and effectiveness of feature selection (Karnagel, [0052]). Milne and Karnagel are analogous art because they both concern feature selection and machine learning.
Milne does not teach wherein processing the training dataset to generate the second set of ranking values includes:
using quantum annealing to determine a minimization of an objective function utilized in the optimization model, and
However, Mandal teaches wherein processing the training dataset to generate the second set of ranking values includes:
using quantum annealing to determine a minimization (Mandal [page 135, last paragraph, line 15 ]: “It is like finding the lowest energy state from a vast dimensional energy landscape. Adiabatic quantum computing holds this property and efficiently finds the ground state—desired global minima of problem.”; Note: The resulting ground state is the minimization.)
of an objective function utilized in the optimization model, and (Mandal [page 142, paragraph 2, line 2 ]: “SA starts with randomly selected feature subset of a particular dataset and iteratively minimizes the QUBO objective function until significant quality solutions are obtained.”;)
It would have been obvious before the effective filing date to combine the plurality of features of Milne and the quantum annealing of Mandal to effectively find the most optimal feature subset (Mandal, page 135, section 2). Milne and Mandal are analogous art because they both concern feature selection and machine learning.
Milne does not teach wherein the minimization of the objective function corresponds to an output value that indicates the ranking values for the plurality of features;
However, Karnagel teaches wherein the minimization of the objective function corresponds to an output vector indicating the second set of ranking values for the plurality of features; (Karnagel [0021]: “In an embodiment, a computer calculates, for each feature of a training dataset, a relevance score based on: a relevance scoring function, and statistics of values, of the feature, that occur in the training dataset. A rank based on relevance scores of the features is calculated for each feature. An optimal subset of the features, based on the ranks of the features, is predicted.”; [0045]: “Thus, blue may be somewhat predictive of positive and, more generally, color may be somewhat predictive of the classification label. Scoring function(s) may calculate a relevance score that is based on correlation of a feature to the label. In an embodiment, correlation may be calculated according to statistics such as mutual information or F score as discussed later herein. Features that somewhat correlate with classification may have a higher relevance score.”; and [0009]: “FIG. 1 is a block diagram that depicts an example computer that trains a regressor to predict how many is an optimal amount of features for training a machine learning (ML) model with a new dataset;” Note: The scoring function outputs ranking values.)
It would have been obvious before the effective filing date to combine the plurality of features of Milne and the feature ranking of Karnagel for efficient and effectiveness of feature selection (Karnagel, [0052]). Milne and Karnagel are analogous art because they both concern feature selection and machine learning.
Milne does not teach determining higher-order interactions between features in the plurality of features, including at least a relevancy between pairs of the plurality of features and the set of labels for the plurality of data samples and a redundancy between groups of three or more of the plurality of features; and
However, Vinh teaches determining higher-order interactions between features in the plurality of features, including at least a relevancy between pairs of the plurality of features and the set of labels for the plurality of data samples and (Vinh [page 2, column 2]: “To address this shortcoming, several works have considered the use of higher-dimensional MI quantities, such as the joint relevancy I(XiXj ; C), the conditional relevancy I(Xi ; C | Xj) and the conditional redundancy I(Xi ⋅, Xj | C)“; page 2, col. 1” and “We take a first step towards this direction by proposing several novel MI based feature selection approaches that take into account higher-order dependency between features, in particular three-way feature interaction (Xi ; Xj | Xk). Our extensive experimental evaluation shows that systematic inclusion of higher-dimensional MI quantities improves the feature selection performance.”;)
a redundancy between groups of three or more of the plurality of features; and (Vinh [page 4]: “This newly obtained redundancy quantity takes into account the second-order interactions between the features, i.e., the three-way feature interaction terms I(Xm⋅, Xj | Xk)”;)
It would have been obvious before the effective filing date to combine the plurality of features of Milne and the high-order dependencies of Vinh to improve mutual information based feature selection (Vinh, Abstract). Milne and Vinh are analogous art because they both concern feature selection.
Milne does not teach updating, by the computer system, the training dataset based on the second set of ranking values for the plurality of features; and
training, by the computer system, the machine learning model based on the updated training dataset.
However, Wang teaches updating, by the computer system, the training dataset based on the second set of ranking values for the plurality of features; and (Wang [page 81, paragraph 2, line 2]: In each fold, training datasets are first fed to the feature selection algorithms, generating different feature subsets for each dataset.”;)
training, by the computer system, the machine learning model based on the updated training dataset. (Wang [page 81, paragraph 2, line 2 ]: “Training datasets with the selected feature subsets are then used to train external classification algorithms”;)
It would have been obvious before the effective filing date to combine the features of Milne using the selected features to train a machine learning model of Wang to effectively generate feature subsets with a high predictive capability (Wang, page 73, Abstract, line 11). Milne and Wang are analogous art because they both concern feature selection and machine learning.
Milne does not teach applying a penalty value to the objective function in response to determining that the variance in the accuracy of the machine learning model across the different test datasets exceeds a variance threshold
However, Heinze‑Deml teaches applying a penalty value to the objective function (Heinze-Deml [page 312, 3.2 CoRe estimator]: “The CoRe estimator is defined in Lagrangian form for penalty ----λ ≥ 0 as … (8)”)
in response to determining that the variance in the accuracy of the machine learning model across the different test datasets exceeds a variance threshold (Heinze-Deml [page 317]: “One option for choosing λ- is to choose the largest penalty weight before the validation
loss increases considerably. This approach would provide the best distributional robustness guarantee that keeps the loss of predictive accuracy in the training distribution within a pre-specified bound.”; and [page 319, 5.1.1 CoRe penalty using the conditional variance of the predicted logits]: “We consider five different training sets which are created as follows. For each person in the standard CelebA training data we count the number of available images and select the 50 identities for which most images are available individually.”)
It would have been obvious before the effective filing date to combine the features of Milne with the conditional variance penalty of Heinze-Deml to improve test accuracies (Heinze-Deml, page 319, 5.1 Eyeglasses detection with small sample size). Milne and Heinze-Deml are analogous art because they both concern feature selection.
Claim 2 is rejected over Milne, Karnagel, Mandal, Vinh, Wang and Heinze-Deml with the incorporation of claim 1.
Regarding claim 2, Milne does not teach wherein the ranking values correspond to a relative ranking of the plurality of features; and wherein the updating includes:
based on the ranking values, selecting, from the plurality of features, a subset of features to include in a reduced feature set.
However, Karnagel teaches wherein the ranking values correspond to a relative ranking of the plurality of features; and wherein the updating includes:
based on the ranking values, selecting, from the plurality of features, a subset of features to include in a reduced feature set. (Karnagel [0021]: In an embodiment, a computer calculates, for each feature of a training dataset, a relevance score based on: a relevance scoring function, and statistics of values, of the feature, that occur in the training dataset. A rank based on relevance scores of the features is calculated for each feature. An optimal subset of the features, based on the ranks of the features, is predicted.”;)
It would have been obvious before the effective filing date to combine the plurality of features of Milne and the feature ranking of Karnagel for efficient and effectiveness of feature selection (Karnagel, [0052]). Milne and Karnagel are analogous art because they both concern feature selection and machine learning.
Claim 4 is rejected over Milne, Karnagel, Mandal, Vinh, Wang and Heinze-Deml with the incorporation of claim 1.
Regarding claim 4, Milne teaches wherein the optimization model is a quadratic unconstrained binary optimization (“QUBO”) model that: (Milne [page 1, Abstract, line 3]: “Quadratic optimization scales exponentially with the number of features, but a QUBO implementation on a quantum annealer has the potential to be faster than classical solvers”; Note: The quantum annealer is a computer system that performs the method.)
Milne does not teach evaluates a measure of mutual information between groups of two or more features and the set of labels for the plurality of data samples; and evaluates a measure of conditional mutual information between a first feature and the set of labels for the plurality of features provided that a group of two or more other features are selected for inclusion in the reduced feature set.
PNG
media_image1.png
56
290
media_image1.png
Greyscale
However, Wang teaches evaluates a measure of mutual information between groups of two or more features and the set of labels for the plurality of data samples; and (Wang [page 75, paragraph 1]: “The common information that two variables (features and the set of labels for the plurality of data samples) share is defined as the mutual information (MI) between two variables as follows: ”
; )
PNG
media_image2.png
28
268
media_image2.png
Greyscale
evaluates a measure of conditional mutual information between a first feature and the set of labels for the plurality of features provided that a group of two or more other features are selected for inclusion in the reduced feature set. (Wang [page 76, paragraphs 1-2]: “They pointed out that parameter b has a great influence on the result of MIFS; if b becomes relatively large, then the MIFS algorithm tends to handle redundancy at the expense of classification accuracy. To eliminate the negative influence of b, Kwak and Choi proposed a more accurate estimation of I(C;fi|fs). They redefined conditional information as follows:
;)
It would have been obvious before the effective filing date to combine the quadratic unconstrained binary optimization model of Milne with the evaluation of mutual information between features of Wang to effectively measure relevancy and redundancy of features (Wang, Abstract). Milne and Wang are analogous art because they both concern feature selection and machine learning.
Claim 6 is rejected over Milne, Karnagel, Mandal, Vinh, Wang and Heinze-Deml with the incorporation of claim 1.
Regarding claim 6, Milne does not teach wherein the ranking values for the plurality of features are continuous values provided within a particular range, and wherein a magnitude of a given one of the ranking values indicates a relative ranking of a corresponding one of the plurality of features.
However, Karnagel teaches wherein the ranking values for the plurality of features are continuous values provided within a particular range, and (Karnagel [0045]: “Scoring function(s) may calculate a relevance score that is based on correlation of a feature to the label. In an embodiment, correlation may be calculated according to statistics such as mutual information or F score as discussed later herein. Features that somewhat correlate with classification may have a higher relevance score.”; and [0047]: “scoring function(s) may be sensitive to the learned importance or coefficient of a feature.”)
wherein a magnitude of a given one of the ranking values indicates a relative ranking of a corresponding one of the plurality of features. (Karnagel [0050]: “Once all features F1-F16 have relevance scores, the features can be comparatively ranked according to relevance score. For example, features F1-F16 may be ranked by descending relevance score. For example, feature F3 may be most important (i.e. rank 1), and feature F2 may be less important (e.g. rank 3). As discussed later herein, features F1-F16 may be redundantly scored by different scoring functions to produce different rankings.”;)
It would have been obvious before the effective filing date to combine the plurality of features of Milne and the feature ranking of Karnagel for efficient and effectiveness of feature selection (Karnagel, [0052]). Milne and Karnagel are analogous art because they both concern feature selection and machine learning.
Claim 7 is rejected over Milne, Karnagel, Mandal, Vinh, Wang and Heinze-Deml with the incorporation of claim 1.
Regarding claim 7, Milne does not teach wherein the updated training dataset includes data values for a subset of features that are included in a reduced feature set, wherein the updated training dataset does not include second data values for one or more of the plurality of features that are not included in the reduced feature set;
However, Wang teaches wherein the updated training dataset includes data values for a subset of features that are included in a reduced feature set, wherein the updated training dataset does not include second data values for one or more of the plurality of features that are not included in the reduced feature set; (Wang [page 81, paragraph 2, line 2]: In each fold, training datasets are first fed to the feature selection algorithms, generating different feature subsets for each dataset. Training datasets with the selected feature subsets (updated training dataset) are then used to train external classification algorithms”;)
It would have been obvious before the effective filing date to combine the features of Milne using the selected features to train a machine learning model of Wang to effectively generate feature subsets with a high predictive capability (Wang, page 73, Abstract, line 11). Milne and Wang are analogous art because they both concern feature selection and machine learning.
Claim 8 is rejected over Milne, Karnagel, Mandal, Vinh, Wang and Heinze-Deml.
Regarding claim 8, Milne teaches a non-transitory, computer-readable medium having instructions stored thereon that are executable by a computer system to perform operations comprising: (Milne [page 1, Abstract, line 3]: “Quadratic optimization scales exponentially with the number of features, but a QUBO implementation on a quantum annealer has the potential to be faster than classical solvers”; Note: The quantum annealer is a computer system that performs the method.)
The remainder of claim 8 is claim 1 in the form of a non-transitory computer readable medium and is rejected for the same reasons as claim 1 stated above.
Dependent claim 9 is claim 2 in the form of a non-transitory computer readable medium and is rejected for the same reasons as claim 2 stated above. For the rejection of the limitations specifically pertaining to the non-transitory computer readable medium of claim 8, see the rejection of claim 8 above.
Claim 11 is rejected over Milne, Karnagel, Mandal, Vinh, Wang and Heinze-Deml with the incorporation of claim 8.
Regarding claim 11, Milne teaches wherein the optimization model is a quadratic unconstrained binary optimization (“QUBO”) model that: (Milne [page 1, Abstract, line 3]: “Quadratic optimization scales exponentially with the number of features, but a QUBO implementation on a quantum annealer has the potential to be faster than classical solvers”; Note: The quantum annealer is a computer system that performs the method.)
Milne does not teach evaluates a measure of mutual information between groups of two or more features and the set of labels for the plurality of data samples; and evaluates a measure of conditional mutual information between a first feature and the set of labels for the plurality of features provided that a group of two or more other features are selected for inclusion in the updated training set.
PNG
media_image1.png
56
290
media_image1.png
Greyscale
However, Wang teaches evaluates a measure of mutual information between groups of two or more features and the set of labels for the plurality of data samples; and (Wang [; page 75, paragraph 1]: “The common information that two variables (features and the set of labels for the plurality of data samples) share is defined as the mutual information (MI) between two variables as follows:
“;)
PNG
media_image2.png
28
268
media_image2.png
Greyscale
evaluates a measure of conditional mutual information between a first feature and the set of labels for the plurality of features provided that a group of two or more other features are selected for inclusion in the updated training set. (Wang [page 76, paragraphs 1-2]: “They pointed out that parameter b has a great influence on the result of MIFS; if b becomes relatively large, then the MIFS algorithm tends to handle redundancy at the expense of classification accuracy. To eliminate the negative influence of b, Kwak and Choi proposed a more accurate estimation of I(C;fi|fs). They redefined conditional information as follows:
;)
It would have been obvious before the effective filing date to combine the quadratic unconstrained binary optimization model of Milne with the evaluation of mutual information between features of Wang to effectively measure relevancy and redundancy of features (Wang, Abstract). Milne and Wang are analogous art because they both concern feature selection and machine learning.
Dependent claim 13 is claim 7 in the form of a non-transitory computer readable medium and is rejected for the same reasons as claim 7 stated above. For the rejection of the limitations specifically pertaining to the non-transitory computer readable medium of claim 8, see the rejection of claim 8 above.
Claim 14 is rejected over Milne, Karnagel, Mandal, Vinh, Wang and Heinze-Deml with the incorporation of claim 8.
Regarding claim 14, Milne teaches wherein the ranking values for the plurality of features are integer values provided within a particular range, and (Milne [page 12, section 8.2 ]: “Figure 7 shows the full range of α from 0 to 1. On the left-hand side, where α is close to zero, the emphasis is on feature independence. This favours small subsets, and since their regression coefficients are often not large enough to ‘‘push’’ the classifier across the cutoff point of p ≥ 0.5, the predicted class is 0. They classify almost all of the samples as ‘‘good credit’’ and achieve the zero-rule’s 70% success rate.”;)
Milne does not teach wherein a magnitude of a given one of the ranking values indicates a relative ranking of a corresponding one of the plurality of features.
However, Karnagel teaches wherein a magnitude of a given one of the ranking values indicates a relative ranking of a corresponding one of the plurality of features. (Karnagel [0050]: “Once all features F1-F16 have relevance scores, the features can be comparatively ranked according to relevance score. For example, features F1-F16 may be ranked by descending relevance score. For example, feature F3 may be most important (i.e. rank 1), and feature F2 may be less important (e.g. rank 3). As discussed later herein, features F1-F16 may be redundantly scored by different scoring functions to produce different rankings.”;)
Claim 15 is rejected over Milne, Karnagel, Mandal, Vinh, Wang and Heinze-Deml.
Regarding claim 15, Milne teaches a method, comprising:
accessing, by a computer system, a training dataset that includes a plurality of data samples, wherein a given one of the plurality of data samples includes data values for a plurality of features; (Milne [page 1, Abstract, line 3]: “Quadratic optimization scales exponentially with the number of features, but a QUBO implementation on a quantum annealer has the potential to be faster than classical solvers”; Note: The quantum annealer is a computer system that performs the method.)
performing, by the computer system, a feature selection operation to select, from the plurality of features, a reduced feature set for the training dataset, wherein the feature selection operation includes: (Milne [page 3, line 1]: “The German Credit Data under consideration was originally published in 1994 by Hans Hofmann at the Institute for Statistics and Econometrics the University of Hamburg. It has been studied extensively (UCI MLR op. cit., Huang op. cit.). The data consists of 20 features (a plurality of features) (7 numerical, 13 categorical) and a binary classification (a set of labels corresponding to the data samples) (good credit or bad credit). There are 1000 rows (plurality of data samples), of which 700 are ‘‘good’’ and 300 are ‘‘bad’’. T”; page 6, section 5, line 1; and “Assume that from the original set of n features we want to select a subset of K features to use in making a credit decision.”; page 4, section 3, Feature selection, line 1; and “In this paper, QUBO Feature Selection is applied to the original Hofmann data set”;)
Milne does not teach processing the training dataset using a feature-ranking-based optimization model to generate a ranking score for respective features of the plurality of features, wherein the ranking scores indicate relative importance of features in the plurality of features, and wherein the processing includes;
However, Karnagel teaches processing the training dataset using a feature-ranking-based optimization model to generate a ranking score for respective features of the plurality of features, (Karnagel [0021]: “In an embodiment, a computer calculates, for each feature of a training dataset, a relevance score based on: a relevance scoring function, and statistics of values, of the feature, that occur in the training dataset. A rank based on relevance scores of the features is calculated for each feature. An optimal subset of the features, based on the ranks of the features, is predicted.”;)
wherein the ranking scores indicate relative importance of features in the plurality of features, and wherein the processing includes; (Karnagel: [0050]: “Once all features F1-F16 have relevance scores, the features can be comparatively ranked according to relevance score. For example, features F1-F16 may be ranked by descending relevance score. For example, feature F3 may be most important (i.e. rank 1), and feature F2 may be less important (e.g. rank 3). As discussed later herein, features F1-F16 may be redundantly scored by different scoring functions to produce different rankings.”;)
It would have been obvious before the effective filing date to combine the plurality of features of Milne and the feature ranking of Karnagel for efficient and effectiveness of feature selection (Karnagel, [0052]). Milne and Karnagel are analogous art because they both concern feature selection and machine learning.
Milne does not teach using quantum annealing to determine a minimization of an objective function utilized in the feature-ranking-based optimization model, and
However, Mandal teaches using quantum annealing to determine a minimization of an objective function utilized in the feature-ranking-based optimization model, and (Mandal [page 135, last paragraph, line 15]: “It is like finding the lowest energy state from a vast dimensional energy landscape. Adiabatic quantum computing holds this property and efficiently finds the ground state—desired global minima of problem.”; Note: The resulting ground state is the minimization.)
It would have been obvious before the effective filing date to combine the plurality of features of Milne and the quantum annealing of Mandal to effectively find the most optimal feature subset (Mandal, page 135, section 2). Milne and Mandal are analogous art because they both concern feature selection and machine learning.
Milne does not teach wherein the minimization of the objective function corresponds to an output value that indicates the ranking scores for the plurality of features; and
However, Karnagel teaches wherein the minimization of the objective function corresponds to an output value that indicates the ranking scores for the plurality of features; and (Karnagel [0021]: “In an embodiment, a computer calculates, for each feature of a training dataset, a relevance score based on: a relevance scoring function, and statistics of values, of the feature, that occur in the training dataset. A rank based on relevance scores of the features is calculated for each feature. An optimal subset of the features, based on the ranks of the features, is predicted.”; and [0045]: “Thus, blue may be somewhat predictive of positive and, more generally, color may be somewhat predictive of the classification label. Scoring function(s) may calculate a relevance score that is based on correlation of a feature to the label. In an embodiment, correlation may be calculated according to statistics such as mutual information or F score as discussed later herein. Features that somewhat correlate with classification may have a higher relevance score.”; Note: The scoring function outputs ranking values.)
It would have been obvious before the effective filing date to combine the plurality of features of Milne and the feature ranking of Karnagel for efficient and effectiveness of feature selection (Karnagel, [0052]). Milne and Karnagel are analogous art because they both concern feature selection and machine learning.
Milne does not teach determining higher-order interactions between the plurality of features, including at least a relevancy between pairs of the plurality of features and the set of labels for the plurality of data samples and a redundancy between groups of three or more of the plurality of features; and
However, Vinh teaches determining higher-order interactions between the plurality of features, including at least a relevancy between pairs of the plurality of features and the set of labels for the plurality of data samples and (Vinh [page 2, column 2 ]: “To address this shortcoming, several works have considered the use of higher-dimensional MI quantities, such as the joint relevancy I(XiXj ; C), the conditional relevancy I(Xi ; C | Xj) and the conditional redundancy I(Xi ⋅, Xj | C)“; page 2, col. 1” and “We take a first step towards this direction by proposing several novel MI based feature selection approaches that take into account higher-order dependency between features, in particular three-way feature interaction (Xi ; Xj | Xk). Our extensive experimental evaluation shows that systematic inclusion of higher-dimensional MI quantities improves the feature selection performance.”;)
a redundancy between groups of three or more of the plurality of features; and (Vinh [page 4]: “This newly obtained redundancy quantity takes into account the second-order interactions between the features, i.e., the three-way feature interaction terms I(Xm⋅, Xj | Xk)”;)
It would have been obvious before the effective filing date to combine the plurality of features of Milne and the high-order dependencies of Vinh to improve mutual information based feature selection (Vinh, Abstract). Milne and Vinh are analogous art because they both concern feature selection.
Milne does not teach subsequent to the feature selection operation, updating, by the computer system, the training dataset to remove data values for a subset of the plurality of features that are not included in the reduced feature set; and
training, by the computer system, a machine learning model based on the updated training dataset.
However, Wang teaches subsequent to the feature selection operation, updating, by the computer system, the training dataset to remove data values for a subset of the plurality of features that are not included in the reduced feature set; and (Wang [page 81, paragraph 2, line 2]: “In each fold, training datasets are first fed to the feature selection algorithms, generating different feature subsets for each dataset.”;)
training, by the computer system, a machine learning model based on the updated training dataset. (Wang [page 81, paragraph 2, line 2 ]: “Training datasets with the selected feature subsets are then used to train external classification algorithms”;)
It would have been obvious before the effective filing date to combine the features of Milne using the selected features to train a machine learning model of Wang to effectively generate feature subsets with a high predictive capability (Wang, page 73, Abstract, line 11). Milne and Wang are analogous art because they both concern feature selection and machine learning.
Claim 16 is rejected over Milne, Karnagel, Mandal, Vinh, Wang and Heinze-Deml with the incorporation of claim 15.
Regarding claim 16, Milne does not teach wherein the ranking scores of the plurality of features are indicated using an output vector that includes a plurality of weighting values indicating the relative importance of the plurality of features.
However, Karnagel teaches wherein the ranking scores of the plurality of features are indicated using an output vector that includes a plurality of weighting values indicating the relative importance of the plurality of features. (Karnagel [0021]: “In an embodiment, a computer calculates, for each feature of a training dataset, a relevance score based on: a relevance scoring function, and statistics of values, of the feature, that occur in the training dataset. A rank based on relevance scores of the features is calculated for each feature. An optimal subset of the features, based on the ranks of the features, is predicted.”; and [0045]: “Thus, blue may be somewhat predictive of positive and, more generally, color may be somewhat predictive of the classification label. Scoring function(s) may calculate a relevance score that is based on correlation of a feature to the label. In an embodiment, correlation may be calculated according to statistics such as mutual information or F score as discussed later herein. Features that somewhat correlate with classification may have a higher relevance score.”; and [0050]: “Once all features F1-F16 have relevance scores, the features can be comparatively ranked according to relevance score.”; Note: The scoring function outputs ranking values and the relevance scores are the output vectors.)
It would have been obvious before the effective filing date to combine the plurality of features of Milne and the feature ranking of Karnagel for efficient and effectiveness of feature selection (Karnagel, [0052]). Milne and Karnagel are analogous art because they both concern feature selection and machine learning.
Claim 17 is rejected over Milne, Karnagel, Mandal, Vinh, Wang and Heinze-Deml with the incorporation of claim 15.
Regarding claim 17, Milne teaches wherein the feature-ranking-based optimization model is a QUBO model. (Milne [page 1, Abstract, line 3]: “Quadratic optimization scales exponentially with the number of features, but a QUBO implementation on a quantum annealer has the potential to be faster than classical solvers”; Note: The quantum annealer is a computer system that performs the method. See page 8, section 8 for the ranking of features.)
Claim 18 is rejected over Milne, Karnagel, Mandal, Vinh, Wang and Heinze-Deml with the incorporation of claim 15.
Regarding claim 18, Milne does not teach wherein the plurality of weighting values for the plurality of features are continuous values provided within a particular range.
However, Karnagel teaches wherein the plurality of weighting values for the plurality of features are continuous values provided within a particular range. (Karnagel [0045]: “Scoring function(s) may calculate a relevance score that is based on correlation of a feature to the label. In an embodiment, correlation may be calculated according to statistics such as mutual information or F score as discussed later herein. Features that somewhat correlate with classification may have a higher relevance score.”; and [0047]: “scoring function(s) may be sensitive to the learned importance or coefficient of a feature.”;)
It would have been obvious before the effective filing date to combine the plurality of features of Milne and the feature ranking of Karnagel for efficient and effectiveness of feature selection (Karnagel, [0052]). Milne and Karnagel are analogous art because they both concern feature selection and machine learning.
Claim 19 is rejected over Milne, Karnagel, Mandal, Vinh, Wang and Heinze-Deml with the incorporation of claim 15.
Regarding claim 19, Milne teaches wherein the weighting values for the plurality of features are integer values provided within a particular range. (Milne [page 12, section 8.2]: “Figure 7 shows the full range of α from 0 to 1. On the left-hand side, where α is close to zero, the emphasis is on feature independence. This favours small subsets, and since their regression coefficients are often not large enough to ‘‘push’’ the classifier across the cutoff point of p ≥ 0.5, the predicted class is 0. They classify almost all of the samples as ‘‘good credit’’ and achieve the zero-rule’s 70% success rate.”;)
Claim 20 is rejected over Milne, Karnagel, Mandal, Vinh, Wang and Heinze-Deml with the incorporation of claim 15.
Regarding claim 20, Milne does not teach wherein the feature-ranking-based optimization model: evaluates a measure of mutual information between groups of two or more features and a set of labels for the plurality of data samples; and evaluates a measure of conditional mutual information between a first feature and the set of labels for the plurality of features provided that a group of two or more other features are selected for inclusion in the reduced feature set.
However, Wang teaches wherein the feature-ranking-based optimization model: evaluates a measure of mutual information between groups of two or more features and a set of labels for the plurality of data samples; and (Wang [page 75, paragraph 1]: “The common information that two variables (features and the set of labels for the plurality of data samples) share is defined as the mutual information (MI) between two variables as
PNG
media_image1.png
56
290
media_image1.png
Greyscale
follows;
“;)
PNG
media_image2.png
28
268
media_image2.png
Greyscale
evaluates a measure of conditional mutual information between a first feature and the set of labels for the plurality of features provided that a group of two or more other features are selected for inclusion in the reduced feature set. (Wang [page 76, paragraphs 1-2]: “They pointed out that parameter b has a great influence on the result of MIFS; if b becomes relatively large, then the MIFS algorithm tends to handle redundancy at the expense of classification accuracy. To eliminate the negative influence of b, Kwak and Choi proposed a more accurate estimation of I(C;fi|fs). They redefined conditional information as follows:
;)
It would have been obvious before the effective filing date to combine the quadratic unconstrained binary optimization model of Milne with the evaluation of mutual information between features of Wang to effectively measure relevancy and redundancy of features (Wang, Abstract). Milne and Wang are analogous art because they both concern feature selection and machine learning.
Claims 3 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Milne, Karnagel, Mandal in view of Vinh, Wang and Heinze-Deml in further view of Weston et al. (US20050131847A1); hereinafter Weston
Claim 3 is rejected over Milne, Karnagel, Mandal, Vinh, Wang, Heinze-Deml and Weston with the incorporation of claim 1.
Regarding claim 3, Milne does not teach wherein, for a particular one of the plurality of features, the selecting includes: comparing a particular one of the ranking values, for the particular feature, to a particular threshold value; and in response to the particular ranking value not meeting the particular threshold value, excluding the particular feature from the reduced feature set.
However, Weston teaches wherein, for a particular one of the plurality of features, the selecting includes: comparing a particular one of the ranking values, for the particular feature, to a particular threshold value; and in response to the particular ranking value not meeting the particular threshold value, excluding the particular feature from the reduced feature set. (Weston [0165]: “Correlation methods such as Golub's method provide a ranked list of genes. The rank order characterizes how correlated the gene is with the separation. Generally, a gene highly ranked taken alone provides a better separation than a lower ranked gene. It is therefore possible to set a threshold (e.g. keep only the top ranked genes) that separates "highly informative genes" from "less informative genes".”;)
It would have been obvious before the effective filing date to combine the plurality of features of Milne and the subset feature selection of Weston to improve generalization performance (Weston, [0319]). Milne and Weston are analogous art because they both concern feature selection and machine learning.
Dependent claim 10 is claim 3 in the form of a non-transitory computer readable medium and is rejected for the same reasons as claim 3 stated above. For the rejection of the limitations specifically pertaining to the non-transitory computer readable medium of claim 8, see the rejection of claim 8 above.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Milne, Karnagel, Mandal, Vinh, Wang and Heinze-Deml in further view of Tan et al. (A multi-objective evolutionary algorithm-based ensemble optimizer for feature selection and classification with neural network models); hereinafter Tan and in further view of Rashid et al. (A Novel Penalty-Based Wrapper Objective Function for Feature Selection in Big Data Using Cooperative Co-Evolution); hereinafter Rashid
Claim 5 is rejected over Milne, Karnagel, Mandal, Vinh, Wang, Heinze-Deml, Tan and Rashid with the incorporation of claim 1.
Regarding claim 5, Milne does not teach wherein the optimization model is an ensemble model, and
However, Tan teaches wherein the optimization model is an ensemble model, and (Tan [Abstract, line 3]: “The aim of the MmGA-based ensemble optimizer is two-fold, i.e. to select a small number of input features for classification and to improve the classification performances of neural network models.”;)
It would have been obvious before the effective filing date to combine the quadratic unconstrained binary optimization model of Milne with an ensemble optimizer of Tan to effectively to select a small number of input features for classification and to improve the classification performances of neural network models (Tan, Abstract). Milne and Tan analogous art because they both concern feature selection and machine learning.
Milne does not teach wherein the objective function utilized in the QUBO model utilizes performance feedback information corresponding to machine learning models that are trained based on candidate feature sets.
However, Rashid teaches wherein the objective function utilized in the QUBO model utilizes performance feedback information corresponding to machine learning models that are trained based on candidate feature sets. (Rashid [page 150117, column 2, last sentence]: “The objective function is defined using different variables, including the number of correctly classified instances (performance feedback information),” the total number of test samples, the number of features selected in the subset, the total number of features in the dataset, and penalty terms”; and [page 150118, column 2, paragraph 1, line 6]: “The new objective function is defined as [equations 5, 6, 7] where Tc is the number of correctly classified instances in the test or training samples; and [page 150117, paragraph 2, line 1]: “The solution with this reduced number of features is then sent to the classifiers (machine learning models trained based on candidate feature sets) to measure accuracy and other metrics. The best individual with a reduced number of features and the highest classification accuracy survives the iterations (feedback information corresponding to the machine learning model).”;)
It would have been obvious before the effective filing date to combine the feature selection of Milne with the performance feedback in the objective function of Rashid to effectively reduce the number of features to decrease computations and improving classification accuracy (Rashid, Abstract, line 8). Milne and Rashid are analogous art because they both concern feature selection and machine learning.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Milne, Karnagel, Mandal, Vinh, Wang and Heinze-Deml in further view of Rashid
Claim 12 is rejected over Milne, Karnagel, Mandal, Vinh, Wang, Heinze-Deml and Rashid with the incorporation of claim 8.
Regarding claim 12, Milne does not teach wherein the objective function utilized in the QUBO model utilizes performance feedback information corresponding to machine learning models that are trained based on candidate feature sets.
However, Rashid teaches wherein the objective function utilized in the QUBO model utilizes performance feedback information corresponding to machine learning models that are trained based on candidate feature sets. (Rashid [page 150117, column 2, last sentence]: “The objective function is defined using different variables, including the number of correctly classified instances (performance feedback information),” the total number of test samples, the number of features selected in the subset, the total number of features in the dataset, and penalty terms”; and [page 150118, column 2, paragraph 1, line 6]: “The new objective function is defined as [equations 5, 6, 7] where Tc is the number of correctly classified instances in the test or training samples; and [page 150117, paragraph 2, line 1]: “The solution with this reduced number of features is then sent to the classifiers (machine learning models trained based on candidate feature sets) to measure accuracy and other metrics. The best individual with a reduced number of features and the highest classification accuracy survives the iterations (feedback information corresponding to the machine learning model).”;)
It would have been obvious before the effective filing date to combine the feature selection of Milne with the performance feedback in the objective function of Rashid to effectively reduce the number of features to decrease computations and improving classification accuracy (Rashid, Abstract, line 8). Milne and Rashid are analogous art because they both concern feature selection and machine learning.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID H TRAN whose telephone number is (703)756-1525. The examiner can normally be reached M-F 9:30 am - 5:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID H TRAN/Examiner, Art Unit 2147
/VIKER A LAMARDO/Supervisory Patent Examiner, Art Unit 2147
Prosecution Insights