DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is responsive to the application filed on 05/18/2023. Claims 1-20 are presented in the case. Claims 1, 8 and 15 are independent claims.
Information Disclosure Statement
The information disclosure statement submitted on 04/19/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claims 1-7 are directed to a system, claims 8-14 are directed to a method and claims 15-20 are directed to a medium. Therefore, the claims are eligible under Step 1 for being directed to a machine, a process and a manufacture respectively.
Independent claims 1 and 8:
Step 2A Prong 1:
Claims recite:
computing, for the first set of features, a first set of measures representing one or more statistical characteristics of values in the first plurality of datasets - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical calculation of computing a set of measures;
comparing the first set of measures against a second set of measures computed based on a second plurality of datasets used for training a second machine learning model - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical relationship by comparing values;
selecting, from a plurality of machine learning model types, a particular machine learning model type for the first machine learning model based on the comparing - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and selecting data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper;
configuring the first machine learning model based on the particular machine learning model type - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and selecting data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper;
determining a set of hyperparameters for training the first machine learning model based on the comparing - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and selecting data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper; and
training the first machine learning model using the first plurality of datasets and based on the set of hyperparameters - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of obtaining data and modifying data to compensate for an artifact, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper, or is a mathematical concept that is achievable through mathematical computation.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because they recite the additional elements:
A system comprising: a non-transitory memory; and one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations - These limitations amount to components of a general purpose computer that applies a judicial exception, by use of conventional computer functions (see MPEP § 2106.05(b)).
obtaining a first plurality of datasets usable for training a first machine learning model, wherein each dataset in the first plurality of datasets comprises a set of values corresponding to a first set of features - the steps recited at a high level of generality, and amounts to mere data gathering which is well known which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are thus directed to the abstract idea.
Step 2B: The claims do not include additional elements that amount to significantly more than the judicial exception.
The additional elements:
A system comprising: a non-transitory memory; and one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations - These limitations amount to components of a general purpose computer that applies a judicial exception, by use of conventional computer functions (see MPEP § 2106.05(b)).
obtaining a first plurality of datasets usable for training a first machine learning model, wherein each dataset in the first plurality of datasets comprises a set of values corresponding to a first set of features - viewed individually or in combination, describes mere data gathering similar to Presenting offers to potential customers and gathering statistics generated based on the testing about how potential customers responded to the offers; the statistics are then used to calculate an optimized price, OIP Technologies, 788 F.3d at 1363, 115 USPQ2d at 1092-93 described in MPEP § 2106.05(g).
Accordingly, these additional elements do not amount to significantly more than the judicial exception. As such, the claims are ineligible.
Independent claim 15:
Step 2A Prong 1:
Claims recite:
computing, for the first set of features and the second set of features, a first set of measures representing one or more statistical characteristics of values in the first plurality of datasets and the second plurality of datasets - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical calculation of computing a set of measures;
comparing the first set of measures against an original set of measures computed based on the first plurality of datasets - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical relationship by comparing values;
selecting, from a plurality of machine learning model types, a second machine learning model type for the first machine learning model based on the comparing - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and selecting data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper;
re-configuring the first machine learning model based on the second machine learning model type - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and selecting data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper;
determining a set of hyperparameters for training the first machine learning model based on the comparing - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and selecting data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper; and
re-training the first machine learning model using the first plurality of datasets and the second plurality of dataset and based on the set of hyperparameters - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of obtaining data and modifying data to compensate for an artifact, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper, or is a mathematical concept that is achievable through mathematical computation.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because they recite the additional elements:
A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations - These limitations amount to components of a general purpose computer that applies a judicial exception, by use of conventional computer functions (see MPEP § 2106.05(b)).
obtaining a first machine learning model which is configured and trained using a first plurality of datasets and a first machine learning model type, wherein each dataset in the first plurality of datasets comprises a set of values corresponding to a first set of features - the steps recited at a high level of generality, and amounts to mere data gathering which is well known which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are thus directed to the abstract idea.
Step 2B: The claims do not include additional elements that amount to significantly more than the judicial exception.
The additional elements:
A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations - These limitations amount to components of a general purpose computer that applies a judicial exception, by use of conventional computer functions (see MPEP § 2106.05(b)).
obtaining a first machine learning model which is configured and trained using a first plurality of datasets and a first machine learning model type, wherein each dataset in the first plurality of datasets comprises a set of values corresponding to a first set of features - viewed individually or in combination, describes mere data gathering similar to Presenting offers to potential customers and gathering statistics generated based on the testing about how potential customers responded to the offers; the statistics are then used to calculate an optimized price, OIP Technologies, 788 F.3d at 1363, 115 USPQ2d at 1092-93 described in MPEP § 2106.05(g).
Accordingly, these additional elements do not amount to significantly more than the judicial exception. As such, the claims are ineligible.
Dependent claims 2, 10 and 18:
Step 2A Prong 1:
Claims recite:
obtaining, from the first plurality of datasets, data values corresponding to a first feature in the first set of features, wherein each of the data values is obtained from a distinct dataset from the first plurality of datasets - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and selecting data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper;
deriving a statistical value from the data values - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and generating data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper; and
computing a first measure in the first set of measures that corresponds to the first feature based on the statistical value - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical calculation of computing a measure.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claims do not provide a practical application and is not considered to be significantly more. As such, the claims are ineligible.
Dependent claims 3, 11 and 17:
Step 2A Prong 1:
Claims recite:
comparing the first set of measures against the second set of measures; and
determining a relationship between the first plurality of datasets and the second plurality of datasets based on the comparing - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and generating data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because they recite the additional elements:
applying the first set of measures and the second set of measures to a grouping algorithm as inputs - These additional elements are recited at a high level of generality and merely invokes a generic computer machinery as a tool to perform the underlying abstract ideas and thus fails to integrate the abstract idea into a practical application. See MPEP 2106.05(f).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are thus directed to the abstract idea.
Step 2B: The claims do not include additional elements that amount to significantly more than the judicial exception.
The additional elements:
applying the first set of measures and the second set of measures to a grouping algorithm as inputs - These additional elements are recited at a high level of generality and merely invokes a generic computer machinery as a tool to perform the underlying abstract ideas. See MPEP 2106.05(f).
Accordingly, these additional elements do not amount to significantly more than the judicial exception. As such, the claims are ineligible.
Dependent claims 4 and 12:
Step 2A Prong 1:
Claims recite:
selecting, from the plurality of machine learning model types, a second machine learning model type for the first machine learning model based on the updated first set of measures - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and selecting data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper;
re-configuring the first machine learning model based on the second machine learning model type - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and generating data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper;
determining a second plurality of hyperparameters for training the first machine learning model based on the updated first set of measures - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and generating data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper; and
training the re-configured first machine learning model using at least one of the first plurality of datasets or the third plurality of datasets based on the second plurality of hyperparameters - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of obtaining data and modifying data to compensate for an artifact, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper, or is a mathematical concept that is achievable through mathematical computation.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because they recite the additional elements:
obtaining a third plurality of datasets usable for training the first machine learning model - the steps recited at a high level of generality, and amounts to mere data gathering which is well known which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g));
updating the first set of measures based on the third plurality of datasets - the steps recited at a high level of generality, and amounts to mere data modifying which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are thus directed to the abstract idea.
Step 2B: The claims do not include additional elements that amount to significantly more than the judicial exception.
The additional elements:
obtaining a third plurality of datasets usable for training the first machine learning model - viewed individually or in combination, describes mere data gathering similar to Presenting offers to potential customers and gathering statistics generated based on the testing about how potential customers responded to the offers; the statistics are then used to calculate an optimized price, OIP Technologies, 788 F.3d at 1363, 115 USPQ2d at 1092-93 described in MPEP § 2106.05(g).
updating the first set of measures based on the third plurality of datasets - the steps recited at a high level of generality, and amounts to mere data modifying which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not amount to significantly more than the judicial exception. As such, the claims are ineligible.
Dependent claims 5, 13 and 19:
Step 2A Prong 1: The claim recites the abstract ideas of claims 1, 8 and 15.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because they recite the additional elements:
wherein the second set of measures corresponds to a second set of features different from the first set of features - the step recited at a high level of generality, and amounts to selecting a particular data source or type of data to be manipulated, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are thus directed to the abstract idea.
Step 2B: The claims do not include additional elements that amount to significantly more than the judicial exception.
The additional elements:
wherein the second set of measures corresponds to a second set of features different from the first set of features - viewed individually or in combination, describes selecting a particular data source or type of data to be manipulated similar to selecting information, based on types of information and availability of information in a power-grid environment, for collection, analysis and display described in MPEP § 2106.05(g).
Accordingly, these additional elements do not amount to significantly more than the judicial exception. As such, the claims are ineligible.
Dependent claims 6, 14 and 20:
Step 2A Prong 1:
Claims recite:
configuring and training a plurality of different versions of the first machine learning model generated using different ones of the plurality of machine learning model types, a plurality of configuration parameters, and a plurality of hyperparameters - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of obtaining data, evaluating data and modifying data to compensate for an artifact, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper, or is a mathematical concept that is achievable through mathematical computation;
evaluating the plurality of different versions of the first machine learning model - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper;
determining a configuration and training setting for the first machine learning model based on the evaluating - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and selecting data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper; and
associating the configuration and training setting with the first set of measures - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and selecting data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claims do not provide a practical application and is not considered to be significantly more. As such, the claims are ineligible.
Dependent claim 7:
Step 2A Prong 1: The claim recites the abstract ideas of claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because it recites the additional elements:
wherein the first set of measures comprise at least one of statistical features of the first plurality of datasets, a central tendency of the first plurality of datasets, a skewness of the first plurality of datasets, a spread among datasets in the first plurality of datasets, one or more patterns of the first plurality of datasets, a frequency of a value in the first plurality of datasets, a presence of outliers in the first plurality of datasets, a correlation between every two measures in the first set of measures, or a type of probability distribution of the first plurality of datasets - the step recited at a high level of generality, and amounts to selecting a particular data source or type of data to be manipulated, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are thus directed to the abstract idea.
Step 2B: The claims do not include additional elements that amount to significantly more than the judicial exception.
The additional elements:
wherein the first set of measures comprise at least one of statistical features of the first plurality of datasets, a central tendency of the first plurality of datasets, a skewness of the first plurality of datasets, a spread among datasets in the first plurality of datasets, one or more patterns of the first plurality of datasets, a frequency of a value in the first plurality of datasets, a presence of outliers in the first plurality of datasets, a correlation between every two measures in the first set of measures, or a type of probability distribution of the first plurality of datasets - viewed individually or in combination, describes selecting a particular data source or type of data to be manipulated similar to selecting information, based on types of information and availability of information in a power-grid environment, for collection, analysis and display described in MPEP § 2106.05(g).
Accordingly, these additional elements do not amount to significantly more than the judicial exception. As such, the claims are ineligible.
Dependent claim 9:
Step 2A Prong 1:
Claim recites:
wherein the determining the configuration comprises selecting, from a plurality of machine learning model types, a particular machine learning model type for configuring the first machine learning model based on the comparing - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and selecting data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claims do not provide a practical application and is not considered to be significantly more. As such, the claims are ineligible.
Dependent claim 16:
Step 2A Prong 1:
Claim recites:
deriving a statistical value from the data values - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and generating data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper; and
computing a first measure in the first set of measures that corresponds to the first feature based on the statistical value - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical calculation of computing a set of measures.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because they recite the additional elements:
obtaining, from the first plurality of datasets and the second plurality of datasets, data values corresponding to a first feature in the first set of features and the second set of features, wherein each of the data values is obtained from a distinct dataset from the first plurality of datasets and the second plurality of datasets - the steps recited at a high level of generality, and amounts to mere data gathering which is well known which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are thus directed to the abstract idea.
Step 2B: The claims do not include additional elements that amount to significantly more than the judicial exception.
The additional elements:
obtaining, from the first plurality of datasets and the second plurality of datasets, data values corresponding to a first feature in the first set of features and the second set of features, wherein each of the data values is obtained from a distinct dataset from the first plurality of datasets and the second plurality of datasets - viewed individually or in combination, describes mere data gathering similar to Presenting offers to potential customers and gathering statistics generated based on the testing about how potential customers responded to the offers; the statistics are then used to calculate an optimized price, OIP Technologies, 788 F.3d at 1363, 115 USPQ2d at 1092-93 described in MPEP § 2106.05(g).
Accordingly, these additional elements do not amount to significantly more than the judicial exception. As such, the claims are ineligible.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-5, 7-13 and 15-19 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by SAHA et al. (hereinafter SAHA), US 20230049418 A1.
Regarding independent claim 1, SAHA teaches a system (Fig. 1, 102) comprising:
a non-transitory memory ([0083] The electronic storages may include non-transitory storage media that electronically stores information); and
one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations comprising ([0082] In some embodiments, the various computers and subsystems illustrated in FIG. 1 may include one or more computing devices that are programmed to perform the functions described herein. The computing devices may include one or more electronic storages (e.g., database(s) 130, which may include dataset database 132, model database 134, training data database 136, etc., or other electronic storages), one or more physical processors programmed with one or more computer program instructions, and/or other components):
obtaining a first plurality of datasets usable for training a first machine learning model, wherein each dataset in the first plurality of datasets comprises a set of values corresponding to a first set of features ([0076] FIG. 12 shows a flowchart of a method 1200 for generating training data including features having minimized correlation, in accordance with one or more embodiments. In an operation 1202, datasets including a plurality of features may be obtained. The datasets may be obtained from one or more data sources (e.g., dataset database 132). Each dataset may include one or more data items, and the data items may represent various features … In some embodiments, operation 1202 may be performed by a subsystem that is the same or similar to scoring subsystem 112);
computing, for the first set of features, a first set of measures representing one or more statistical characteristics of values in the first plurality of datasets ([0077] In an operation 1204, a plurality of correlation scores indicating a correlation between features of the plurality of features may be computed … In some embodiments, operation 1204 may be performed by a subsystem that is the same or similar to scoring subsystem 112);
comparing the first set of measures against a second set of measures computed based on a second plurality of datasets used for training a second machine learning model ([0078] In an operation 1206, a plurality of feature clusters may be generated. Each cluster may include one or more features that are determined to be correlated with one another. For example, if two features are determined to be correlated, both of those features may be clustered into a same feature cluster. Features that are determined to lack correlation (e.g., correlation score is less than a threshold correlation score, correlation score is zero) may be included in different feature clusters … In some embodiments, operation 1206 may be performed by a subsystem that is the same or similar to clustering subsystem 114);
selecting, from a plurality of machine learning model types, a particular machine learning model type for the first machine learning model based on the comparing ([0079] In an operation 1208, a machine learning model may be selected based on a set of input features of the machine learning model and the plurality of clusters. Each machine learning model (e.g., machine learning models stored in model database 134) may take, as input, a set of input features. For example, with reference to FIG. 6, a first machine learning model may be associated with a first set of input features including “Feature 1,” “Feature 2,” and “Feature 3,” whereas a second machine learning model may be associated with a second set of input features including “Feature 1” and “Feature 4.” In some embodiments, the sets of input features associated with each machine learning model may be determined. In some embodiments, a determination may be made as to whether, for a given set of input features, two or more of the features of the set of input features are included in a same feature cluster. For example, with reference to FIG. 7, feature cluster A includes "Feature 1" and "Feature 2," and feature cluster B includes "Feature 3" and "Feature 4." However, the first set of input features includes "Feature 1," "Feature 2," and "Feature 3," and both "Feature 1" and "Feature 2" are included in feature cluster A. Therefore, if the first machine learning model is selected, at least some of the features used as inputs for the model will be correlated, which can lead to decreased accuracy and performance of the machine learning model. As another example, the second set of input features includes “Feature 1” and “Feature 4,” which are respectively included in feature cluster A and feature cluster B. Therefore, if the second machine learning model is selected, none of the input features for the model will be correlated. In some embodiments, because no input features will be correlated, the second machine learning model may be selected … In some embodiments, operation 1208 may be performed by a subsystem that is the same or similar to model selection subsystem 116);
configuring the first machine learning model based on the particular machine learning model type ([0080] In an operation 1210, a subset of datasets may be selected based on the set of input features of the selected machine learning model. Using the previous example, if the second machine learning model is selected, datasets including “Feature 1” and “Feature 4” may be selected from dataset database 132. In some embodiments, the selected datasets may only include the features of the selected model’s set of input features. However, in some embodiments, additional features may be included in those datasets. In some cases, additional quality checks may be performed to ensure that the datasets that are selected do not include any correlated features. In some embodiments, the datasets that are selected may be a subset of the obtained datasets from operation 1202. In some embodiments, operation 1210 may be performed by a subset that is the same or similar to training subsystem 118);
determining a set of hyperparameters for training the first machine learning model based on the comparing ([0081] In an operation 1212, training data may be generated based on the selected subset of datasets. The training data may be generated such that the training data includes some or all of the subset of datasets. In some embodiments, generating the training data may be part of generating build data. The build data may include the training data and test data, where the test data is used to test an accuracy of the trained machine learning model. The training data, upon generation, may be stored in training data database 136 and used to train the selected machine learning model. In some embodiments, operation 1212 may be performed by a subset that is the same or similar to training subsystem 118; Fig. 8; [0058] In some embodiments, training subsystem 118 may be configured to generate build data 808 based on datasets 806. Generating build data 808 may include formatting, transforming, curating, or performing other processes to engineer the features included in datasets 806 for being used to training a machine learning model. For instance, datasets 806 may be organized such that data items included in datasets 806 can be input to a machine learning model, and the hyperparameters of the machine learning model can be adjusted to minimize a cost function of the mode); and
training the first machine learning model using the first plurality of datasets and based on the set of hyperparameters ([0060] In some embodiments, training subsystem 118 may automatically begin training the machine learning model after build data 808 has been generated; [0106] training the first machine learning model using the training data to obtain a trained machine learning model).
Regarding dependent claim 2, SAHA teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. SAHA further teaches wherein the computing the first set of measures (Fig. 3) comprises:
obtaining, from the first plurality of datasets, data values corresponding to a first feature in the first set of features, wherein each of the data values is obtained from a distinct dataset from the first plurality of datasets ([0041] At 304, scoring subsystem 112, upon receipt of datasets 302, may extract features from each of datasets 302. In some embodiments, scoring subsystem 112 may parse datasets 302 to identify the data items stored therein, and further identify which features are represented by each of those data items. Identifying the features may include performing a semantic analysis of the data items to identify entities described by or included within each data item, resolving a label (e.g., tag) for each entity, and attributing the label to each entity (if those entities are not labeled));
deriving a statistical value from the data values ([0041] After the data items are parsed, the features included within the data items may be grouped together by the data item with which they were extracted from. In some embodiments, the features may be grouped based on similarity to one another. For example, each of groups 308 represents a group of features extracted from datasets 302. For instance, a first group may include features Xai, XA2, ..., XAN, a second group may include features XB1, XB2, ..., XBN, and an M-th group including features XM1, XM2, ..., XMN. In some embodiments, each of the M groups may include a same number of features (e.g., N features), however, some groups may include fewer (or more) features); and
computing a first measure in the first set of measures that corresponds to the first feature based on the statistical value ([0042] At 306, a correlation score 310 for pairs of features may be computed. In some embodiments, the correlation score may be between features of a same group (e.g., a correlation score between feature XA1 and feature XA2), between features of different groups (e.g., a correlation score between feature XA1 and feature XB1), or other combinations).
Regarding dependent claim 3, SAHA teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. SAHA further teaches wherein the comparing the first set of measures against the second set of measures comprises:
applying the first set of measures and the second set of measures to a grouping algorithm as inputs ([0043] Returning to FIG. 1, clustering subsystem 114 may cluster features together based on the correlation scores. The result of the clustering may be feature clusters each including features that are uncorrelated with one another. As an example, with reference to FIG. 4, clustering subsystem 114 may be configured to identify, for each feature, one or more correlated features at 402);
comparing the first set of measures against the second set of measures ([0043] Correlated features may be identified based on correlation scores 310, which indicate whether one feature is correlated to another feature. Identifying correlated features may include identifying how strongly or weakly two features are to one another, but, in particular, identifying whether two features are correlated); and
determining a relationship between the first plurality of datasets and the second plurality of datasets based on the comparing ([0044] In some embodiments, at 404, correlated features may be clustered together to generate clustering data 408. Clustering data 408 may include each feature extracted from the datasets (e.g., datasets 302) and, for each of the features, any other features determined to be correlated thereto; [0045] clustering subsystem 114 may further be configured to generate a ranking of the features included within a given feature cluster. The ranking may be determined based on the correlation scores for each feature).
Regarding dependent claim 4, SAHA teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. SAHA further teaches wherein the operations further comprise:
obtaining a third plurality of datasets usable for training the first machine learning model ([0026] The training data may be used to train the machine learning model, whereas the test data may be used to determine how well the machine learning model has been trained. In some embodiments, if the machine learning model is determined to be trained poorly (e.g., an accuracy of the model is less than a threshold accuracy), then new datasets may be retrieved from dataset database 132, and the new datasets may be used to develop new training data and new test data to retrain the model; [0058] if the accuracy does not satisfy the threshold accuracy condition, then new training data may be generated from datasets 806, additional datasets retrieved from dataset database 132 that also include input features 802);
updating the first set of measures based on the third plurality of datasets ([0058] This process may be repeated until the accuracy of the machine learning model satisfies the threshold accuracy condition, or until another stopping criterion is satisfied; [0077] In an operation 1204, a plurality of correlation scores indicating a correlation between features of the plurality of features may be computed);
selecting, from the plurality of machine learning model types, a second machine learning model type for the first machine learning model based on the updated first set of measures ([0058] This process may be repeated until the accuracy of the machine learning model satisfies the threshold accuracy condition, or until another stopping criterion is satisfied; [0079] In an operation 1208, a machine learning model may be selected based on a set of input features of the machine learning model and the plurality of clusters);
re-configuring the first machine learning model based on the second machine learning model type ([0058] This process may be repeated until the accuracy of the machine learning model satisfies the threshold accuracy condition, or until another stopping criterion is satisfied; [0080] In an operation 1210, a subset of datasets may be selected based on the set of input features of the selected machine learning model);
determining a second plurality of hyperparameters for training the first machine learning model based on the updated first set of measures ([0058] This process may be repeated until the accuracy of the machine learning model satisfies the threshold accuracy condition, or until another stopping criterion is satisfied; [0081] In an operation 1212, training data may be generated based on the selected subset of datasets. The training data may be generated such that the training data includes some or all of the subset of datasets. In some embodiments, generating the training data may be part of generating build data. The build data may include the training data and test data, where the test data is used to test an accuracy of the trained machine learning model. The training data, upon generation, may be stored in training data database 136 and used to train the selected machine learning model. In some embodiments, operation 1212 may be performed by a subset that is the same or similar to training subsystem 118; Fig. 8; [0058] In some embodiments, training subsystem 118 may be configured to generate build data 808 based on datasets 806. Generating build data 808 may include formatting, transforming, curating, or performing other processes to engineer the features included in datasets 806 for being used to training a machine learning model. For instance, datasets 806 may be organized such that data items included in datasets 806 can be input to a machine learning model, and the hyperparameters of the machine learning model can be adjusted to minimize a cost function of the mode); and
training the re-configured first machine learning model using at least one of the first plurality of datasets or the third plurality of datasets based on the second plurality of hyperparameters ([0058] This process may be repeated until the accuracy of the machine learning model satisfies the threshold accuracy condition, or until another stopping criterion is satisfied; [0060] In some embodiments, training subsystem 118 may automatically begin training the machine learning model after build data 808 has been generated; [0106] training the first machine learning model using the training data to obtain a trained machine learning model).
Regarding dependent claim 5, SAHA teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. SAHA further teaches wherein the second set of measures corresponds to a second set of features different from the first set of features ([0042] a correlation score 310 for pairs of features may be computed. In some embodiments, the correlation score may be between features of a same group (e.g., a correlation score between feature XA1 and feature XA2), between features of different groups (e.g., a correlation score between feature XA1 and feature XB1), or other combinations).
Regarding dependent claim 7, SAHA teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. SAHA further teaches wherein the first set of measures comprise at least one of statistical features of the first plurality of datasets, a central tendency of the first plurality of datasets, a skewness of the first plurality of datasets, a spread among datasets in the first plurality of datasets, one or more patterns of the first plurality of datasets, a frequency of a value in the first plurality of datasets, a presence of outliers in the first plurality of datasets, a correlation between every two measures in the first set of measures, or a type of probability distribution of the first plurality of datasets ([0027] At data cleaning 216, the build data may be treated if missing observations are present, or if it is determined that any outliers are present in the data. In order to ensure that the model is accurately trained, the build data should accurately reflect the types of data the model is to expect in real-world applications. Therefore, identifying outliers, or other abnormalities, in the data prior to being used to train the model can eliminate potential sources of error; [0034] For each feature included in the datasets, a correlation score may be computed. The correlation score may indicate how well correlated each feature is to each other feature. In some embodiments, the correlation score may be represented using a Pearson score, computed using a Pearson Correlation Coefficient. In some embodiments, the correlation score may be represented using a Spearman Coefficient score computed using a Spearman Correlation Coefficient. In some embodiments, the correlation score may be represented using a Variance Inflation Factor (VIF) computed by determining how much a variance of an estimated regression coefficient is increased due to collinearity).
Regarding independent claim 8, claim 8 contains substantially similar limitations to those found in claim 1. Therefore, it is rejected for the same reason as claim 1 above.
Regarding dependent claim 9, claim 9 contains substantially similar limitations to those found in claim 1. Therefore, it is rejected for the same reason as claim 1 above.
Regarding dependent claim 10, claim 10 contains substantially similar limitations to those found in claim 2. Therefore, it is rejected for the same reason as claim 2 above.
Regarding dependent claim 11, claim 11 contains substantially similar limitations to those found in claim 3. Therefore, it is rejected for the same reason as claim 3 above.
Regarding dependent claim 12, claim 12 contains substantially similar limitations to those found in claim 4. Therefore, it is rejected for the same reason as claim 4 above.
Regarding dependent claim 13, claim 13 contains substantially similar limitations to those found in claim 5. Therefore, it is rejected for the same reason as claim 5 above.
Regarding independent claim 15, claim 15 contains substantially similar limitations to those found in claims 1 and 4. Therefore, it is rejected for the same reason as claims 1 and 4 above.
Regarding dependent claim 16, claim 16 contains substantially similar limitations to those found in claims 1 and 2. Therefore, it is rejected for the same reason as claims 1 and 2 above.
Regarding dependent claim 17, claim 17 contains substantially similar limitations to those found in claim 3. Therefore, it is rejected for the same reason as claim 3 above.
Regarding dependent claim 18, claim 18 contains substantially similar limitations to those found in claim 2. Therefore, it is rejected for the same reason as claim 2 above.
Regarding dependent claim 19, claim 19 contains substantially similar limitations to those found in claim 5. Therefore, it is rejected for the same reason as claim 5 above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 6, 14 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over SAHA as applied in claims 1, 8 and 15, in view of Ma et al. (hereinafter Ma), US 20200401948 A1.
Regarding dependent claim 6, SAHA teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. SAHA does not explicitly teach wherein the operations further comprise:
configuring and training a plurality of different versions of the first machine learning model generated using different ones of the plurality of machine learning model types, a plurality of configuration parameters, and a plurality of hyperparameters;
evaluating the plurality of different versions of the first machine learning model;
determining a configuration and training setting for the first machine learning model based on the evaluating; and
associating the configuration and training setting with the first set of measures.
However, in the same field of endeavor, Ma teaches
configuring and training a plurality of different versions of the first machine learning model generated using different ones of the plurality of machine learning model types, a plurality of configuration parameters, and a plurality of hyperparameters (Fig. 2; [0045] an exploration apparatus 212 trains each machine learning model using a different training configuration (e.g., training configuration 1 244, training configuration n 246). Each training configuration contains a set of features to be inputted into the corresponding machine learning model; [0048] Each training configuration also, or instead, includes one or more hyperparameters for the corresponding machine learning model. For example, the hyperparameters include a convergence parameter that adjusts the rate of convergence of the machine-learning model. In another example, the hyperparameters include a clustering parameter that controls the amount of clustering (e.g., number of clusters) in a clustering technique and/or classification technique that utilizes clusters);
evaluating the plurality of different versions of the first machine learning model ([0051] After exploration apparatus 212 trains global and personalized versions of a given machine learning model using training dataset 216 and the corresponding training configuration, exploration apparatus 212 evaluates the performance (e.g., performance 1 240, performance n 242) of the machine learning model using evaluation dataset 218);
determining a configuration and training setting for the first machine learning model based on the evaluating ([0052] exploration apparatus 212 trains and evaluates multiple machine learning models using different training configurations to explore different feature sets and/or hyperparameters for the machine learning models. In turn, exploration apparatus 212 identifies feature sets and/or hyperparameters that result in the best-performing machine learning model.); and
associating the configuration and training setting with the first set of measures ([0055] a sampling apparatus 204 generates a sampled training dataset 224 that includes records 228 that are sampled from training dataset 216 and a sampled evaluation dataset 226 that includes records 230 that are sampled from evaluation dataset 218).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of generating sampled evaluation and training datasets repeated a number of times and using each sampled training dataset and the corresponding sampled evaluation dataset with a different training configuration to train and evaluate the machine learning model as suggested in Ma into SAHA’s system because both of these systems are addressing training and evaluating machine learning models. This modification would have been motivated by the desire to facilitate machine learning and/or analytics by mechanisms for improving the creation, profiling, management, sharing, and reuse of features and/or machine learning models (Ma, [0005]).
Regarding dependent claim 14, claim 14 contains substantially similar limitations to those found in claim 6. Therefore, it is rejected for the same reason as claim 6 above.
Regarding dependent claim 20, claim 20 contains substantially similar limitations to those found in claim 6. Therefore, it is rejected for the same reason as claim 6 above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
HIGGINS et al. (US 20210065053 A1) discloses automated data processing and machine learning model generation.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMY P HOANG whose telephone number is (469)295-9134. The examiner can normally be reached M-TH 8:30-5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JENNIFER WELCH can be reached at 571-272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AMY P HOANG/Examiner, Art Unit 2143
/JENNIFER N WELCH/Supervisory Patent Examiner, Art Unit 2143