Detailed Action
This action is written in response to the application filed 9/18/24. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
Claims 1-6 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
In determining whether the claims are subject matter eligible, the Examiner applies the 2019 USPTO Patent Eligibility Guidelines, as well as guidance from MPEP § 2106.
Step 1: Is the claim to a process, machine, manufacture, or composition of matter? Yes—claim 1 recites a method, which is a process.
Step 2A, prong one: Does the claim recite an abstract idea, law of nature or natural phenomenon? Yes—the claim recites one or more limitations which—under their broadest reasonable interpretation—covers performance of the limitation in the mind (see table below).
Claim limitation
Examiner analysis
1.An outlier removal method for removing an outlier included in training data that comprises data of an explanatory variable and an objective variable used for machine learning, the method comprising: …
an outlier determination step of determining an outlier by determining whether each data included in the training data is an outlier based on a calculation result of the evaluation metrics calculation step by using both the first evaluation metric and the second evaluation metric as metrics when the each data is used as the test data; and
This is a mental process akin to a human evaluation/judgment.
an outlier removal step of removing data determined to be an outlier in the outlier determination step.
This is a mental process akin to a human evaluation/judgment.
Because the claim recites limitations which can practically be implemented as mental processes, the claim recites a mental process.
Step 2A, prong two: Does the claim recite additional elements that integrate the judicial exception into a practical application? No—removing outliners form a data set is a thinking and modeling tool with potential applications in any domain. The Applicants do not address any particular real-world problem in their claim.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No—the additional limitations are addressed below:
an evaluation metrics calculation step comprising a model creation step of dividing the training data into teacher data and test data and
This is well-understood, routine and conventional activity within the field of machine learning. See citations to James infra. [The Examiner notes that this reference is an introductory undergraduate machine learning text.]
creating a regression model representing a correlation between the explanatory variable and the objective variable using the teacher data,
This is well-understood, routine and conventional activity within the field of machine learning. See id.
a first evaluation metrics calculation step of calculating an evaluation metric using the test data on the created regression model to provide a first evaluation metric, and
This is well-understood, routine and conventional activity within the field of machine learning. See id.
a second evaluation metrics calculation step of calculating an evaluation metric using the teacher data on the created regression model to provide a second evaluation metric, and
This is well-understood, routine and conventional activity within the field of machine learning. See id.
repeating the model creation step, the first evaluation metrics calculation step, and the second evaluation metrics calculation step for a predetermined number of times;
This is merely repeating the well-understood, routine and conventional activities identified above.
For the reasons above, claim 1 is rejected as being directed to non-patentable subject matter under §101. This rejection applies equally to independent claim 6, which recite an analogous device, as well as to dependent claims 2-5. The additional limitations of the dependent claims are addressed briefly below. Taken alone, the additional elements of the dependent claims above do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation.
Claim limitation
Examiner analysis
2.The outlier removal method, according to claim 1, wherein the outlier determination step includes ranking the first evaluation metric and ranking the second evaluation metric, respectively, calculating a metric value based on both ranks using an arbitrary data as the test data, and determining whether the arbitrary data is an outlier based on the obtained metric value.
This is a mental process akin to a human judgment/opinion.
3.The outlier removal method, according to claim 1, wherein, in the evaluation metrics calculation step, the division is performed in such a manner that only one data in the training data is used as the test data and a remaining data is used as the teacher data.
This is a mental process akin to a human judgment/opinion.
4.The outlier removal method, according to claim 1, wherein a prediction error is used in at least one of the first evaluation metrics and the second evaluation metrics, and at least one of mean error (ME), mean absolute error (MAE) and root mean square error (RMSE) and one of mean percentage error (MPE), mean absolute percentage error (MAPE) and root mean square percentage error (RMSPE) are used as the prediction error.
This is merely additional information about one or more previously identified mental processes.
5.The outlier removal method, according to claim 4, wherein the prediction error is used as the first evaluation metric and a determination coefficient is used as the second evaluation metric.
This is a mental process akin to a human judgment/opinion.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
The following references are relied upon in the rejections below:
James (James G, Witten D, Hastie T, Tibshirani R. An introduction to statistical learning. Second edition. New York: springer; 2021 Aug. PP 1-296.)
Jones (US 2022/0277232 A1)
Lee (US 2023/0209368 A1)
Claims 1-2 and 4-6 are rejected under 35 U.S.C. 103 as being unpatentable over James and Lee.
Regarding claims 1 and 6, James discloses an outlier removal method for removing an outlier included in training data that comprises data of an explanatory variable and an objective variable used for machine learning, the method comprising:
an evaluation metrics calculation step comprising a model creation step of dividing the training data into teacher data and test data and creating a regression model representing a correlation between the explanatory variable and the objective variable using the teacher data,
P. 30, “The MSE in (2.5) is computed using the training data that was used to fit the model, and so should more accurately be referred to as the training MSE. But in general, we do not really care how well the method works on the training data. Rather, we are interested in the accuracy of the predictions that we obtain when we apply our method to previously unseen test data.”
test data :: P. 30, “test data”
teacher data :: all training data, excluding the designated “test data”. [Also referred to as a training set throughout the reference.]
P. 183, “We now split the observations into a test set, containing the first 1,000 observations, and a training set, containing the remaining observations.”
a first evaluation metrics calculation step of calculating an evaluation metric using the test data on the created regression model to provide a first evaluation metric, and
P. 30, “We want to choose the method that gives the lowest test MSE, as opposed to the lowest training MSE.”
a second evaluation metrics calculation step of calculating an evaluation metric using the teacher data on the created regression model to provide a second evaluation metric, and
P. 30, “The MSE in (2.5) is computed using the training data that was used to fit the model, and so should more accurately be referred to as the training MSE.
repeating the model creation step, the first evaluation metrics calculation step, and the second evaluation metrics calculation step for a predetermined number of times;
P. 127, “(h) Repeat (a)–(f) after modifying the data generation process in such a way that there is less noise in the data. The model (3.39) should remain the same. …. (i) Repeat (a)–(f) after modifying the data generation process in such a way that there is more noise in the data.”
an outlier determination step of determining an outlier by determining whether each data included in the training data is an outlier based on a calculation result of the evaluation metrics calculation step by using … the first evaluation metric … when the each data is used as the test data; and
P. 97, “For instance, in this example, the RSE is 1.09 when the outlier is included in the regression, but it is only 0.77 when the outlier is removed. Since the RSE is used to compute all confidence intervals and p-values, such a dramatic increase caused by a single data point can have implications for the interpretation of the fit. Similarly, inclusion of the outlier causes the R2 to decline from 0.892 to 0.805.”
P. 98, “Residual plots can be used to identify outliers. In this example, the outlier is clearly visible in the residual plot illustrated in the center panel of Figure 3.12. But in practice, it can be difficult to decide how large a residual needs to be before we consider the point to be an outlier. To address this problem, instead of plotting the residuals, we can plot the studentized residuals, computed by dividing each residual ei by its estimated standard error. Observations whose studentized residuals are greater than 3 in absolute value are possible outliers. In the right-hand panel of Figure 3.12, the outlier’s studentized residual exceeds 6, while all other observations have studentized residuals between −2 and 2. If we believe that an outlier has occurred due to an error in data collection or recording, then one solution is to simply remove the observation.”
an outlier removal step of removing data determined to be an outlier in the outlier determination step.
Id.
Lee discloses the following further limitation which James does not explicitly disclose:
an outlier determination step of determining an outlier by determining whether each data included in the training data is an outlier based on a calculation result of the evaluation metrics calculation step by using both the first evaluation metric and the second evaluation metric as metrics when the each data is used as the test data; and
[0501] “When the learning is unsuccessful, the AP may increase a learning failure parameter by 1. The learning failure parameter is defined as the number of learning failures. The learning failure means a case in which the training error is not small enough or the difference between the training error and the test error (or generalization error) is not small enough in the learning process of the machine learning network by the AP, but is not limited thereto.”
[0382] “For example, if the Training Error is not small enough or the difference between the Training Error and the Generalization Error (so-called, Test Error) is not small enough, a learning result of the machine learning network is classified into the failure.”
At the time of filing, it would have been obvious to a skilled machine learning engineer to combine the techniques disclosed by James throughout the ISLR textbook. The Examiner notes that this is an introductory machine learning textbook, which is often assigned for undergraduate coursework. Specific motivations include:
Building statistical models is an inherently cyclical process:
model architectures, hyperparameters, parameters, and data sets are chosen;
models are trained and tested;
model performance is evaluated;
model architectures, hyperparameters, parameters can be tuned, with the goal of increasing model performance;
data sets (both training and test) can be evaluated for anomalies / outliers;
this process repeats until a satisfactory performance is achieved.
Real-world data sets are often messy. They often contain outliers which perhaps should not be included in the training or test sets, because they arise from circumstances that differ significantly from expected conditions. Judgment matters here; it often makes sense to remove significant outliers, as noted by James.
A poor training MSE may be enough reason to identify and remove outliers from a training set. A poor test MSE may be enough reason to identify and remove outliers from a test set.
Furthermore, at the time of filing, it would have been obvious to a skilled machine learning engineer to combine the technique taught by Lee for explicitly measuring the difference between training error and test error with the techniques disclosed by James. This metric can specifically indicate problems with the test set, eg an outlier which negatively affects the model’s performance on the test set.
Regarding claim 2, James discloses the further limitation wherein the outlier determination step includes ranking the first evaluation metric and ranking the second evaluation metric, respectively,
P. 270, “The regsubsets() function has a built-in plot() command which can be used to display the selected variables for the best model with a given number of predictors, ranked according to the BIC, Cp, adjusted R2, or AIC.”.
Lee discloses the further limitation which James does not disclose:
calculating a metric value based on both ranks using an arbitrary data as the test data, and determining whether the arbitrary data is an outlier based on the obtained metric value.
[0501] “When the learning is unsuccessful, the AP may increase a learning failure parameter by 1. The learning failure parameter is defined as the number of learning failures. The learning failure means a case in which the training error is not small enough or the difference between the training error and the test error (or generalization error) is not small enough in the learning process of the machine learning network by the AP, but is not limited thereto.”
[0382] “For example, if the Training Error is not small enough or the difference between the Training Error and the Generalization Error (so-called, Test Error) is not small enough, a learning result of the machine learning network is classified into the failure.”
Regarding claim 4, James discloses the further limitation wherein a prediction error is used in at least one of the first evaluation metrics and the second evaluation metrics, and at least one of mean error (ME), mean absolute error (MAE) and root mean square error (RMSE) and one of mean percentage error (MPE), mean absolute percentage error (MAPE) and root mean square percentage error (RMSPE) are used as the prediction error.
[This is a Markush group.]
P. 432, “We report the mean absolute error on the test data.”
Regarding claim 5, James discloses the further limitation wherein the prediction error is used as the first evaluation metric and a determination coefficient is used as the second evaluation metric.
Prediction error :: p. 68, “residual standard error (RSE)”.
The Examiner interprets “determination coefficient” according to its broadest reasonable interpretation in view of the Applicant’s specification at [0022]: “determination coefficient R2”.
P. 68 “The quality of a linear regression fit is typically assessed using two related quantities: the residual standard error (RSE) and the R2 statistic.”
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over James, Lee and Jones.
Regarding claim 3, Jones discloses the following further limitation which neither James/Lee discloses wherein, in the evaluation metrics calculation step, the division is performed in such a manner that only one data in the training data is used as the test data and a remaining data is used as the teacher data.
[0121] For this discussion, a 70%/30% split of {{circumflex over (X)}, Ŷ} is used for training (n records) and testing (j records) (e.g., 70% of the records are training and 30% are testing), however any suitable split may be employed, such as, e.g., 50%/50%, 60%/40%, 80%/20%, 90%/10%, 95%/5%, or other suitable train/test split.
Although Jones does not teach using one datum only in a training set, the passage above indicates that a wide range of values may be appropriate for a train/test split. Such values are often tuned by experimentation, ie values are chosen, and models are trained, tested, and evaluated accordingly. Through trial-and-error, machine learning engineers can determine which ratios are optimal. This is part of routine optimization of statistical models, as suggested by the Jones excerpt above. (See generally MPEP 2144.05 regarding the obviousness of similar and overlapping ranges, amounts, and proportions.) The Examiner notes that in this context, the exact proportion for training/test splits is not critical, because a model trained with any quantity of data can still yield useful results.
Additional Relevant Prior Art
The following reference was identified by the Examiner as being relevant to the disclosed invention, but is not relied upon in any rejection:
Cmielowski discloses a machine learning system featuring a technique for removing outliers to improve model performance. See generally cols. 2-3. (US 11,501,239 B2)
Conclusion
Information regarding the status of an application may be found at the USPTO Patent Center at https://patentcenter.uspto.gov. Any inquiry concerning this communication should be directed to Vincent Gonzales at (571) 270-3837. The examiner can normally be reached Monday-Friday 7 am to 4 pm MT. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang, can be reached at (571) 270-7092.
/Vincent Gonzales/Primary Examiner, Art Unit 2124
