DETAILED ACTION
Status of Claims
Claim(s) 1, 4-5, 7-11, 14-15, 17-19, and 22 are pending and are examined herein.
Claim(s) 1, 10, and 11 have been Amended. Claim(s) 2-3, 6, 12-13, 16, 20-21, and 23-24 are Cancelled.
Claim(s) 1, 4-5, 7-11, 14-15, 17-19, and 22 remain rejected under 35 U.S.C. § 101 and 35 U.S.C. § 103.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 02/18/2025 has been entered.
Response to Amendment
The amendment filed on February 18, 2025 has been entered. Claims 1, 4-5, 7-11, 14-15, 17-19, and 22 are pending in the application. Applicant’s amendments to claims have overcome the objection and the §112 rejection previously set forth in the Non-Final Office Action mailed on October 15, 2024. Applicant’s amendments to the claims have been fully considered and are addressed in the rejections below.
Response to Arguments
Applicant's arguments, with respect to the rejection under 35 U.S.C. § 101 filed on 02/18/2025, have been fully considered but they are not persuasive.
Applicant’s argument (Pp. 9-10 of the remarks): Applicant argues that the claims do not recite a judicial exception because the claimed steps cannot practically be performed entirely by the human mind, as required under the 219 Revised Patent Subject Matter Eligibility Guidance and MPEP 2106. Applicant points out that limitations such as applying a machine learning model to test data accessed via network, and determining score distributions using a mixture model, involve operations that the human mind is not practically equipped to perform alone. Therefore, the claims are directed to patent-eligible subject matter at Step 2A, Prong One.
Examiner's response: The examiner respectfully disagrees with the applicant assertion that the claims do not recite a judicial exception under Step 2A, Prong One of the subject matter eligibility analysis. The claims, as drafted, recite steps involving statistical analysis (e.g., score distribution, clustering, and statistical comparisons) that fall within the judicial exception of mental processes and/or mathematical concepts, as outlined in MPEP § 2106.04(a)(2).
Additionally, the steps of determining score distributions using a mixture model, identifying high score clusters, comparing these clusters using a T-test recite, and determining whether the model’s performance has degraded beyond a threshold are all steps that define statistical data analysis. These steps are directed to mental processes and/or mathematical concepts, and are processes that can be practically performed mentally or with physical aid (e.g., pen and paper). See MPEP § 2106.04(a)(2)(III).
While not all the limitations are directed to the judicial exception (abstract idea), the remaining limitations are insufficient to integrate the judicial exception into a practical application or amount to significantly more under Step 2A, Prong 2, and Step 2B. For example, the recitation of “applying the machine learning model to test data from different time periods” represents a high-level application of machine learning operations without specifying any particular improvement or technical implementation. In other words, the claim invokes a computer or other machinery in its ordinary capacity merely as a tool to perform an existing process. Moreover, merely reciting generic computer components and data access via a network to collect data does not transform the abstract idea into a patent eligible subject matter. Such elements are well-understood, routine, conventional activities that do not meaningfully limit the claim or improve computer technology, see MPEP § 2106.05(f) and § 2106.05(g).
Applicant’s argument (Pp. 11-14 of the remarks): Applicant argues that even if the claims are found to recite a judicial exception, the claims are nonetheless patent-eligible under Step 2A, Prong Two or Step 2B because they integrate the alleged judicial exception into a practical application and amount to significantly more. Applicant asserts that the claims reflect an improvement in technology or a technical field, particularly in the validation of machine learning models. For example, amended claim 1 recite steps of applying a machine learning model to different test dataset over different time periods, generating score distributions using a mixture model, identifying high score clusters, and comparing these cluster using a T-test, and applying a Gaussian Density Estimator (GDE). Applicant contends that these features provide technical benefits, including robust statistical techniques and reducing overfitting.
Examiner's response: The examiner respectfully disagrees. The claim primarily directed to the process of validating the performance of a machine learning model by statistically analyzing the outputs of the model through statistical techniques, data evaluation, and decision-making steps that can be performed in the human mind and/or with the aid of physical tool (e.g., pen and paper). As discussed above, the additional elements recited in the claim are not sufficient to integrate the judicial exception into a practical application or amount to inventive concept. They merely recite generic computer components or add insignificant extra-solution activity to the judicial exception.
For example, the step of applying machine learning model to different test dataset to generate output scores, which was considered as an additional element under Step 2A, Prong Two, amounts to no more than a general instructions to apply an abstract idea of validating the machine learning model. This high-level operation of machine learning in the context simply represents invoking generic computer components to perform conventional process. While the model output is used in the subsequent analysis, it is only used in the context of applying statistical analysis to assess the validity of the model across different runs based on the model’s results, i.e., within the abstract idea itself. These additional elements therefore amount to necessary data gathering and processing steps for carrying out the judicial exception.
Additionally, Applicant’s argument improperly focuses on the judicial exception as providing the alleged improvement. As stated in MPEP § 2106.04(a), a judicial exception alone cannot provide the improvement or amount to inventive concept. Any alleged improvement to the functioning of a computer or another technology can be provided by one or more additional elements in the claim that integrate the exception into a practical application. Furthermore, MPEP § 2106.04(d) explains that the "improvements" analysis in Step 2A determines whether the claim pertains to an improvement to the functioning of a computer or to another technology without reference to what is well-understood, routine, conventional activity. In this case, it can be shown that the alleged improvement comes the judicial exception and that the identified additional elements are not, themselves or when considered in combination, amount to the alleged improvement. Rather, the judicial exception alone is modified by the identified additional elements in a high-level and generic manner.
Accordingly, the claimed additional elements fail to provide meaningful limitations that can transform the exception into a patent-eligible application.
In view of the above, the rejection under 35 U.S.C. § 101 is maintained. For more details on the subject matter eligibility analysis of the claim, the Examiner respectfully refers to the detailed provided in the rejection under 35 U.S.C. § 101.
Applicant's arguments, with respect to the rejection under 35 U.S.C. § 103 filed on 02/18/2025 (see remarks Pp. 15-16) have been fully considered but are not persuasive.
The arguments amount to a general allegation that the cited references fail to teach or suggest the amended claim, without specifically identifying distinctions or novel features that differentiate the claim over the cited art. Moreover, Applicant’s arguments are moot in view of the current ground of rejection.
The Examiner refers to the rejection under 35 U.S.C. § 103 for more details.
Information Disclosure Statement
The non-patent literature documents in the following information disclosure statements were not considered due to the following reasons:
IDS 06/30/2021: ArcGIS for Desktop, “Overlay analysis” Accessed 06.30.2021. https://desktop.arcgis.com/en/arcmap/10.3/analyze/commonly-used-tools/overlay-analysis.htm.
The IDS indicates that this document was retrieved on 06/30/2021. However, Examiner found multiple archives of this web page reference through the Wayback Machine (web.archive.org) with archive dates 04/11/2021 and 05/07/2021, but none with an archive retrieval date of 06/30/2021. See MPEP 609.04(a), 609.05(a), and 37 CFR 1.98 for the requirements for submitting archived Internet publications. Applicant is requested to provide the URL of the website (that includes the archive retrieval date from which the archived copy of the Web page was obtained) for further consideration of this document.
IDS 06/30/2021: Scikit Learn. “3.3 Metrics and scoring: quantifying the quality of predictions” Accessed 06.30.2021. https://scikit-learn.org/stable/modules/model_evaluation.html.
The IDS indicates that this document was retrieved on 06/30/2021. However, Examiner found multiple archives of this web page reference through the Wayback Machine (web.archive.org) with archive dates between 01/16/2021 and 04/15/2021, but none with an archive retrieval date of 06/30/2021. See MPEP 609.04(a), 609.05(a), and 37 CFR 1.98 for the requirements for submitting archived Internet publications. Applicant is requested to provide the URL of the website (that includes the archive retrieval date from which the archived copy of the Web page was obtained) for further consideration of this document.
IDS 06/30/2021: Splunk Machine Learning Tool Kit User Guide. “Scoring metrics in the Machine Learning Toolkit” Accessed 06.30.2021. https://docs.splunk.com/Documentation/MLApp/5.2.1/User/ScoreCommand.
The IDS indicates that this document was retrieved on 06/30/2021. However, Examiner found multiple archives of this web page reference through the Wayback Machine (web.archive.org) with archive dates of 4/19/2021 and 05/09/2021, but none with an archive retrieval date of 06/30/2021. See MPEP 609.04(a), 609.05(a), and 37 CFR 1.98 for the requirements for submitting archived Internet publications. Applicant is requested to provide the URL of the website (that includes the archive retrieval date from which the archived copy of the Web page was obtained) for further consideration of this document.
Claim Objections
Claims 9 and 19 is objected to under 37 CFR 1.75(c) as being in improper dependent form because :
Claims 6 and 19 depend from canceled claims 6 and 16. Applicant is required to amend claims 9 and 19 to depend from an appropriate base claim. The subject matter of claims 9 and 19 has been examined as depending from claims 1 and 11, respectively.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
When considering subject matter eligibility under 35 U.S.C. 101, it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter (Step 1). If the claim does fall within one of the statutory categories, the second step in the analysis is to determine whether the claim is directed to a judicial exception (Step 2A). The Step 2A analysis is broken into two prongs. In the first prong (Step 2A, Prong 1), it is determined whether or not the claims recite a judicial exception (e.g., mathematical concepts, mental processes, certain methods of organizing human activity). If it is determined in Step 2A, Prong 1 that the claims recite a judicial exception, the analysis proceeds to the second prong (Step 2A, Prong 2), where it is determined whether or not the claims integrate the judicial exception into a practical application. If it is determined at step 2A, Prong 2 that the claims do not integrate the judicial exception into a practical application, the analysis proceeds to determining whether the claim is a patent-eligible application of the exception (Step 2B). If an abstract idea is present in the claim, any element or combination of elements in the claim must be sufficient to ensure that the claim integrates the judicial exception into a practical application, or else amounts to significantly more than the abstract idea itself. Applicant is advised to consult MPEP 2106 for more details of the analysis.
Under Step 1 analysis,
Claims 1, 4-5, and 7-9 recite a method (representing a process);
Claims 10 recite a non-transitory computer readable medium (representing an article of manufacture); and
Claims 11, 14-15, 17-19, and 22 recite a system (representing a machine).
Therefore, each of the claims falls into one of the four statutory categories (i.e., process, machine, article of manufacture, or composition of matter).
Claims 1, 4-5, 7-11, 14-15, 17-19, and 22
are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more, and hence is not patent-eligible subject matter.
Regarding Amended Claim 1,
Step 2A Prong 1: The claim recites an abstract idea enumerated in the 2019 PEG.
determining, … using a mixture model, a first score distribution for a first run of the machine learning model based on the first plurality of scores; determining, … using a mixture model, a second score distribution for a second run of the machine learning model based on the second plurality of scores; (The “determining” steps, as drafted and under their broadest reasonable interpretation, are directed to concepts that fall within the mental process and/or mathematical concept groupings of abstract ideas. Specifically, the cat of determining a score distribution based on observed outputs of a machine learning model using statistical analysis and algorithms encompasses processes that can be manually performed by an individual (e.g., a data scientist) with the aid of physical tools (e.g., pen and paper). The use of a mixture model, which is a conventional statistical tool used to estimate a probability distribution over data, further reflects a mathematical concept. The statistical or algorithmic estimation involved in generating such a distribution can be performed mentally with the aid of pen and paper, and thus also represents as a mental process. According to MPEP § 2106.04(a)(2)(I), performing a resampled statistical analysis to generate resampled distribution is a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification. Furthermore, MPEP § 2106.04(a)(2)(III) explains that the use of a physical aid (e.g., pencil and paper or a slide rule) to help perform a mental step (e.g., a mathematical calculation) does not negate the mental nature of the limitation, but simply accounts for variations in memory capacity from one person to another. Accordingly, these “determining” steps recite abstract ideas and fall within a judicial exception.)
identifying, …, a portion of the first score distribution comprising a first high scores cluster for the first score distribution; identifying, …, a portion of the second score distribution comprising second high scores cluster for the second score distribution; (The “identifying” steps, as drafted and under their broadest reasonable interpretation, fall within the category of mental processes. These steps encompass acts that can be performed in the human mind, including observation, evaluation, determination, and opinion, as outlined in MPEP § 2106.04(a)(2)(III). The identification of high score clusters involves extracting a subset of values—such as the rightmost peak of the score distribution, that represents a statistical subpopulation based on an evaluation of the distribution. This is part of a statistical analysis and merely adds additional step of identifying the portion that best represents the distribution’s most confident outputs. This form of data analysis and subpopulation selection can be performed mentally by a person, such as a data scientist examining plotted (e.g., histogram) distributions, or with the assistance of physical aid (e.g., pen and paper or graphing tool). Accordingly, these are mental steps involving evaluation and analysis of data that can be performed in the human mind and/or with the aid of physical tools. See MPEP § 2106.04(a)(2)(III).)
comparing, … using a T-test, a portion of the first high scores cluster and a portion of the second high scores cluster to evaluate a change in performance of the machine learning model from the first time period to the second time period, wherein the portion of the first high scores cluster and the portion of the second high scores cluster is extracted by fitting a subset of the first high scores cluster and a subset of the second high scores cluster into a Gaussian Density Estimator (GDE); (The “comparing” step, as drafted and under its broadest reasonable interpretation, is directed to an abstract idea of a mental process. This step involves observing two sets of numerical data—namely, portions of high-score clusters, evaluating whether their means are statistically different, and drawing a conclusion about a change in model performance. As such, the use of a T-test to perform the comparison represents a form of statistical analysis that can be performed in the human mind, including observation, evaluation, and judgment, as described in MPEP § 2106.04(a)(2)(III). Additionally, the step of fitting data into a Gaussian density estimator (GDE) is part of the abstract idea of identifying a portion of high score clusters within a respective distribution. This involves estimating a probability distribution using a gaussian density function, which encompasses mathematical functions and statistical techniques. Accordingly, this step also falls within the mental process category, as it involves statistical data analysis using an algorithm or formula to identify the highest density points in a distribution. This is an operation that can be perfumed mentally with the aid of physical tools (e.g., pen and paper).)
determining, … based on the comparing of the at least the portion of the first and second high scores clusters, that the machine learning model is validated, (The “determining” step, as drafted and under its broadest reasonable interpretation, falls within the mental process category of abstract idea. This step involves drawing a conclusion about the validity of machine learning mode based on the results of a prior statistical comparison between portions of high score clusters. Such a determination is an act of evaluation and decision, i.e., forming an opinion or conclusion based on statistical data analysis, which can be performed in the human mind. This involves identifying changes over time that indicate that the performance (i.e., quality or accuracy) of the ML model over time has not degraded, where an indication that the performance of the ML model has not degraded over time represents a determination that the ML model is validated, while an indication that the performance of the ML model has degraded over time represents a determination that the ML model is not validated. An individual can establish a set of model validation criteria representing rules/condition and/or thresholds that measures the performance accuracy of the ML model, and apply the established model validation criteria to the results from the earlier comparison of the respective portions of the high score clusters to make a determination whether the existing model satisfies/meets the validation criteria. As noted in MPEP § 2106.04(a)(2)(III), mental processes include acts of observation, evaluation, judgment, and opinion. Accordingly, this step recites an abstract idea and thus, represents an act that can be performed in the human mind.)
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
Additional elements:
applying, … , a machine learning model to first test data observed over a first time period to generate a first plurality of outputs and a first plurality of scores associated with confidence levels of the first plurality of outputs, …. applying, … , the machine learning model to second test data observed over a second time period to generate a second plurality of outputs and a second plurality of scores associated with confidence levels of the second plurality of outputs, … (This amounts to no more than merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, as discussed in MPEP § 2106.05(f). The recited applying limitations describe the application of a generic machine learning model to time-separated test data in order to generate confidence scores. However, the model is not claimed as novel or improved, and the application is used solely as a data preparation step to support the later abstract evaluation of the model’s validity i.e., the generated confidence scores are used in the process of validating the model through statistical analysis. Accordingly, the claim invokes computers or other machinery in their ordinary capacity merely as a tool to perform an excising process. Examiner notes that the high-level recitation of applying a machine learning model to different test data merely reflects conventional machine learning operations, which are equivalent to applying computers and/or computer instructions to perform a known process.)
The claimed “model validator,” which is configured to perform the steps of applying, determining, identifying, comparing, and determining that the machine learning model is validated, amounts to no more than merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, as discussed in MPEP § 2106.05(f). The “model validator” is described as a combination of hardware and software components including processing circuitry (e.g., processor), memory, and a network interface, which are used to apply input data to produce output data and perform subsequent steps such as determining, identifying, and comparing. However, these steps, as previously noted, recite mental processes. Thus, the model validator is directed to a form of merely implementing those mental processes via instructions on a generic computer. Accordingly, the claim merely invokes generic computer components as a tool to perform the judicial exception. This does not add a meaningful limitation to the overall process of validating model performance based on statistical analysis and, therefore, fails to integrate the abstract idea into a practical application.
wherein the first test data is accessed via a network that communicatively couples the model validator and at least one data source; … wherein the second test data is accessed from the at least one data source via the network; (First, these limitations merely define data gathering steps—i.e., accessing data from an external source over a network, that are necessary for performing the claimed process. As explained in MPEP § 2106.05(g), such data gathering is considered insignificant extra-solution activity when it is used in a claim simply to obtain data for use in a process that recites a judicial exception. Accordingly, this aspect of the limitation does not add meaningful limitation to the claim. Second, the broader recitation that the network communicatively couples the “model validator” and the data source merely describes a generic computer environment, in which a computer system communicates with an external data source over a network. This constitutes the use of conventional computer components to perform generic computer functions, which does not add a meaningful limitations representing applying generic computer components to perform generic computer functions. As such, they do not meaningfully limit the claim and do not integrate the judicial exception into a practical application.)
The limitations reciting that “the second time period is after and does not overlap with the first time period” and that “the first test data is different from the second test data” merely establish that the model is applied to data collected at different time periods or from different datasets. These are contextual constraints that do not result in a technical improvement or integrate the judicial exception into a practical application. Instead, they constitute insignificant extra-solution activity (i.e., a data gathering, pre/post-solution activity), as discussed in MPEP § 2106.05(g). Accordingly, they do not impose meaningful limitation to the abstract idea or integrate the judicial exception into a practical application.
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
As explained above under Step 2A, Prong Two, the additional elements, such as the limitation reciting “applying the machine learning model to first and second test data,” merely invoke generic computer components to perform an existing process. The high-level recitation of applying a ML model to input data reflects a conventional use of known computer components. This does not meaningfully limit the abstract idea or add any inventive concept to the process of validating the model’s performance based on the statistical analysis on the output of the model. Mere instructions to implement an abstract idea on a computer cannot provide an inventive concept. See MPEP § 2106.05(f).
Furthermore, the recitation that the model validator comprises a processing circuitry, a memory, and a network interface configured to perform the claimed steps merely defines a generic computer components at a high level of generality. These components are standard elements of a conventional computing system and, as such, amount to no more than the use of generic computer component to perform the abstract idea. Accordingly, these elements does not constitute an inventive concept. See MPEP § 2106.05(f).
Additionally, the claim recites insignificant extra-solution activity such as “accessing the first and second test data via a network,” which reflects a routine data access function. Courts have recognized that functions such as “receiving or transmitting data over a network” and “storing and retrieving information in memory” are well‐understood, routine, and conventional functions when claimed in a generic manner, as is the case here. Therefore, this element remains an insignificant extra-solution activity and does not contribute an inventive concept. See MPEP § 2106.05(d).
Even when evaluating the additional elements individually and in combination with the judicial exception, the claim as a whole does not amount to significantly more than the abstract idea. The additional elements — including the recitation of a model validator to apply machine learning model and access testing data with different time periods, merely recite generic computing components and routine data collection operations. These elements do not impose any meaningful limits on the abstract idea, do not improve the functioning of a computer or other technology, and do not amount to an inventive concept under MPEP § 2106.05(a)-(h).
Therefore, claim 1 does not recite patent-eligible subject matter.
Regarding Previously Presented Claim 4,
Step 2A Prong 1: Claim 4, which incorporates the rejection of claim 1, recites further limitation such as:
wherein each of the first high scores cluster and the second high scores cluster is a rightmost portion of the respective score distribution. (This limitation is part of the abstract idea recited in claim 1. Claim 4 merely defines the identified high scores clusters as the rightmost portion of the distribution. Accordingly, identifying the high score cluster location relative to a low score distribution based on the ordering and/or analyzing a plotted score distribution is directed to a process that can be performed in the human mind including observation, evaluation, judgment or opinion. See MPEP § 2106.04(a)(2)(III).)
Step 2A Prong 2: The claim does not recite additional element that integrates the judicial exception into a practical application.
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception.
Therefore, claim 4 is ineligible.
Regarding Previously Presented Claim 5,
Step 2A Prong 1: Claim 5, which incorporates the rejection of claim 1, recites further limitation such as:
sampling from each of the first high scores cluster and the second high scores cluster in order to obtain a first sample and a second sample, wherein the compared at least a portion of the first and second high scores clusters includes the first sample and the second sample. (This limitation is part of the abstract idea recited in claim 1. Claim 5 merely involves identifying representative data points from each of the respective high scores clusters as samples to perform a comparison between the respective samples. Accordingly, deriving samples from high score clusters and performing comparison is part of the statistical data analysis that can be performed in the human mind with physical aid (e.g., pen and paper or a slide rule). See MPEP § 2106.04(a)(2)(III).)
Step 2A Prong 2: The claim does not recite additional element that integrates the judicial exception into a practical application.
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception.
Therefore, claim 5 is ineligible.
Regarding Original Claim 7,
Step 2A Prong 1: Claim 7, which incorporates the rejection of claim 1, recites further limitation such as:
determining a recall for each of the first run and the second run; and comparing the recall for the first run with the recall for the second run in order to determine whether the recall has decreased more than a threshold between the first run and the second run, wherein the machine learning model is determined as not validated when the recall has decreased more than the threshold between the first run and the second run. (These limitations are part of the abstract idea recited in claim 1. Claim 7 merely adds a recall metric that is used to assess the output of the ML model for the first and second dataset. Thus, the process of calculating the respective recall values to perform statistical analysis and comparison on the model’s output is part of the abstract idea of validating the performance of the machine learning model. Additionally, comparing the respective recall values for each run represents an act of evaluation. Accordingly, the process of calculating recall values associated with the first and second runs and comparing the difference to a threshold, to analyze the change in performance of the model, encompasses the mental processes. These steps represents mental processes (involving observations, judgments, evaluations, and opinions) that are implementable in a human mind, with aid of pen and paper. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2: The claim does not recite additional element that integrates the judicial exception into a practical application.
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception.
Therefore, claim 7 is ineligible.
Regarding Original Claim 8,
Step 2A Prong 1: Claim 8, which incorporates the rejection of claim 7, recites further limitation such as:
determining, for each of the first run and the second run, at least one standard deviation for the respective recall of the run, wherein it is determined whether the recall has decreased more than a threshold between the first run and the second run based on the determined standard deviations. (These limitations are part of the abstract ideas recited in claims 1 and 7. Claim 8 merely defines the determination of standard deviations for recall values and a comparison of whether the recall has decreased more than a threshold between runs. Thus, the process involves using a general recall metric to calculate the respective recall values associated with each respective run, and further analyze the output data to calculate respective standard deviations associated with each respective recall value. This limitation involves a mathematical calculation, namely statistical measure to assess the confidence or variability in the recall metric, and then it’s used to help decide whether a decrease in recall is statistically significant (i.e., more than a threshold difference, accounting for variance). This is directed to the mental process that can be practically performed in the human mind with physical aid (e.g., pen and paper). See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2: The claim does not recite additional element that integrates the judicial exception into a practical application.
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception.
Therefore, claim 8 is ineligible.
Regarding Original Claim 9,
Step 2A Prong 1: Claim 9, which incorporates the rejection of claim 1, recites further limitation such as:
determining a precision for each of the first run and the second run; and comparing the precision for the first run with the precision for the second run in order to determine whether the precision has decreased more than a threshold between the first run and the second run, wherein the machine learning model is determined as not validated when the precision has decreased more than the threshold between the first run and the second run. (These limitations are part of the abstract idea recited in claims 1. Claim 9 merely defines the comparison of statistical evaluation metrics, specifically, the precision of a machine learning model’s outputs across two different runs, to determine whether the model’s performance has degraded beyond a threshold. This process that applies a general precision metric formula to calculate respective precision values for the first and second runs is directed to a series of analyses and decision-making processes, all of which are mental processes (involving observations, judgments, evaluations, and opinions) that are implementable in a human mind, with aid of pen and paper. Additionally, taking a difference between two precision values, and compare this change against a threshold to further determine a change in performance that is greater than a threshold defines an act of evaluating information and decision-making processes. Therefore, these limitation recite the use of standard mathematical/statistical measures and comparisons to evaluate the performance of the model, which constitute a mental processes and/or mathematical concepts, both of which fall within the judicial exceptions. Moreover, the condition that the model is “determined as not validated” based on whether the precision decreases beyond a threshold is a decision-making step based on the analysis identified above, and not computer functioning step that provides technological improvement. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2: The claim does not recite additional element that integrates the judicial exception into a practical application.
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception.
Therefore, claim 9 is ineligible.
Regarding Amended Claim 10,
The claim recites similar limitations as corresponding claim 1. Therefore, the same analysis (subject matter eligibility analysis) that was utilized for claim 1, as described above, is equally applicable to claim 10. The only difference is that claim 1 is drawn to a method, and claim 7 is drawn to A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute a process. The recitation of “A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute a process” is directed to the applying of mere instructions (on a generic computer) to implement a judicial exception, and hence this additional claim element does not further integrate the judicial exception into a practical application, nor does it add significantly more than the judicial exception, alone or in combination with other elements in the claim. See MPEP 2106.05(f).
Therefore, claim 10 is ineligible.
Regarding Amended Claim 11,
The claim recites similar limitations as corresponding claim 1. Therefore, the same analysis (subject matter eligibility analysis) that was utilized for claim 1, as described above, is equally applicable to claim 11. The only difference is that claim 1 is drawn to a method, and claim 11 is drawn to a processing system.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
Claim 11 further recites: A system for machine learning model validation, comprising: a processing circuitry; and a memory, the memory containing instructions that, when executed by the processing circuitry, … (These additional elements merely describe generic computer components and/or computer instructions to perform the method. Merely adding a generic computer, generic computer components, or a programmed computer to perform generic computer functions does not automatically overcome an eligibility rejection. See MPEP § 2106.05(f).)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As explain above in Step 2A, Prong Two, the additional elements merely describe generic computer components that are configured to perform the aforementioned abstract idea. Mere instruction to apply the exception on computer cannot provide an inventive concept.
Therefore, claim 11 is ineligible.
Regarding previously presented Claim 14,
The claim recites similar limitations as corresponding claim 4. Therefore, the same subject matter eligibility analysis including (abstract idea) that was utilized for claim 4, as described above, is equally applicable to claim 14.
Therefore, claim 14 is not patent-eligible subject matter, for the same reasons provided in Claim 4.
Regarding previously presented Claim 15,
The claim recites similar limitations as corresponding claim 5. Therefore, the same subject matter eligibility analysis including (abstract idea) that was utilized for claim 5, as described above, is equally applicable to claim 15.
Therefore, claim 15 is not patent-eligible subject matter, for the same reasons provided in Claim 5.
Regarding Original Claim 17,
The claim recites similar limitations as corresponding claim 7. Therefore, the same subject matter eligibility analysis including (abstract idea) that was utilized for claim 7, as described above, is equally applicable to claim 17.
Therefore, claim 17 is not patent-eligible subject matter, for the same reasons provided in Claim 7.
Regarding Original Claim 18,
The claim recites similar limitations as corresponding claim 8. Therefore, the same subject matter eligibility analysis including (abstract idea) that was utilized for claim 8, as described above, is equally applicable to claim 18.
Therefore, claim 18 is not patent-eligible subject matter, for the same reasons provided in Claim 8.
Regarding Original Claim 19,
The claim recites similar limitations as corresponding claim 9. Therefore, the same subject matter eligibility analysis including (abstract idea) that was utilized for claim 9, as described above, is equally applicable to claim 19.
Therefore, claim 19 is not patent-eligible subject matter, for the same reasons provided in Claim 9.
Regarding Previously Presented Claim 22,
Step 2A Prong 1: Claim 22, which incorporates the rejection of claim 11, recites further limitation such as:
wherein the determine, based on the comparison, whether the machine learning model is validated, is based solely on the compare of the at least the portion of the first and second high scores clusters. (This limitation is part of the abstract idea identified in claim 11. Claim 22 merely defines that the determination of whether the ML model is validated is based solely on the comparison of at least portion of the first and second high score clusters. This merely emphasis that the evaluation of the model is based on the act of comparison, which has been determined to be the abstract idea of a mental process. Thus, the concept of evaluating a model’s performance using statistical analysis of the model’s outputs is a process that would cover the performance in the human mind with physical aid (e.g., pen and paper). The act of basing a validation decision “solely” on a comparison between high score clusters involves evaluation and decision making process that can be established in the human mind and/or with aid of pen and paper. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2: The claim does not recite additional element that integrates the judicial exception into a practical application.
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception.
Therefore, claim 22 is ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1, 4, 10-11, 14, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Goldszmidt et al. (Pub. No.: US 20210224687 A1) in view of Raz et al., (Pub. No.: US 20200242505 A1), further in view of Moghadam et al., (Pub. No.: US 20220138504 A1), further in view of Harris et al., (Pub. No.: US 20030187584 A1), and further in view of Forman et al., (Pub. No.: US 8744987 B1).
Regarding Amended Claim 1,
Goldszmidt discloses the following:
A method for validating a machine learning model, comprising: (Goldszmidt, [0045] “…FIG. 1 shows a block diagram of a machine-learning-based processing environment 100 according to some embodiments of the invention. A machine-learning model controller 105 can be configured to train, avail, evaluate and/or update a machine-learning model. The machine-learning model may be trained, availed evaluated, updated and/or used for one or more clients, associated with a first client device 110 a and a second client device 110 a.. …”)
applying, by a model validator, a machine learning model to first test data observed over a first time period to generate a first plurality of outputs and a first plurality of scores associated with confidence levels of the first plurality of outputs, (Goldszmidt, [0031] “A first training input data set that includes a plurality of first input data elements can be accessed. …, Each of the plurality of first input data elements can be multi-dimensional. … A first accuracy metric can be determined that characterizes a first performance of the machine-learning model using a first testing data set. The first testing data set can include at least part of the first training input data set and at least part of the corresponding label data set; …” [0045] “FIG. 1 shows a block diagram of a machine-learning-based processing environment 100 according to some embodiments of the invention. A machine-learning model controller 105 can be configured to train, avail, evaluate and/or update a machine-learning model. The machine-learning model may be trained, availed evaluated, updated and/or used for one or more clients, associated with a first client device 110 a and a second client device 110 a.” [0016] “For each model, one or more metrics can be generated based on the predictions, with the metric(s) characterizing a degree to which the classifier could accurately and/or confidently predict to which input data set individual data elements correspond.” [0028] “Each input data set of the multiple input data sets can correspond to a distinct time period relative to the time period associated with each other input data set of the multiple input data sets. For each input data set of the multiple input data sets, a classification metric is determined that characterizes a degree to which a classifier accurately and/or confidently classified ….”) wherein the first test data is accessed via a network that communicatively couples the model validator and at least one data source; (Goldszmidt, [0047] “Machine-learning model controller includes an input-data detector 120 that receives and/or retrieves input data that includes a set of input data elements. …, An input data element may be collected (for example) by detecting user inputs received at a user device 115 (or client device 110), detecting data collected by a sensor of a user device 115, extracting information from a communication received from a user device 115 (e.g., representing a webpage request or webpage interaction), etc.” [0078] “Transceiver subsystem 208 can allow electronic device 200 to transmit signals, receive signals and/or communicate wirelessly with various electronic devices. Transceiver subsystem 208 can include a component, such as an antenna and supporting circuitry to enable data communication over a wireless medium, e.g., using near-field communication (NFC), Bluetooth Low Energy, Bluetooth® (a family of standards promulgated by Bluetooth SIG, Inc.), Zigbee, Wi Fi (IEEE 802.11 family standards), or other protocols for wireless data communication.”)
applying, by the model validator, the machine learning model to second test data observed over a second time period to generate a second plurality of outputs and a second plurality of scores associated with confidence levels of the second plurality of outputs, wherein the second time period is after and does not overlap with the first time period, wherein the first test data is different from the second test data, wherein the second test data is accessed from the at least one data source via the network; (Goldszmidt, [0011] “The trained machine-learning model can be used to process other input data (e.g., production input data), which may be multi-dimensional.” [0031] “A second accuracy metric can be determined that characterizes a second performance of the machine-learning model using a second testing data set. The second testing data set can include the second input data set and a second corresponding training label data set.” [0149] “At block 710, a second input data set that includes second input data elements can be accessed. …, The second input data elements may have been collected after the first input data elements, at a different device as compared to the first input data elements and/or in association with a different user as compared to the first input data elements.” [0150] “At block 715, a first accuracy metric that characterizes a performance of the machine-learning model in association with the first training data set can be determined. At block 720, a second accuracy metric that characterizes a performance of the machine-learning model in association wit