Response to Amendment
This communication is in response to the amendment filed on 3/4/2026. Claims 1-15 are pending.
Claim Objections
The objections to Claims 1 and 15 are withdrawn based on the amendments filed on 3/4/2026.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-15 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Is the Claim to a Process, Machine, Manufacture or Composition of Matter?
Claim 1 recites a computer-implemented method, and Claim 15 recites an apparatus. Thus, the claims are to a method and a machine, which are among the statutory categories of invention.
Step 2A: Prong One: Does the Claim Recite an Abstract Idea?
Independent claim 1 recites:
A computer-implemented method of monitoring performance of a predictive computer-implemented model, PCIM, that is used to monitor the status of a first system, wherein the PCIM receives as inputs observed values for a plurality of features relating to the first system, and the PCIM determines whether to issue status alerts based on the observed values, wherein the method comprises:
obtaining reference information for the PCIM, wherein the reference information for the PCIM comprises a first set of values for the plurality of features relating to the first system in a first time period;
determining a set of reference probability distributions from the first set of values, the set of reference probability distributions comprising a respective reference probability distribution for each of the features that is determined from the values of the respective feature in the first set of values [the examiner finds that the foregoing underlined element recites mathematical concepts, and a mental process because they can be performed by a human using pen and paper];
obtaining operational information for the PCIM, wherein the operational information for the PCIM comprises a second set of values for the plurality of features relating to the first system in a second time period that is after the first time period;
determining a set of operational probability distributions from the second set of values, the set of operational probability distributions comprising a respective operational probability distribution for each of the features that is determined from the values of the respective feature in the second set of values [the examiner finds that the foregoing underlined element recites mathematical concepts, and a mental process because they can be performed by a human using pen and paper];
determining a drift measure for the PCIM representing a measure of drift in performance of the PCIM between the first time period and the second time period, wherein the drift measure is based on a comparison of the set of reference probability distributions and the set of operational probability distributions [the examiner finds that the foregoing underlined element recites mathematical concepts, and a mental process because they can be performed by a human using pen and paper]; and
outputting the drift measure, the output of the drift measure enabling correction of the drift.
Step 2A: Prong Two: Does the Claim Recite Additional Elements That Integrate The Abstract Idea Into a Practical Application?
The elements that are not underlined above are the additional elements (i.e., “a predictive computer-implemented model, PCIM, that is used to monitor the status of a first system, wherein the PCIM receives as inputs observed values for a plurality of features relating to the first system, and the PCIM determines whether to issue status alerts based on the observed values”; “obtaining reference information for the PCIM, wherein the reference information for the PCIM comprises a first set of values for the plurality of features relating to the first system in a first time period”; “obtaining operational information for the PCIM, wherein the operational information for the PCIM comprises a second set of values for the plurality of features relating to the first system in a second time period that is after the first time period”; and “outputting the drift measure, the output of the drift measure enabling correction of the drift”).
The examiner submits that each of the following additional elements does no more than generally link the use of the abstract idea to a particular technological environment or field of use because they are merely an incidental or token addition to the claim that does not alter or affect how the process steps of the abstract idea are performed. The PCIM is a broadly recited, generic computer-implemented mode which amounts to not more than using generic computer hardware to perform mathematical operations. The obtaining steps merely recite receiving of data for use in the abstract idea, and the outputting step merely outputs a result of the abstract idea (it is noted that correcting the drift is not actually required to be performed).
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. For example, there is no indication that the combination of elements improves the functioning of a computer or improves any other technology.
Step 2B: Does the Claim Recite Additional Elements That Amount to Significantly More Than the Abstract Idea?
The examiner submits that the additional elements do not amount to significantly more than the abstract idea for the same reasons discussed above with respect to the conclusion that the additional elements do not integrate the abstract idea into a practical application.
Independent Claim 15 recites the same steps as Claim 1, and is also not patent eligible.
Dependent Claims 2-13 merely recite further details of the mathematical concepts and/or mental process, and are also not patent eligible. Dependent Claim 14 merely recites generic computer hardware for implementing the abstract idea.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-8 and 14-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Raz et al (U.S. Pub. No. 2020/0242505, hereinafter “Raz”) in view of Xu et al (U.S. Pub. No. 2018/0150036, hereinafter “Xu”).
Regarding Claim 1, Raz teaches a computer-implemented method of monitoring performance (Fig. 1, statistical testing 112) of a predictive computer-implemented model, PCIM (Fig. 1, machine learning model 104), that is used to monitor the status of a first system, wherein the PCIM receives as inputs observed values for a plurality of features relating to the first system (paragraphs [0002] and [0019], multiple sensors acquire data from system; paragraphs [0018], machine learning model 104 finds patterns in, makes predictions about, or makes decisions about production data 108), wherein the method comprises: obtaining reference information for the PCIM, wherein the reference information for the PCIM comprises a first set of values for the plurality of features relating to the first system in a first time period (training data 106); determining a set of reference probability distributions from the first set of values, the set of reference probability distributions comprising a respective reference probability distribution for each of the features that is determined from the values of the respective feature in the first set of values (statistical testing 112, paragraph [0020], labels are equated to claimed features); obtaining operational information for the PCIM, wherein the operational information for the PCIM comprises a second set of values for the plurality of features relating to the first system in a second time period that is after the first time period (production data 108); determining a set of operational probability distributions from the second set of values, the set of operational probability distributions comprising a respective operational probability distribution for each of the features that is determined from the values of the respective feature in the second set of values (statistical testing 112, paragraph [0020], labels are equated to claimed features); determining a drift measure for the PCIM representing a measure of drift in performance of the PCIM between the first time period and the second time period, wherein the drift measure is based on a comparison of the set of reference probability distributions and the set of operational probability distributions (paragraphs [0020]-[0021], drift between training data 106 and production data 108 is detected based on confidence-per-label distributions); and outputting the drift measure, the output of the drift measure enabling correction of the drift (results data 114 includes results of statistical testing 112; paragraph [0004], improve ML performance even though data drift has occurred).
Raz does not specifically teach that the PCIM determines whether to issue status alerts based on the observed values. However, Xu teaches that the PCIM determines whether to issue status alerts based on the observed values (paragraphs [0037]-[0038]). It would have been obvious to one skilled in the art before the effective filing date of the invention to include the alerts of Xu in the system of Raz, in order to plan outages instead of reacting to a failure (see Xu, paragraph [0037]), and because data received from a process may demonstrate an inherent non-stationary nature (see Xu, paragraph [0018]).
Regarding Claim 2, Raz in view of Xu teaches everything that is claimed above with respect to Claim 1. Raz further teaches wherein the step of determining the drift measure comprises, for each feature relating to the first system, comparing one or more statistical measures for the reference probability distribution of said feature to one or more statistical measures for the operational probability distribution of said feature (paragraph [0020], confidence-per-label distributions are compared between training data 106 and production data 108; labels are equated to claimed features).
Regarding Claim 3, Raz in view of Xu teaches everything that is claimed above with respect to Claim 2. Raz further teaches wherein the step of comparing comprises, for each feature relating to the first system and for each statistical measure, determining a distance measure for said feature and statistical measure from the value of said statistical measure for the reference probability distribution and the value of said statistical measure for the operational probability distribution (paragraph [0020], Kolmogrorov-Smirnov test determines distance measure between probability distributions).
Regarding Claim 4, Raz in view of Xu teaches everything that is claimed above with respect to Claim 2. Raz further teaches wherein the one or more statistical measures comprises any one or more of: a mean of the probability distribution (Fig. 4, mean), a standard deviation of the probability distribution, a density of the probability distribution, and one or more shape parameters defining the shape of the probability distribution (no patentable weight due to “one or more of”).
Regarding Claim 5, Raz in view of Xu teaches everything that is claimed above with respect to Claim 1. Raz further teaches wherein the first set of values for the plurality of features is a training set of values that was used to train the PCIM, and the first time period is a time period before the PCIM is monitoring the status of the first system (training data 106).
Regarding Claim 6, Raz in view of Xu teaches everything that is claimed above with respect to Claim 5. Raz further teaches wherein: the reference information for the PCIM further comprises reference performance information indicating an expected reliability of the PCIM for the first system based on the training set of values (paragraph [0020], confidence-per-label distribution using training data 106; confidence is equated to claimed performance information); the operational information for the PCIM further comprises operational performance information indicating the operational reliability of the PCIM for the first system in the second time period (paragraph [0020], confidence-per-label distribution using production data 108; confidence is equated to claimed performance information); and the drift measure is further based on a comparison of the reference performance information and the operational performance information (paragraph [0020], data drift between training data 106 and production data 108 determined based on statistical tests).
Raz does not specifically teach the reference performance information indicating an expected reliability of the PCIM in issuing status alerts for the first system, and the operational information for the PCIM further comprises operational performance information indicating the operational reliability of the PCIM in issuing status alerts for the first system (emphasis added). However, Xu teaches issuing status alerts for a system (paragraphs [0037]-[0038]). It would have been obvious to one skilled in the art before the effective filing date of the invention to include the alerts of Xu in the system of Raz, in order to plan outages instead of reacting to a failure (see Xu, paragraph [0037]), and because data received from a process may demonstrate an inherent non-stationary nature (see Xu, paragraph [0018]).
Regarding Claim 7, Raz in view of Xu teaches everything that is claimed above with respect to Claim 1. Raz further teaches wherein the first set of values for the plurality of features is a set of values obtained during use of the PCIM, and the first time period is a time period where the PCIM is monitoring the status of the first system (paragraph [0020], drift may be determined for data from two different time windows in production).
Regarding Claim 8, Raz in view of Xu teaches everything that is claimed above with respect to Claim 7. Raz further teaches wherein: the reference information for the PCIM further comprises reference performance information indicating the reliability of the PCIM for the first system in the first time period (paragraph [0020], confidence-per-label distribution using production data 108; confidence is equated to claimed performance information); the operational information for the PCIM further comprises operational performance information indicating the operational reliability of the PCIM for the first system in the second time period (paragraph [0020], drift may be determined for data from two different time windows in production; confidence-per-label distribution using production data 108; confidence is equated to claimed performance information); and the drift measure is further based on a comparison of the reference performance information and the operational performance information (paragraph [0020], data drift determined based on statistical tests; drift may be determined for data from two different time windows in production).
Raz does not specifically teach the reference information for the PCIM further comprises reference performance information indicating the reliability of the PCIM in issuing status alerts for the first system in the first time period, and the operational information for the PCIM further comprises operational performance information indicating the operational reliability of the PCIM in issuing status alerts for the first system in the second time period (emphasis added). However, Xu teaches issuing status alerts for a system (paragraphs [0037]-[0038]). It would have been obvious to one skilled in the art before the effective filing date of the invention to include the alerts of Xu in the system of Raz, in order to plan outages instead of reacting to a failure (see Xu, paragraph [0037]), and because data received from a process may demonstrate an inherent non-stationary nature (see Xu, paragraph [0018]).
Regarding Claim 14, Raz in view of Xu teaches everything that is claimed above with respect to Claim 1. Raz in view of Xu further teaches a non-transitory computer program product comprising a computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform the method of claim 1 (see Fig. 5 and paragraph [0018] of Raz, and the rejection of Claim 1, above).
Regarding Claim 15, Raz teaches an apparatus for monitoring performance (Fig. 1, statistical testing 112) of a predictive computer-implemented model, PCIM (Fig. 1, machine learning model 104), that is used to monitor the status of a first system, wherein the PCIM receives as inputs observed values for a plurality of features relating to the first system (paragraphs [0002] and [0019], sensors acquire data from system; paragraphs [0018], machine learning model 104 finds patterns in, makes predictions about, or makes decisions about production data 108), wherein the apparatus comprises a processing unit (Fig. 5) is configured to: obtain reference information for the PCIM, wherein the reference information for the PCIM comprises a first set of values for the plurality of features relating to the first system in a first time period (training data 106); determine a set of reference probability distributions from the first set of values, the set of reference probability distributions comprising a respective reference probability distribution for each of the features that is determined from the values of the respective feature in the first set of values (statistical testing 112, paragraph [0020], labels are equated to claimed features); obtain operational information for the PCIM, wherein the operational information for the PCIM comprises a second set of values for the plurality of features relating to the first system in a second time period that is after the first time period (production data 108); determine a set of operational probability distributions from the second set of values, the set of operational probability distributions comprising a respective operational probability distribution for each of the features that is determined from the values of the respective feature in the second set of values (statistical testing 112, paragraph [0020], labels are equated to claimed features); determine a drift measure for the PCIM representing a measure of drift in performance of the PCIM between the first time period and the second time period, wherein the drift measure is based on a comparison of the set of reference probability distributions and the set of operational probability distributions (paragraphs [0020]-[0021], drift between training data 106 and production data 108 is detected based on confidence-per-label distributions); and cause the output the drift measure, the output of the drift measure enabling correction of the drift (results data 114 includes results of statistical testing 112; paragraph [0004], improve ML performance even though data drift has occurred).
Raz does not specifically teach that the PCIM determines whether to issue status alerts based on the observed values. However, Xu teaches that the PCIM determines whether to issue status alerts based on the observed values (paragraphs [0037]-[0038]). It would have been obvious to one skilled in the art before the effective filing date of the invention to include the alerts of Xu in the system of Raz, in order to plan outages instead of reacting to a failure (see Xu, paragraph [0037]), and because data received from a process may demonstrate an inherent non-stationary nature (see Xu, paragraph [0018]).
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Raz in view of Xu and Honda et al (U.S. Pub. No. 2019/0277913).
Regarding Claim 9, Raz in view of Xu teaches everything that is claimed above with respect to Claim 6. Raz does not specifically teach wherein each of the reference performance information and the operational performance information comprise one or more of a true positive rate, a false positive rate, a true negative rate (no patentable weight due to “one or more of”) and a false negative rate. However, Honda teaches in paragraphs [0080]-[0082] that the false positive rate, false negative rate, and true positive rate are used as performance criteria for machine learning models. It would have been obvious to one skilled in the art before the effective filing date of the invention to include the false positive rate, false negative rate, and true positive rate of Honda in the system of Raz, because these rate are used as criteria to determine whether a machine learning model is production worthy and will predict relevant target features (see Honda, paragraphs [0081]-[0082]).
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Raz in view of Xu and Walters et al (U.S. Pub. No. 2020/0012900, hereinafter “Walters”).
Regarding Claim 10, Raz in view of Xu teaches everything that is claimed above with respect to Claim 1. Raz does not specifically teach wherein the method further comprises: obtaining values of one or more further features relating to the first system, the one or more further features comprising any of a presence of a log file for the first system, a warranty status of a component of the first system (no patentable weight due to “any of”), a version of software or firmware used by the first system (no patentable weight due to “any of”); and wherein the drift measure is further based on the values of the one or more further features. However, Walters teaches obtaining values of one or more further features relating to the first system, the one or more further features comprising any of a presence of a log file for the first system (paragraphs [0070] and [0127], data source can include web log data or system logs data); and wherein the drift measure is further based on the values of the one or more further features (Fig. 18 blocks 1802 and 1806, paragraphs [0177] and [0181], received data is used to determine data drift). It would have been obvious to one skilled in the art before the effective filing date of the invention to include the log data of Walters in the system of Raz, because log data is a source of actual data about a system (see Walters, paragraph [0070]).
Claim(s) 11-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Raz in view of Xu and Maughan et al (U.S. Pub. No. 2017/0330109, hereinafter “Maughan”).
Regarding Claim 11, Raz in view of Xu teaches everything that is claimed above with respect to Claim 1. Raz does not specifically teach wherein the method further comprises: analysing the PCIM to identify the plurality of features relating to the first system that are used by the PCIM. However, Maughan teaches in paragraphs [0070]-[0072] analyzing model input data for features that contribute to drift. It would have been obvious to one skilled in the art before the effective filing date of the invention to include the feature analysis of Maughan in the system of Raz, in order to indicate the importance of drifted features (see Maughan, paragraph [0070] and because characteristics of data may drift or change over time (see Maughan, paragraph [0005]).
Regarding Claim 12, Raz in view of Xu teaches everything that is claimed above with respect to Claim 1. Raz does not specifically teach wherein the method further comprises: evaluating the drift measure to identify one or more of the features that have contributed to the value of the drift measure; and analysing the identified one or more features that have contributed to the value of the drift measure to determine corrections to the operation of the PCIM to reduce the drift measure. However, Maughan teaches evaluating the drift measure to identify one or more of the features that have contributed to the value of the drift measure (paragraph [0070], drift indicator provided at feature granularity); and analysing the identified one or more features that have contributed to the value of the drift measure to determine corrections to the operation of the PCIM to reduce the drift measure (paragraph [0070], importance, priority, or ranking of drifted features determined; paragraphs [0071-[0072], complete or partial retraining to account for drift based on drift statistics). It would have been obvious to one skilled in the art before the effective filing date of the invention to include the feature analysis and correction to operations that are taught in Maughan in the system of Raz, in order to indicate the importance of drifted features (see Maughan, paragraph [0070] and because characteristics of data may drift or change over time (see Maughan, paragraph [0005]).
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Raz in view of Xu and Bonissone et al (U.S. Pub. No. 2015/0355901, hereinafter “Bonissone”).
Regarding Claim 13, Raz in view of Xu teaches everything that is claimed above with respect to Claim 1. Raz does not specifically teach wherein the method further comprises: analysing the determined drift measure to estimate a remaining life of the PCIM. However, Raz does disclose determining a drift measure (paragraph [0001]). Further, Bonissone teaches analysing the determined drift measure (Table 1, data drift) to estimate a remaining life of the PCIM (paragraphs [0002] and [0023], forecasting an estimate of a remaining useful life for the model; it is also noted that determining that the model needs to be retrained would include determining that the remaining useful life is zero). It would have been obvious to one skilled in the art before the effective filing date of the invention to include the remaining useful life determination of Bonissone in the system of Raz, because model obsolescence is a major impediment to the success of deployment of models (see Bonissone, paragraph [0001]).
Response to Arguments
Applicant's arguments filed 3/4/2026 have been fully considered but they are not persuasive.
Regarding the 101 rejections, Applicant argues on pages 9-13 that the claims are an improvement to a technical field, and therefore are integrated into a practical application, because the drift measure output by the abstract idea enables correction of drift in a PCIM. However, it is noted that the Claims merely output the drift measure in a generic, unspecified manner, and do not require any actual correction of the drift to be performed. Therefore, the Claims do not include any improvement to the PCIM.
Regarding the 103 rejections, Applicant argues on pages 15-16 that because Raz teaches determination of data drift (i.e., a reduction in accuracy of a model of an underlying system due to changes in the data received from the underlying system over time), that Raz does not teach the drift measure recited in the claims. The Examiner disagrees, because the drift measure of the claims encompasses data drift (see the second paragraph of the Background of the Invention section of Applicant’s Specification as filed, which states that “However, the systems may evolve over time, e.g. there may be changes in aspects of system hardware, and/or there can be subtle drifts or changes in usage patterns, software, firmware, etc. which can cause the accuracy of the predictive model to drift or deteriorate over time. For example, if there are changes to the structure of system data that is input to the predictive model, or if changes in software cause certain key words to change, the predictive model’s performance can be affected” (emphasis added)).
On pages 16-17, Applicant requests clarity regarding the claimed “plurality of features”. The various sensors taught in Raz are an example of the many different types of input data, such as data from various different sensor types, that may be received from an underlying system in Raz (see Raz, paragraph [0019]); these different types of data are equated to the claimed observed values. The input data of Raz is also associated with various labels relating to the underlying system, which are assigned to the input data as appropriate for the underlying system, e.g., data associated with different types or events may have different labels (see e.g. paragraph [0022]). These labels are equated to the plurality of features recited in the claims, because the input data of Raz includes observed values for each of a plurality of labels. Probability distributions are then determined in Raz on a per-label basis (paragraph [0020]).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CYNTHIA L DAVIS whose telephone number is (571)272-1599. The examiner can normally be reached Monday-Friday, 7am to 3pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Shelby A Turner can be reached at (571)272-6334. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CYNTHIA L DAVIS/Examiner, Art Unit 2857
/SHELBY A TURNER/Supervisory Patent Examiner, Art Unit 2857
Prosecution Insights