DETAILED ACTION
This non-final office action is responsive to application 18/623,622 as submitted on April 1st 2024.
Claim status is currently pending and under examination for claims 1-20 of which independent claims are 1, 14 and 20.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Independent Claims 1, 14 and 20
Step 2A Prong One: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, independent claim 1, under the broadest reasonable interpretation, recites the following limitations that are abstract ideas:
selecting, by the computing system and based on configuration settings, a process to perform on the sequence of model output data, wherein the process is selected from a plurality of available processes; (mental process)
performing the selected process, by the computing system and based on at least a portion of the sequence of model output data, to generate information about performance of the model over time; (mental process)
The “selecting” step involves identifying a process to perform on model output data based on configuration settings which amounts to no more than observations, evaluations, and judgments that can be performed in the human mind or with the use of a physical aid (e.g., pen and paper). The claim recites the step of selecting a process at a high degree of generality, thus the step is not required to have any specific level of complexity that would preclude the step from being mental processes. Therefore, the “selecting” step is considered to be mental processes, see MPEP § 2106.04(a)(2)(III).
The “performing” step involves performing a selected process and using model output data to generate information about model performance. Performing a selected process and generating performance information amounts to no more than observations, evaluations, and judgments that can be performed in the human mind or with the use of a physical aid (e.g., pen and paper). Many artificial intelligence techniques and processes can be practically performed as mental processes, typically with a physical aid, e.g. decision trees. The claim recites the step of “performing the selected process” at a high degree of generality, thus the step is not required to have any specific level of complexity that would preclude the step from being mental processes. Therefore, the “performing” step is considered to be mental processes, see MPEP § 2106.04(a)(2)(III).
Therefore, the independent claim recites a judicial exception. Independent claims 14 and 20 recite similar limitations corresponding to claim 1, therefore the same subject matter eligibility analysis is applied.
Step 2A Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the judicial exception recited above is not integrated into a practical application. The claims recite the following additional elements, but these additional elements are not sufficient to integrate the judicial exception into a practical application:
capturing, by a computing system, a sequence of model output data generated by a model, wherein the sequence of model output data includes information about a plurality of predictions made by the model, (MPEP § 2106.05(g) necessary data gathering and insignificant extra-solution activity to the judicial exception)
and wherein each prediction in the plurality of predictions is generated by the model in response to a different set of model input data; (MPEP § 2106.05(f) mere instructions to implement an abstract idea on a computer, or generally links exception to a technological environment)
selecting, by the computing system and based on configuration settings, a process to perform on the sequence of model output data, wherein the process is selected from a plurality of available processes; (MPEP § 2106.05(f) mere instructions to implement an abstract idea on a computer, or generally links exception to a technological environment)
performing the selected process, by the computing system and based on at least a portion of the sequence of model output data, to generate information about performance of the model over time; (MPEP § 2106.05(f) mere instructions to implement an abstract idea on a computer, or generally links exception to a technological environment)
and sending, by the computing system and to a downstream system, control signals to modify operation of the downstream system based on the information about performance of the model over time. (MPEP § 2106.05(g) necessary data gathering and insignificant extra-solution activity to the judicial exception)
processing circuitry and a storage device, wherein the processing circuitry has access to the storage device and is configured to (claim 14) (MPEP § 2106.05(f) mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea)
Non-transitory computer-readable media comprising instructions that, when executed, cause processing circuitry of a computing system to (claim 20) (MPEP § 2106.05(f) mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea)
The “capturing” step amounts to mere data gathering and is recited at a high level of generality, thus adding insignificant extra-solution activity to the judicial exception – see MPEP § 2106.05(g). Under MPEP § 2106.05(d), such additional elements have been found by the courts to not integrate a judicial exception into a practical application.
The “wherein each prediction …” step is recited at a high-level of generality such that the limitation amounts to no more than mere instructions to “apply” the judicial exception on a computer. It can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of computers, see MPEP § 2106.05(f).
The “selecting” step requires a “computing system” to select a process. The computing system is used to apply the recited judicial exception without placing any limitation on how the computing system operates. The limitation amounts to mere instructions to “apply” the judicial exception on a computer. It can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of computers, see MPEP § 2106.05(f).
The “performing” step requires a “computing system” to perform a selected process. The computing system is used to apply the recited judicial exception without placing any limitation on how the computing system operates. The limitation amounts to mere instructions to “apply” the judicial exception on a computer. It can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of computers, see MPEP § 2106.05(f).
The “sending” step amounts to mere data gathering and is recited at a high level of generality, thus adding insignificant extra-solution activity to the judicial exception – see MPEP § 2106.05(g). Under MPEP § 2106.05(d), such additional elements have been found by the courts to not integrate a judicial exception into a practical application.
The remaining additional elements are recited at a high-level of generality such that they amount to no more than mere instructions to “apply” an exception using a generic component. Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea, see MPEP § 2106.05(f).
Therefore, the above limitations do not integrate the judicial exception into a practical application.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. The claims do not include additional elements that are sufficient for the claims to amount to significantly more than the judicial exception.
In regards to the “capturing” and “sending” steps, these steps add insignificant extra-solution activity. An extra-solution activity is a well-understood, routine and conventional (WURC) activity per MPEP § 2106.05(d)(II), “the courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. i. Receiving or transmitting data over a network, e.g., using the Internet to gather data.” The “capturing” and “sending” steps do not integrate the judicial exception into a practical application and do not amount to significantly more.
In regards to the “computing system” in the “selecting” step, the limitations are recited so generically such that they amount to no more than mere instructions to “apply” the judicial exception on a computer using generic computer components. Mere instructions to apply a judicial exception cannot provide an inventive concept. See MPEP § 2106.05(f).
In regards to the “computing system” in the “performing” step, the limitations are recited so generically such that they amount to no more than mere instructions to “apply” the judicial exception on a computer using generic computer components. Mere instructions to apply a judicial exception cannot provide an inventive concept. See MPEP § 2106.05(f).
In regards to the “wherein each prediction …” step and the remaining additional elements, the limitations are recited so generically such that they amount to no more than mere instructions to “apply” the judicial exception on a computer using generic computer components. Mere instructions to apply a judicial exception cannot provide an inventive concept. See MPEP § 2106.05(f).
Therefore, independent claims 1, 14 and 20 are not patent eligible.
Dependent Claims 2-13 and 15-19
The remaining dependent claims being rejected do not recite additional elements, whether considered individually or in combination, that are sufficient to integrate the judicial exception into a practical application or amount to significantly more than a judicial exception.
Claim limitation
Examiner analysis
2 and 15. The method of claim 1, wherein the model is a first model,
wherein the model input data is first model input data,
wherein the sequence of model output data is a sequence of first model output data,
wherein the plurality of predictions is a first plurality of predictions,
and wherein the method further comprises: capturing, by the computing system, a sequence of second model output data generated by a second model,
wherein the sequence of second model output data includes information about a second plurality of predictions made by the second model,
wherein each prediction in the second plurality of predictions is generated by the second model in response to a different set of second model input data;
and performing the selected process, by the computing system and based on at least a portion of the sequence of second model output data, to generate information about performance of the second model over time.
This is merely additional information about one or more previously identified mental processes.
The “performing the selected process” step is a mental process akin to a human evaluation/judgment/observation.
3 and 16. The method of claim 2, wherein sending the control signals includes: sending control signals to modify operation of the downstream system further based on the information about performance of the second model over time.
This is merely additional information about one or more previously identified mental processes.
4 and 17. The method of claim 2, wherein the downstream system is a first downstream system,
and wherein sending the control signals includes: sending control signals to modify operation of a second downstream system based on the information about performance of the second model over time.
This is merely additional information about one or more previously identified mental processes.
5 and 18. The method of claim 1, wherein the selected process includes assessing accuracy of the model, and wherein the method further comprises: sending, by the computing system and to a business unit computing system, alerts about model inaccuracies.
This is merely additional information about one or more previously identified mental processes.
6 and 19. The method of claim 1, wherein the selected process includes assessing accuracy of the model, and wherein sending the control signals includes: sending control signals to model retraining infrastructure to cause the model retraining infrastructure to retrain the model.
This is merely additional information about one or more previously identified mental processes.
7. The method of claim 1, wherein the selected process includes performing analytics on the model output data, wherein the method further comprises: sending, by the computing system and to a business unit computing system, near-real time business intelligence reports.
This is merely additional information about one or more previously identified mental processes.
8. The method of claim 1, wherein the selected process includes performing analytics on the model output data, and wherein sending the control signals includes: sending control signals to a downstream computing system that responds to the control signals by modifying operation of a production system.
This is merely additional information about one or more previously identified mental processes.
9. The method of claim 8, wherein sending control signals to the computing system further includes: enabling the downstream computing system to cause the production system to change how the production system performs at least one of: monitoring for fraud, fulfilling online sales orders, processing loans, processing loan applications, or selecting an advertisement.
This is merely additional information about one or more previously identified mental processes.
10. The method of claim 1, wherein the selected process includes monitoring health of the model, and wherein performing the selected process includes: identifying an underperforming aspect of the model.
This is a mental process akin to a human evaluation/judgment/observation.
11. The method of claim 10, wherein sending the control signals includes: sending control signals to model remediation infrastructure to cause the model remediation infrastructure to remediate the underperforming aspect of the model.
This is merely additional information about one or more previously identified mental processes.
12. The method of claim 1, wherein the selected process includes performing load balancing of resources used by a production system, and wherein sending the control signals includes: sending control signals to adjust, based on predictions made by the model, allocations of resources used by the production system.
This is merely additional information about one or more previously identified mental processes.
13. The method of claim 1, wherein the selected process includes performing load balancing of resources used by the computing system, and wherein sending the control signals includes: sending control signals to adjust, based on predictions made by the model, allocations of resources used by the computing system.
This is merely additional information about one or more previously identified mental processes.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
The following are the references relied upon in the rejections below:
Mckay (US 20250139706 A1)
Claims 1-11 and 14-20 are rejected under 35 U.S.C. 102a(2) as being anticipated by Mckay.
Regarding Claim 1, Mckay teaches:
A method comprising ([Abstract] “method includes accessing a predictive performance metric of a machine learning model”):
capturing, by a computing system, a sequence of model output data generated by a model ([0065] “the drift detector 206 monitors the predictive performance metrics of the deployed model over time as new data comes in (e.g., streaming data 214). This includes accuracy metrics like error rate.”
[0004] “Drift means that the model provides predictions with lower accuracy than during the training period.”
See Figure 2 depicting drift detector 206 (‘computing system’) is part of a monitoring system.
A drift detector (‘computing system’) monitors predictive performance metrics (accuracy, error rate) of a deployed model. The deployed model uses streamed data to make predictions and the drift detector monitors the accuracy of the predictions, therefore the drift detector is constantly capturing a sequence of predictions (‘sequence of model output data’) generated by the deployed model.),
wherein the sequence of model output data includes information about a plurality of predictions made by the model ([0078] “the drift detector 206 uses performance metrics like accuracy/error rate and analyzes the data distribution over time windows to quantify changes in performance.”
[0065] “The drift detector 206 analyzes these performance metrics using a concept drift detection algorithm like ADWIN. ADWIN works by dividing the performance data into windows and comparing the average performance between windows”
A drift detector divides performance metrics into time windows to quantify changes in performance. Performance metrics are determined by calculating accuracy or error rate of a prediction generated by a deployed model, therefore the predictions generated by the model (‘sequence of model output’) must include time data (‘information about a plurality of predictions’) since performance metrics derived from predictions are divided into time windows.),
and wherein each prediction in the plurality of predictions is generated by the model in response to a different set of model input data (([0065] “the drift detector 206 monitors the predictive performance metrics of the deployed model over time as new data comes in (e.g., streaming data 214)”
A drift detector calculates performance metrics of a deployed model over time as new data is streamed, therefore the deployed model uses new streamed data to make predictions (and therefore the new streamed data is a different set of model input data).);
selecting, by the computing system and based on configuration settings, a process to perform on the sequence of model output data ([0108-0114] “the concept drift detection system (implemented in post-deployment system 126) could be applied to machine learning models in many different industries beyond insurance, including: Fraud detection—Monitor for drifts in fraud patterns in credit card transactions, claims, etc. Retrain models to detect new fraud types. Supply chain—Detect changes in product demand, pricing, or delivery times. Adapt inventory and pricing models. Healthcare—Identify shifts in patient health metrics or costs. Update risk and resource allocation models … Financial Services—Identify changes in lending risk or investment patterns”
[0117] “The key is applying automated drift detection and retraining for any predictive models applied to non-stationary data streams, where relationships can evolve over time. This allows the models to stay relevant in dynamic environments across many industries”
[0067-0071] “The frequency of operating the drift detector 206 depends on several factors: Expected rate of change—How quickly are underlying data relationships likely to change? Monitoring should be more frequent for volatile environments. Model criticality—More frequent monitoring needed for models where poor predictions have high impact (e.g. fraud detection). Data volume—Enough data needed between checks to reliably assess performance. May need less frequent checks for low volume data. Computational cost—Frequent monitoring incurs higher compute costs. Need to balance with value of up-to-date models”
Multiple machine learning models can be deployed (see [0031]) and a concept drift detection system (comprised of drift detector (‘computing system’), see Fig. 2), based on the industry a model is developed for, can perform fraud detection, detect changes in product demand, identify shifts in patient health metrics, identify changes in lending risk, or drift detection (processes). Therefore, a concept drift detection system uses the industry of a model to determine which process to perform (fraud detection, changes in product demand), and therefore selecting, by the concept drift detection system (computing system) a process to perform on the sequence of model output data (model predictions).
The drift detector uses the industry of a model to determine how frequently it performs a process (drift detection). Expected rate of change, model criticality, data volume, and computational cost are factors (‘configuration settings’) that determine how frequently the drift detector is operated, therefore selecting based on configuration settings, a process to perform on the sequence of model output data (model predictions).),
wherein the process is selected from a plurality of available processes (A concept drift detection system (comprised of drift detector) can perform fraud detection, detect changes in product demand, identify shifts in patient health metrics, identify changes in lending risk, or drift detection (‘plurality of available processes’).);
performing the selected process, by the computing system and based on at least a portion of the sequence of model output data, to generate information about performance of the model over time ([0065] “the drift detector 206 monitors the predictive performance metrics of the deployed model over time as new data comes in (e.g., streaming data 214). This includes accuracy metrics like error rate.”
[0124] “the post-deployment system 126 could be used for insurance risk modeling. Insurance risk models predict the risk associated with policyholders based on factors like past claims, vehicle type, driving history etc. These models are trained on historical claims data. But risk profiles and claim patterns can change over time, causing concept drift. Without drift detection, an outdated risk model will provide inaccurate risk assessments and quotes as new claims data comes in. The proposed drift detection system would monitor the performance of the deployed risk model on new claims streaming in.”
A drift detector (‘computing system’) monitors predictive performance metrics (accuracy, error rate) of a deployed model. The deployed model uses streamed data to make predictions and the drift detector monitors the accuracy of the predictions over time, therefore the drift detector uses model predictions (‘portion of the sequence of model output data’) to generate information about performance of the deployed model over time (performance metrics over time). A deployed model can be an insurance risk model and a drift detector can monitor the performance of the risk model on new incoming claims data over time (therefore performing a selected process (identify changes in lending risk, see [0114]) to generate information about model performance).);
and sending, by the computing system and to a downstream system, control signals to modify operation of the downstream system based on the information about performance of the model over time (The Examiner interprets “control signals” according to its broadest reasonable interpretation (BRI) in view of the Applicant’s specification as encompassing a request. The Examiner notes that the applicant provides no meaningful definition of this term in their specification. This interpretation is consistent with the illustrative descriptions in the Applicant’s specification at [0095], (see excerpt below).
Applicant’s written description at [0095]: “Consumption framework 140 may send control signals to control one or more downstream systems (304). Specifically, monitoring platform 161 may send control signals to the downstream system, instructing the apparatus to perform a specific operation. In one example, monitoring platform 161 outputs a series of signals to a downstream system, such as model retraining infrastructure 191. Model retraining infrastructure 191 receives the signals and determines that the signals include instructions for retraining model 121A. Model retraining infrastructure 191 interacts with model 121A to retrain model 121A. Accordingly, consumption framework 140 controls the operation of a downstream system (i.e., model retraining infrastructure 191) to cause model 121A to be retrained.”
[0123] “If the drift detector 316 detects a concept drift (based on the user feedback), it can request retraining the model from the machine learning platform 122.”
[0120] “The deployment system 124 handles deploying machine learning models trained by the machine learning platform 122 so they can be used to generate predictions.”
A drift detector (‘computing system’) sends a request (‘control signals’) to machine learning platform (‘downstream system’) so that the platform can retrain a model suffering from concept drift (‘information about performance of the model over time’). See [0065] describing concept drift is detected by using performance metrics of a model over time. A request is sent to the machine learning platform that instructs the platform to perform model retraining, therefore a request is ‘control signals’ that instruct the platform to perform an operation (model retraining). When the platform performs model retraining (the operation), the operation of the platform is modified since the platform was previously not retraining the model.).
Regarding Claims 2 and 15, Mckay teaches:
The method of claim 1, wherein the model is a first model ([0065] “the drift detector 206 monitors the predictive performance metrics of the deployed model over time as new data comes in (e.g., streaming data 214). This includes accuracy metrics like error rate.”),
wherein the model input data is first model input data (A deployed model (‘first model’) uses streaming data (‘first model input data’) as input data.),
wherein the sequence of model output data is a sequence of first model output data (The deployed model uses streaming data to make predictions and the drift detector monitors the accuracy of the predictions, therefore the drift detector is constantly capturing a sequence of predictions (‘sequence of first model output data’) generated by the deployed model.),
wherein the plurality of predictions is a first plurality of predictions (A drift detector is constantly capturing predictions (‘first plurality of predictions’) generated by the deployed model using streaming data.),
and wherein the method further comprises: capturing, by the computing system, a sequence of second model output data generated by a second model ([0124] “the post-deployment system 126 could be used for insurance risk modeling. Insurance risk models predict the risk associated with policyholders based on factors like past claims, vehicle type, driving history etc. These models are trained on historical claims data. But risk profiles and claim patterns can change over time, causing concept drift. Without drift detection, an outdated risk model will provide inaccurate risk assessments and quotes as new claims data comes in.”
See Fig. 2 depicting post-deployment system 126 includes drift detector (‘computing system’).
An insurance risk model (‘second model’) uses claims data to generate risk assessments and quotes (‘sequence of second model output data’).),
wherein the sequence of second model output data includes information about a second plurality of predictions made by the second model ([0124] “A drift detector like ADWIN would analyze the model's accuracy over time windows and detect if a significant drop occurs, indicating potential drift. It would generate a warning and report detailing changes in claims data distribution, model performance metrics, etc.”
A drift detector monitors an insurance risk model’s accuracy over time using time windows, therefore the predictions generated by the model (‘sequence of second model output data’) must include time data (‘information about a second plurality of predictions’) since performance metrics derived from predictions are divided into time windows.),
wherein each prediction in the second plurality of predictions is generated by the second model in response to a different set of second model input data ([0124] “But risk profiles and claim patterns can change over time, causing concept drift. Without drift detection, an outdated risk model will provide inaccurate risk assessments and quotes as new claims data comes in.”
An insurance risk model uses incoming new claims data (‘different set of second model input data’) to determine if risk assessments and quotes (‘each prediction in the second plurality of predictions’) generated by the model are still accurate over time.);
and performing the selected process, by the computing system and based on at least a portion of the sequence of second model output data, to generate information about performance of the second model over time (([0124] “an outdated risk model will provide inaccurate risk assessments and quotes as new claims data comes in. … A drift detector like ADWIN would analyze the model's accuracy over time windows … generate a warning and report detailing changes in claims data distribution, model performance metrics, etc.”
A drift detector (‘computing system’) can monitor the performance of the insurance risk model’s risk assessments and quotes (‘portion of the sequence of second model output data’) over time, therefore performing a selected process (identifying changes in lending risk, see [0114]) to generate information (performance metrics) about model performance over time.).
Regarding Claims 3 and 16, Mckay teaches:
The method of claim 2, wherein sending the control signals includes: sending control signals to modify operation of the downstream system further based on the information about performance of the second model over time ([0124] “A drift detector like ADWIN would analyze the model's accuracy over time windows and detect if a significant drop occurs, indicating potential drift. It would generate a warning and report detailing changes in claims data distribution, model performance metrics, etc. An expert (e.g., user feedback module 210) reviews the report and confirms if a real drift occurred.”
[0089] “In response to detecting a drift, the drift detector 206 generates a drift detection warning to the user feedback module 210 of the human-AI collaboration system 204 and to the client device 106.”
[0105] “If the user confirms the drift is real, they can request the model be retrained on the latest data to update it and maintain accuracy.”
[0107] “the human-AI collaboration system 204 allows for leveraging human expertise in the loop improves accuracy of drift detection and reduces disruptive false detections through confirmation of true drifts”
A drift detector generates and sends a warning and report (‘control signals’) to a human-AI collaboration system (‘downstream system’). Upon receiving user confirmation that a detected drift is real, the human-AI collaboration system sends a model retrain request to machine learning platform 122 (see Figure 2). A drift detector generates a report describing an insurance risk model’s performance metrics over time (‘information about performance of the second model over time’). By sending a warning and report to the collaboration system, the collaboration system can send a model retrain request, therefore the warning and report are ‘control signals’ that modify the operation of the collaboration system (whether the collaboration system sends a request or not).).
Regarding Claims 4 and 17, Mckay teaches:
The method of claim 2, wherein the downstream system is a first downstream system ([0123] “If the drift detector 316 detects a concept drift (based on the user feedback), it can request retraining the model from the machine learning platform 122.”
[0120] “The deployment system 124 handles deploying machine learning models trained by the machine learning platform 122 so they can be used to generate predictions.”
A drift detector sends a request (‘control signals’) to machine learning platform (‘first downstream system’) so that the platform can retrain a model suffering from concept drift.),
and wherein sending the control signals includes: sending control signals to modify operation of a second downstream system based on the information about performance of the second model over time ([0089] “In response to detecting a drift, the drift detector 206 generates a drift detection warning to the user feedback module 210 of the human-AI collaboration system 204 and to the client device 106.”
[0105] “If the user confirms the drift is real, they can request the model be retrained on the latest data to update it and maintain accuracy.”
A drift detector generates and sends a warning and report (‘control signals’) to a human-AI collaboration system (‘second downstream system’). Upon receiving user confirmation that a detected drift is real, the human-AI collaboration system sends a model retrain request to machine learning platform 122 (see Figure 2). A drift detector generates a report describing an insurance risk model’s performance metrics over time (‘information about performance of the second model over time’), see [0124]. By sending a warning and report to the collaboration system, the collaboration system can send a model retrain request, therefore the warning and report are ‘control signals’ that modify the operation of the collaboration system (whether the collaboration system sends a request or not).
Regarding Claims 5 and 18, Mckay teaches:
The method of claim 1, wherein the selected process includes assessing accuracy of the model ([0124] “The proposed drift detection system would monitor the performance of the deployed risk model on new claims streaming in. A drift detector like ADWIN would analyze the model's accuracy over time windows and detect if a significant drop occurs, indicating potential drift.”),
and wherein the method further comprises: sending, by the computing system and to a business unit computing system, alerts about model inaccuracies ([0124] “A drift detector like ADWIN would analyze the model's accuracy over time windows and detect if a significant drop occurs, indicating potential drift. It would generate a warning and report detailing changes in claims data distribution, model performance metrics, etc.”
[0041-0042] “Using concept drift detection strategies like ADWIN to identify significant changes in the performance metrics that likely indicate a drift. Generating warnings and notifications for users when drift is detected to indicate retraining may be necessary.”
[0089] “the drift detector 206 generates a drift detection warning to the user feedback module 210”
[0090] “The user feedback module 210 allows the user to review the warning and report and confirm true drifts that require retraining the model with new data, or reject false drifts by adjusting the drift sensitivity to reduce false detections.”
A drift detector (‘computer system’) generates warnings (‘alerts’) describing changes in model performance metrics (accuracy, see [0078]). The drift detector sends the generated warnings to a user feedback module (‘business unit computing system’) that allows the user to review the warnings. The drift detector generates performance data about an insurance risk model’s ability to generate accurate risk assessments and quotes (see [0124]), therefore the user feedback module that the detector sends warnings to is a ‘business unit computing system’.).
Regarding Claims 6 and 19, Mckay teaches:
The method of claim 1, wherein the selected process includes assessing accuracy of the model ([0040-0041] “Continuously monitoring the performance of deployed models by tracking prediction accuracy/error metrics. Using concept drift detection strategies like ADWIN to identify significant changes in the performance metrics that likely indicate a drift.”),
and wherein sending the control signals includes: sending control signals to model retraining infrastructure to cause the model retraining infrastructure to retrain the model ([0123] “If the drift detector 316 detects a concept drift (based on the user feedback), it can request retraining the model from the machine learning platform 122.”
[0120] “The deployment system 124 handles deploying machine learning models trained by the machine learning platform 122 so they can be used to generate predictions.”
A drift detector sends a model retraining request (‘control signals’) to a machine learning platform (‘model retraining infrastructure’) to have the platform retrain the model the detector detected as having a drift.).
Regarding Claim 7, Mckay teaches:
The method of claim 1, wherein the selected process includes performing analytics on the model output data ([0108-0114] “the concept drift detection system (implemented in post-deployment system 126) could be applied to machine learning models in many different industries beyond insurance, including: Fraud detection—Monitor for drifts in fraud patterns in credit card transactions, claims, etc. Retrain models to detect new fraud types”
[0073-0074] “typical monitoring frequencies could be: High frequency—Daily or weekly for models on rapidly changing data like user behavior, fraud detection.”),
wherein the method further comprises: sending, by the computing system and to a business unit computing system, near-real time business intelligence reports ([0089] “the drift detector 206 also generates a drift report (containing an analysis of what may have caused the drift)”
[0124] “An expert (e.g., user feedback module 210) reviews the report and confirms if a real drift occurred.”
[0020] “If the user determines from the automatically generated report that the detection of a drift has been caused by noise in the data then the user can decrease the detector's sensitivity to drift to reduce false detections in the presence of noise.”
[0048-0052] “Some advantages of the automated approach include: Drifts can be detected sooner compared to periodic human review. This allows taking corrective action faster before model accuracy degrades further. … Allows for 24/7 monitoring versus limited windows for human analysis.”
A drift detector (‘computer system’) generates a drift report (‘business intelligence report’) describing an analysis of what caused a detected drift for a model. The drift detector sends the report to a user feedback module (‘business unit computing system’) for an expert to review the report and determine the cause of the drift. A fraud detection model’s drifts are monitored 24/7 to monitor for changes in fraud patterns, therefore reports are near-real time business intelligence reports since reports are generated by an automated drift detector and are used to capture changes in fraud patterns (and therefore the user feedback module is a ‘business unit computing system’).).
Regarding Claim 8, Mckay teaches:
The method of claim 1, wherein the selected process includes performing analytics on the model output data ([0108-0114] “the concept drift detection system (implemented in post-deployment system 126) could be applied to machine learning models in many different industries beyond insurance, including: Fraud detection—Monitor for drifts in fraud patterns in credit card transactions, claims, etc. Retrain models to detect new fraud types”
[0067-0069] “The frequency of operating the drift detector 206 depends on several factors: Expected rate of change—How quickly are underlying data relationships likely to change? Monitoring should be more frequent for volatile environments. Model criticality—More frequent monitoring needed for models where poor predictions have high impact (e.g. fraud detection).”),
and wherein sending the control signals includes: sending control signals to a downstream computing system that responds to the control signals by modifying operation of a production system ([0123] “If the drift detector 316 detects a concept drift (based on the user feedback), it can request retraining the model from the machine learning platform 122.”
[0120] “The deployment system 124 handles deploying machine learning models trained by the machine learning platform 122 so they can be used to generate predictions.”
A drift detector sends a request (‘control signals’) to machine learning platform (‘downstream computing system’) so that the platform can retrain a fraud detection model having concept drift. Once the machine learning platform retrains the model, a deployment system (‘production system’) deploys the model so that the model can be used to generate predictions. Therefore, the machine learning platform responds to the request (‘control signals’) by causing (‘modifying operation’) the deployment system to deploy the model.).
Regarding Claim 9, Mckay teaches:
The method of claim 8, wherein sending control signals to the computing system further includes: enabling the downstream computing system to cause the production system to change how the production system performs at least one of: monitoring for fraud, fulfilling online sales orders, processing loans, processing loan applications, or selecting an advertisement ([0108-0114] “the concept drift detection system (implemented in post-deployment system 126) could be applied to machine learning models in many different industries beyond insurance, including: Fraud detection—Monitor for drifts in fraud patterns in credit card transactions, claims, etc. Retrain models to detect new fraud types”
[0123] “If the drift detector 316 detects a concept drift (based on the user feedback), it can request retraining the model from the machine learning platform 122.”
[0120] “The deployment system 124 handles deploying machine learning models trained by the machine learning platform 122 so they can be used to generate predictions.”
A drift detector sends a request to machine learning platform (‘downstream computing system’) so that the platform can retrain a fraud detection model having concept drift. Once the machine learning platform retrains the model, a deployment system (‘production system’) deploys the model so that the model can be used to generate predictions. When a fraud detection model is determined to have concept drift, the model is retrained so that the model can detect new fraud types, thereby changing how the deployment system generates predictions of fraud detection (‘monitoring for fraud’), and therefore enabling the machine learning platform to cause the deployment system to change how it monitors for fraud.).
Regarding Claim 10, Mckay teaches:
The method of claim 1, wherein the selected process includes monitoring health of the model (The Examiner interprets “health of the model” according to its BRI in view of the Applicant’s specification as encompassing quantified compute time and required resources of a model. This interpretation is consistent with the illustrative descriptions in the Applicant’s specification at [0045], (see excerpt below).
Applicant’s written description at [0045]: “Also, by performing model performance health checks, particularly across a sequence of model output data 102 collected over a period of time, consumption framework 140 may identify problems with one or more models 121 that might not be otherwise apparent though assessments of the accuracy of the predictions made by a model or through assessments based on a single prediction or a limited period of time. For example, health assessments for model 121 may identify performance, timeliness, and/or resource consumption issues with models 121 that may negatively affect other systems.”
[0154] “the deployment system 124/post-deployment system 126 monitors a performance of the machine learning model. For example, the deployment system 124/post-deployment system 126 intermittently assesses the performance of the machine learning model as new data comes in, such that an updated score can be derived representing the model's most recent performance. … Model performance can also be quantified in terms of compute time and required resources such that if the frequency or type of data being ingested changes causing a drop in efficiency or speed, the user may be alerted to this.”
A deployment system monitors a performance of a machine learning model by quantifying model performance in terms of compute time and required resources. The quantified model performance is used to determine if ingested data causes the model to be less efficient (has resource consumption issues) or causes a drop in speed (timeliness issues), therefore monitoring the quantified compute time and required resources of a model is ‘monitoring health of the model’.),
and wherein performing the selected process includes: identifying an underperforming aspect of the model ([0155] “The deployment system 124/post-deployment system 126 determines whether the performance/accuracy of the machine learning model is acceptable (e.g., above a threshold score). If the deployment system 124/post-deployment system 126 determines that the performance/accuracy of the machine learning model is no longer acceptable, the action system 408 redefines the task at the task system 406 or suggests changes to the training data at dataset ingestion system 404.”
A deployment system compares the quantified model performance (represented as a score, see [0154]) to a threshold score to determine if the performance of a machine learning model is unacceptable (therefore identifying an underperforming aspect (compute time or required resources) of the model).).
Regarding Claim 11, Mckay teaches:
The method of claim 10, wherein sending the control signals includes: sending control signals to model remediation infrastructure to cause the model remediation infrastructure to remediate the underperforming aspect of the model ([0155] “The deployment system 124/post-deployment system 126 determines whether the performance/accuracy of the machine learning model is acceptable (e.g., above a threshold score). If the deployment system 124/post-deployment system 126 determines that the performance/accuracy of the machine learning model is no longer acceptable, the action system 408 redefines the task at the task system 406 or suggests changes to the training data at dataset ingestion system 404. For example, if performance is no longer acceptable, the action system 408 raises an alert to the user 132 through communication means (e.g., email/text), and provide suggestions of the cause of the problem and remedial steps. The action system 408 can also update the model based on the latest data or stop the model from making predictions.”
[0153] “the deployment system 124/post-deployment system 126 continuously monitors a performance of the machine learning model (used by the service application 418) and provides a feedback to the dataset ingestion system 404 and the task system 406 via the action system 408”
See Figure 4 depicting a deployment system sends data to an action system.
When a deployment system determines that model performance is unacceptable, the deployment system sends feedback (‘control signals’) to the action system (‘model remediation infrastructure’) to suggest reasons of model performance issues and perform remedial steps (updating the model or suggesting changes to training data). Therefore, when the deployment system sends feedback to the action system to perform remedial steps, the feedback is ‘control signals’ that cause the action system to update the model to remediate the performance issues (compute time or required resources) of the model.).
Regarding Claim 14, the rejection of claim 1 is incorporated. The difference in scope being:
A computing system comprising processing circuitry and a storage device ([0198] “a computing apparatus comprising: a Processor; and a memory”),
wherein the processing circuitry has access to the storage device and is configured to ([0198] “a memory storing instructions that, when executed by the Processor, configure the apparatus to”).
Regarding Claim 20, the rejection of claim 1 is incorporated. The difference in scope being:
Non-transitory computer-readable media comprising instructions that, when executed, cause processing circuitry of a computing system to ([0207] “a non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to”
[0198] processor).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The following are the references relied upon in the rejections below:
Kolen (US 20200306632 A1)
Claims 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over Mckay in view of Kolen.
Regarding Claim 12, Mckay teaches: The method of claim 1, however Mckay does not teach performing load balancing of resources used by a production system, which is taught by Kolen:
wherein the selected process includes performing load balancing of resources used by a production system ([0042] “The service controller 1080 determines, based on the prediction provided by the service predictor 1070, how to adjust the computing resources in the service pool 1030 such that the service pool 1030 contains a sufficient level of computing resources to execute, process, or perform the tasks”
[0037] “The service pool 1030 includes any servers, virtual machines, microservices, data, and/or other computing resources needed or desired to generate responses to the user interactions performed by the users playing the video game (sometimes collectively referred to herein as “computing resources”). In the example above, when the service pool 1030 receives a request to generate the audio message to be played on the UI device 1010 in response to the user initiating a conversation with the NPC, the request may be processed by a microservice loaded onto a virtual machine and ready for execution, where the microservice is configured to generate such audio messages based on user data (for example, the identity of the user's in-game playable character).”
A service controller uses predictions to adjust computing resources in a service pool. The service pool contains a microservice that processes requests to generate audio messages for execution during active gameplay, therefore the microservice is a ‘production system’ that has its computing resources adjusted (‘performing load balancing of resources’).),
and wherein sending the control signals includes: sending control signals to adjust, based on predictions made by the model, allocations of resources used by the production system (The Examiner interprets “control signals” according to its BRI in view of the Applicant’s specification as encompassing a provided prediction. The Examiner notes that the applicant provides no meaningful definition of this term in their specification. This interpretation is consistent with the illustrative descriptions in the Applicant’s specification at [0095].
[0041] “The service predictor 1070 uses the predictive models 1060 to predict the level of computing resources. For example, the service predictor 1070 may predict how many servers are needed, how many virtual machines or containers are needed, how many of which microservices need to be loaded on which virtual machines or containers, what kind of data need to be loaded on which virtual machines or containers, and the like.”
[0042] “The service controller 1080 determines, based on the prediction provided by the service predictor 1070, how to adjust the computing resources in the service pool 1030 such that the service pool 1030 contains a sufficient level of computing resources to execute, process, or perform the tasks triggered by the predicted number and type of user interactions at a given time. The service controller 1080 may act as a just-in-time inventory system, making sure the microservices and data are available and ready to be used at the right time.”
A service predictor uses predictive models to predict how many microservices need to be loaded on each virtual machine and the data used on each virtual machine. The service predictor sends its prediction to a service controller so that the service controller can adjust computing resources in a service pool to ensure a microservice (‘production system’) that generates audio messages has enough computing resources to execute its task. Therefore, a prediction provided by a service predictor is ‘control signals’ since the prediction causes the service controller to adjust computing resources (‘allocations of resources’) used by the microservice.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the method of Mckay with the computing resource adjustment technique disclosed by Kolen to use a machine learning model to predict computing resource usage. By using a machine learning model to predict computing resource usage, future computing resource usage by production systems can be automatically predicted and adjusted, thereby ensuring that production systems have enough computing resources to execute and perform their tasks.
Regarding Claim 13, Mckay teaches: The method of claim 1, however Mckay does not teach performing load balancing of resources used by a computing system, which is taught by Kolen:
wherein the selected process includes performing load balancing of resources used by the computing system ([0042] “The service controller 1080 determines, based on the prediction provided by the service predictor 1070, how to adjust the computing resources in the service pool 1030 such that the service pool 1030 contains a sufficient level of computing resources to execute, process, or perform the tasks”
[0037] “The service pool 1030 includes any servers, virtual machines, microservices, data, and/or other computing resources needed or desired to generate responses to the user interactions performed by the users playing the video game (sometimes collectively referred to herein as “computing resources”). In the example above, when the service pool 1030 receives a request to generate the audio message to be played on the UI device 1010 in response to the user initiating a conversation with the NPC, the request may be processed by a microservice loaded onto a virtual machine and ready for execution”
A service controller uses predictions to adjust computing resources in a service pool. The service pool contains servers and virtual machines (‘computing systems’) that have their computing resources adjusted (‘performing load balancing of resources’).),
and wherein sending the control signals includes: sending control signals to adjust, based on predictions made by the model, allocations of resources used by the computing system (The Examiner interprets “control signals” according to its BRI in view of the Applicant’s specification as encompassing a provided prediction. The Examiner notes that the applicant provides no meaningful definition of this term in their specification. This interpretation is consistent with the illustrative descriptions in the Applicant’s specification at [0095].
[0041] “The service predictor 1070 uses the predictive models 1060 to predict the level of computing resources. For example, the service predictor 1070 may predict how many servers are needed, how many virtual machines or containers are needed, how many of which microservices need to be loaded on which virtual machines or containers, what kind of data need to be loaded on which virtual machines or containers, and the like.”
[0042] “The service controller 1080 determines, based on the prediction provided by the service predictor 1070, how to adjust the computing resources in the service pool 1030 such that the service pool 1030 contains a sufficient level of computing resources to execute, process, or perform the tasks triggered by the predicted number and type of user interactions at a given time. The service controller 1080 may act as a just-in-time inventory system, making sure the microservices and data are available and ready to be used at the right time.”
A service predictor uses predictive models to predict how many microservices need to be loaded on each virtual machine, the data used on each virtual machine, and the number of servers needed. The service predictor sends its prediction to a service controller so that the service controller can adjust computing resources in a service pool to ensure server and virtual machines (‘computing systems’) have enough computing resources to execute their tasks. Therefore, a prediction provided by a service predictor is ‘control signals’ since the prediction causes the service controller to adjust computing resources (‘allocations of resources’) used by the servers and virtual machines.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the method of Mckay with the computing resource adjustment technique disclosed by Kolen to use a machine learning model to predict computing resource usage. By using a machine learning model to predict computing resource usage, future computing resources used by computing systems can be automatically predicted and adjusted, thereby ensuring that computing systems have enough computing resources to execute and perform their tasks.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Sethi et al. (US 20250021257 A1) teaches performing model refitting after determining if a model’s accuracy meets desired performance metrics.
Schierz et al. (US 20210390455 A1) teaches near-real time model performance monitoring to retrain or replace a model if model performance degradation is detected.
Gaddam (“Advanced Data & Model Drift Detection at Scale”) teaches automatically initiating model retraining after detecting a drift in predicted probability distributions of incoming streaming data.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PEDRO J MORALES whose telephone number is (571)272-6106. The examiner can normally be reached 8:30 AM - 6:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MIRANDA M HUANG can be reached at (571)270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PEDRO J MORALES/Examiner, Art Unit 2124
/MIRANDA M HUANG/Supervisory Patent Examiner, Art Unit 2124
Prosecution Insights