DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the Application filed on March 28, 2023. Claims 1-20 are pending in the case. Claims 1, 10, and 19 are the independent claims.
This action is non-final.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea (mental steps) without significantly more. This judicial exception is not integrated into a practical application because any additional elements amount to implementing the abstract idea on a generic computer. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Regarding independent claims 1, 10, and 19, and relying on the evaluation flowchart in MPEP 2106:
Step 1 (Is the claim to a process, machine, manufacture, or composition of matter?): Yes. Claim 1 is a system (machine). Claim 10 is a storage medium (article of manufacture). Claim 19 is a method (process).
Step 2a Prong One (Does the claim recite an abstract idea?): Yes. Claims 1, 10, and 19 recite:
determine/determining whether the data stream matches a declarative mapping protocol (a mental process of evaluation/determination);
determine/determining one or more deviation instances in an instance in which the data stream does not match the declarative mapping protocol (a mental process of evaluation/determination);
determine/determining one or more prescriptive actions for the one or more deviation instances (a mental process of evaluation/determination).
Under the broadest reasonable interpretation, these steps may be performed mentally, using mental observation and mental determination, including by a human using a physical aid such as pen and paper, including a human mentally performing observations and mentally performing mathematical calculations, and therefore correspond to the Mental Processes grouping.
Step 2a Prong Two (Does the claim recite additional elements that integrate the judicial exception into a practical application?): No. Claims 1, 10, and 19 and 11 additionally recite:
receive/receiving, from one or more control automation modules, a data stream, wherein the data stream comprises data associated with the one or more control automation module (insignificant extra-solution activity as discussed in MPEP 2106.05(g));
transmit/transmitting the data stream to a gauging and monitoring module (insignificant extra-solution activity as discussed in MPEP 2106.05(g));
implement/implementing, using an intelligence restoration module, the one or more prescriptive actions on the one or more control automation modules (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f));
(in claim 1) a processing device; a non-transitory storage device containing instructions when executed by the processing device, causes the processing device to perform the steps discussed above (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f));
(in claim 10) the computer program product comprising at least one non-transitory computer-readable medium having computer-readable program code portions embodied therein, the computer-readable program code portions comprising executable portions configured to perform the steps discussed above (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f));
a computer-implemented method as discussed above (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f)).
Therefore, in view of the considerations set forth in MPEP 2106.04(d), 2106.05(a)-(c) and (e)-(h), the additional elements as disclosed above alone or in combination do not integrate the judicial exception into a practical application as they are mere insignificant extra solution activity, combined with implementing the abstract idea using generic computer components.
Step 2b (Does the claim recite additional elements that amount to siqnificantly more than the judicial exception): No. Relying on the same analysis as Step 2a Prong Two (see MPEP 2106.05.I.A: Limitations that the courts have found not to be enough to qualify as “significantly more” when recited in a claim with a judicial exception include:…Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, e.g., a limitation indicating that a particular function such as creating and maintaining electronic records is performed by a computer, as discussed in Alice Corp., 573 U.S. at 225-26, 110 USPQ2d at 1984 (see MPEP 2106.05(f));…Simply appending well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception...; Adding insignificant extra-solution activity to the judicial exception, as discussed in MPEP 2106.05(g);…)), claims 1, 10, and 19 do not recite any additional elements that amount to significantly more than the abstract idea. As discussed above, Claims 1, 10, and 19 and 11 additionally recite:
receive/receiving, from one or more control automation modules, a data stream, wherein the data stream comprises data associated with the one or more control automation module (insignificant extra-solution activity as discussed in MPEP 2106.05(g), which can be reevaluated as well-understood, routine, conventional activity of transmitting data over a network as discussed in MPEP 2106.05(d));
transmit/transmitting the data stream to a gauging and monitoring module (insignificant extra-solution activity as discussed in MPEP 2106.05(g), which can be reevaluated as well-understood, routine, conventional activity of transmitting data over a network as discussed in MPEP 2106.05(d));
implement/implementing, using an intelligence restoration module, the one or more prescriptive actions on the one or more control automation modules (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f));
(in claim 1) a processing device; a non-transitory storage device containing instructions when executed by the processing device, causes the processing device to perform the steps discussed above (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f));
(in claim 10) the computer program product comprising at least one non-transitory computer-readable medium having computer-readable program code portions embodied therein, the computer-readable program code portions comprising executable portions configured to perform the steps discussed above (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f));
a computer-implemented method as discussed above (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f)).
The additional elements as discussed above, in combination with the abstract idea, are not sufficient to amount to significantly more than the judicial exception as they are well, understood, routine and conventional activity as disclosed in combination with generic computer functions and components used to implement the abstract idea.
With respect to claims 2, 11, and 20:
Step 2a Prong One: incorporates the rejection of claims claims 1, 10, and 19.
Step 2a Prong Two: the claims additionally recite:
receiving, from one or more control automation modules, a data lake, wherein the data lake comprises data associated with the one or more control automation modules (insignificant extra-solution activity as discussed in MPEP 2106.05(g)); and
transforming the data lake into a data stream (insignificant extra-solution activity as discussed in MPEP 2106.05(g)).
Step 2b: the claims additionally recite:
receiving, from one or more control automation modules, a data lake, wherein the data lake comprises data associated with the one or more control automation modules (insignificant extra-solution activity as discussed in MPEP 2106.05(g), which can be reevaluated as well-understood, routine, conventional activity of transmitting data over a network as discussed in MPEP 2106.05(d)); and
transforming the data lake into a data stream (insignificant extra-solution activity as discussed in MPEP 2106.05(g), which can be reevaluated as well-understood, routine, conventional activity of data gathering and outputting as discussed in MPEP 2106.05(d)).
With respect to claims 3 and 12:
Step 2a Prong One: incorporates the rejection of claims 1 and 10. The claims additionally recite determine, using an intelligence monitoring module, a deviation classification of the graphical representation (mental process of evaluation/determination).
Step 2a Prong Two: the claims additionally recite:
wherein executing the instructions further causes the processing device to/wherein the computer program product further comprises an executable portion configured to: (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f))
in response to transmitting the data stream to the gauging and monitoring module, transmit the data stream to a data distribution analyzer (insignificant extra-solution activity as discussed in MPEP 2106.05(g)),
wherein the data distribution analyzer is configured to create a graphical representation of the data stream, and wherein the graphical representation comprises one or more representations of the data stream (insignificant extra-solution activity as discussed in MPEP 2106.05(g)).
Step 2b: the claims additionally recite:
wherein executing the instructions further causes the processing device to/wherein the computer program product further comprises an executable portion configured to: (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f))
in response to transmitting the data stream to the gauging and monitoring module, transmit the data stream to a data distribution analyzer (insignificant extra-solution activity as discussed in MPEP 2106.05(g), which can be reevaluated as well-understood, routine, conventional activity of transmitting data over a network as discussed in MPEP 2106.05(d)),
wherein the data distribution analyzer is configured to create a graphical representation of the data stream, and wherein the graphical representation comprises one or more representations of the data stream (insignificant extra-solution activity as discussed in MPEP 2106.05(g), which can be reevaluated as well-understood, routine, conventional activity of data gathering and outputting as discussed in MPEP 2106.05(d)).
With respect to claims 4 and 13:
Step 2a Prong One: incorporates the rejection of claims 3 and 12. The claims additionally recite:
determine that the deviation classification is associated with a deviation in performance (mental process of evaluation/determination);
determine a retraining protocol in response to determining that the deviation classification is associated with the deviation in performance (mental process of evaluation/determination).
Step 2a Prong Two: the claims additionally recite:
wherein executing the instructions further causes the processing device to/ wherein the computer program product further comprises an executable portion configured to: (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f))
implement the retraining protocol on the one or more control automation modules (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f)).
Step 2b: the claims additionally recite:
wherein executing the instructions further causes the processing device to/ wherein the computer program product further comprises an executable portion configured to: (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f))
implement the retraining protocol on the one or more control automation modules (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f)).
With respect to claims 5 and 14:
Step 2a Prong One: incorporates the rejection of claims 3 and 12. The claims additionally recite
determine that the deviation classification is associated with a deviation in procedure (mental process of evaluation/determination)
determine an intelligent interpretation protocol in response to determining that the deviation classification is associated with the deviation in procedure (mental process of evaluation/determination).
Step 2a Prong Two: the claims additionally recite:
wherein executing the instructions further causes the processing device to/wherein the computer program product further comprises an executable portion configured to: (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f))
implement the intelligent interpretation protocol on the one or more control automation modules (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f)).
Step 2b: the claims additionally recite:
wherein executing the instructions further causes the processing device to/wherein the computer program product further comprises an executable portion configured to: mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f))
implement the intelligent interpretation protocol on the one or more control automation modules mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f)).
With respect to claims 6 and 15:
Step 2a Prong One: incorporates the rejection of claims 1 and 10. The claims additionally recite determine, using the intelligence restoration module, a drift classification of the data stream (mental process of evaluation/determination).
Step 2a Prong Two: the claims additionally recite wherein executing the instructions further causes the processing device to/wherein the computer program product further comprises an executable portion configured to (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f)).
Step 2b: the claims additionally recite wherein executing the instructions further causes the processing device to/wherein the computer program product further comprises an executable portion configured to (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f)).
With respect to claims 7, and 16:
Step 2a Prong One: incorporates the rejection of claims 6 and 15. The claims additionally recite:
determine that the drift classification is associated with a data drift classification (mental process of evaluation/determination);
determine a range of one or more individual features in response to determining that the drift classification is associated with the data drift classification (mental process of evaluation/determination);
calculate a data feature importance threshold (mental process of evaluation/determination including mental performance of a mathematical calculation).
Step 2a Prong Two: the claims additionally recite:
wherein executing the instructions further causes the processing device to: (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f))
create a suggested data drift model (insignificant extra-solution activity as discussed in MPEP 2106.05(g)).
Step 2b: the claims additionally recite:
wherein executing the instructions further causes the processing device to: (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f))
create a suggested data drift model (insignificant extra-solution activity as discussed in MPEP 2106.05(g), which can be reevaluated as well-understood, routine, conventional activity of transmitting data over a network as discussed in MPEP 2106.05(d)).
With respect to claims 8 and 17:
Step 2a Prong One: incorporates the rejection of claims 6 and 15. The claims additionally recite:
determine that the drift classification is associated with a performance drift classification and determining that the drift classification is associated with the performance drift classification (mental process of evaluation/determination);
calculate a performance feature importance threshold (mental process of evaluation/determination including mental performance of a mathematical calculation); and
monitor the data stream for a concept drift (mental process of observation);
Step 2a Prong Two: the claims additionally recite:
wherein executing the instructions further causes the processing device to/ wherein the computer program product further comprises an executable portion configured to: (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f))
transmit the data stream to a bias correction module in response (insignificant extra-solution activity discussed in MPEP 2106.05(g));
create a suggested performance drift model (insignificant extra-solution activity as discussed in MPEP 2106.05(g))
Step 2b: the claims additionally recite:
wherein executing the instructions further causes the processing device to/ wherein the computer program product further comprises an executable portion configured to: (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f))
transmit the data stream to a bias correction module in response (insignificant extra-solution activity as discussed in MPEP 2106.05(g), which can be reevaluated as well-understood, routine, conventional activity of transmitting data over a network as discussed in MPEP 2106.05(d));
create a suggested performance drift model (insignificant extra-solution activity as discussed in MPEP 2106.05(g), which can be reevaluated as well-understood, routine, conventional activity of data gathering and output as discussed in MPEP 2106.05(d)).
With respect to claims 9 and 18:
Step 2a Prong One: incorporates the rejection of claims 8 and 17.
Step 2a Prong Two: the claims additionally recite wherein the bias correction module further comprises at least one of: a representational bias module configured to mitigate representational bias of the data stream; a confirmation bias module configured to mitigate confirmation bias of the data stream; a selection bias module configured to mitigate selection bias of the data stream; or a survivorship bias module configured to mitigate survivorship bias of the data stream (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f)).
Step 2b: the claims additionally recite wherein the bias correction module further comprises at least one of: a representational bias module configured to mitigate representational bias of the data stream; a confirmation bias module configured to mitigate confirmation bias of the data stream; a selection bias module configured to mitigate selection bias of the data stream; or a survivorship bias module configured to mitigate survivorship bias of the data stream (mere instructions to apply the exception using generic computer components as discussed in MPEP 2106.05(f)).
Therefore, in view of the considerations set forth in MPEP 2106.04(d), 2106.05(a)-(c) and (e)-(h), the additional elements as recited in the dependent claims discussed above alone or in combination do not integrate the judicial exception into a practical application as they are mere insignificant extra solution activity, combined with implementing the abstract idea using generic computer components, and limitations describing a field of use or technological environment. The additional elements as discussed above, in combination with the abstract idea, are not sufficient to amount to significantly more than the judicial exception as they are well, understood, routine and conventional activity as disclosed in combination with generic computer functions and components used to implement the abstract idea, and limitations describing a field of use or technological environment.
Claim Rejections – 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Ramamurthy et al. (US 20220405619 A1).
With respect to claims 1, 10, and 19, Ramamurthy teaches a system for monitoring and removing drift in machine learning models, the system comprising: a processing device; a non-transitory storage device containing instructions when executed by the processing device, causes the processing device to perform the steps of a method; a computer program product for monitoring and removing drift in machine learning models, the computer program product comprising at least one non-transitory computer-readable medium having computer-readable program code portions embodied therein, the computer-readable program code portions comprising executable portions configured to perform the method (e.g. paragraph 0029, functions carried out by processor executing instructions stored in memory; paragraph 0142, described process performed by processing logic that comprises hardware and software; instructions run on processor; paragraph 0158, computing device 1100 including processors 14; paragraph 0159, computing device 1100 including computer readable media/storage media, storing information such as instructions; paragraph 0160, memory including instructions that, when executed by processors, cause the processors to perform described functionality); and the computer-implemented method for monitoring and removing drift in machine learning models, the computer-implemented method comprising:
receiving, from one or more control automation modules, a data stream, wherein the data stream comprises data associated with the one or more control automation modules (e.g. paragraph 0023, receiving set of stream data; paragraph 0031, stream data mapped to machine learning models that need the data to make predictions; stream data is data that is continually produced, generated, and/or received by one or more data sources, such as sensors, server and security logs, real-time advertising data, click-stream data from apps and websites, etc.; paragraph 0044, stream data used as input data for machine learning models; paragraph 0082, Fig. 2B, raw input data 220 input to deployed model 224 to generate output data from any number of sources such as files, services, databases, data stores, sensors, etc.; paragraph 0088, Fig. 2B, storing the raw input data 220, preprocessed input data 222, and predictions made by deployed model 224 to journal computer objects; paragraph 0094, streamed data from data sources including clinical entities, financial entities, operational entities, populational entities, and utility dataset entities; paragraph 0096, source databases/data sources; i.e. a stream of input data is received from various sources, including as raw input data from sensors, applications, websites, etc., and as a corresponding stream of predictions made by associated models using the raw input data; Examiner notes that the specification of the instant application does not appear to explicitly define what constitutes a “control automation module,” and instead only indicates, as non-limiting examples, that it may include document classification, field extraction, data validation, data reconciliation, condition monitoring, end decisioning, and need of remediation modules (paragraph 0066 of the specification), and that these modules may be the result of training using training data (paragraph 0085 of the specification); therefore the various data sources, such as applications, sensors, etc., and/or machine learning modules which generate predictions using raw input data, of Ramamurthy appear to be analogous to the recited “control automation modules” of instant claims such as applications, condition monitoring modules, etc., where these modules may be trained/based on machine learning);
transmitting the data stream to a gauging and monitoring module (e.g. paragraph 0023, storing stream data to first computer object; trained machine learning model generating estimate using stream data as input; paragraph 0033, mapping streamed data to specific inputs required by machine learning models, allowing specific stream data to be parsed and mapped to machine learning models that need the specific data; paragraph 0034, triggering event detector responsible to detecting triggering event in data; paragraph 0044, model and dataset monitoring component monitors models and input data/data stream for detecting one or more anomalies; i.e. the streamed data is transmitted to at least a component which monitors the data stream in order to detect triggering events);
determining whether the data stream matches a declarative mapping protocol (e.g. paragraph 0023, estimate of trained machine learning model stored to second computer object; scanning first/second computer object to detect an anomaly, i.e. data drift or model drift, with the set of stream data and/or estimate; paragraph 0034, triggering event detector 104 detecting triggering event in data (i.e. the streaming data); paragraph 0037 triggering event is a data condition, which refers to some logical condition that occurs in data, such as the input data drifting outside of a threshold, the input data being of a specific type or coming from a specific source, the input data meeting some value threshold, etc.; see also paragraph 0045, rules, policies, or thresholds provided as conditional statements, such as data scientists providing definitions of ranges of performance calculations and thresholds, etc.; compare with paragraphs 0072-0073 of the specification of the instant application, indicating that a “declarative mapping protocol” may include threshold values, such as based on rules and specifications (expected data type, range of acceptable values, expected frequency of data, etc.) which define the expected format and data structure of the data stream);
determining one or more deviation instances in an instance in which the data stream does not match the declarative mapping protocol (e.g. paragraph 0023, detecting anomaly, i.e. data drift or model drift; paragraph 0044, model and dataset monitoring component initiating functionality for detecting one or more anomalies; paragraph 0045, detecting anomalies based on predetermined rules anomalies, or thresholds; paragraph 0046, anomaly is specific deviation in data beyond or outside of some threshold; detecting whether there is data drift anomaly or model drift anomaly; paragraph 0047, detected anomaly may include model performance such as accuracy, distance to ground truth, loss, etc.; paragraph 0048, detected anomaly may be bias in machine learning model);
determining one or more prescriptive actions for the one or more deviation instances (e.g. paragraph 0023, based on detection of anomaly, feeding stream data and/or estimate to trained machine learning model to update the model; objects also used to perform varied functionality such as replaying/re-accessing different models, retraining machine learning model, publishing notifications, etc.; paragraph 0063, based at least in part on detection of an anomaly, updating the model, retraining model with newly received data, backfilling older model with new model, etc.; paragraph 0064, evaluating machine learning models, based on evaluating, causing model to be deployed, such a based on comparative amounts of loss, data drift, etc.); and
implementing, using an intelligence restoration module, the one or more prescriptive actions on the one or more control automation modules (e.g. paragraph 0023, based on detection of anomaly, feeding stream data and/or estimate to trained machine learning model to update the model; objects also used to perform varied functionality such as replaying/re-accessing different models, retraining machine learning model, publishing notifications, etc.; paragraph 0063, based at least in part on detection of an anomaly, updating the model, retraining model with newly received data, backfilling older model with new model, etc.; paragraph 0064, evaluating machine learning models, based on evaluating, causing model to be deployed, such a based on comparative amounts of loss, data drift, etc.).
With respect to claims 2, 11, and 20, Ramamurthy teaches all of the limitations of claims 1, 10, and 19 as previously discussed, and further teaches the method further comprising receiving, from one or more control automation modules, a data lake, wherein the data lake comprises data associated with the one or more control automation modules; and transforming the data lake into a data stream (e.g. paragraph 0024, receiving large pool of stream data from various data sources; extracting from large pool of stream data, subset of data, mapping to particular model, feeding the subset of data to the particular model).
With respect to claims 3 and 12, Ramamurthy teaches all of the limitations of claims 1 and 10 as previously discussed, and further teaches wherein executing the instructions further causes the processing device to/wherein the computer program product further comprises an executable portion configured to:
in response to transmitting the data stream to the gauging and monitoring module, transmit the data stream to a data distribution analyzer, wherein the data distribution analyzer is configured to create a graphical representation of the data stream, and wherein the graphical representation comprises one or more representations of the data stream (e.g. paragraph 0125, describing journal data structure 600 of Fig. 6 as representing structure populated when data stream is received; paragraph 0131, describing journal data structure 700 of Fig. 7 as representing structure populated by journaling component 108; paragraph 0138, indicating that publishing proxy 112 produces or causes to be displayed the GUI of Fig. 8; user viewing predictions and stream data; paragraph 0139, generating lists of anomalies (data drift at particular timestamps); presenting insights generated by insight generator; querying journal data structure to formulate data and cause presentation of timestamp and ML prediction values; presenting window 807-1 which lists timestamp and payload input data values of journal data structure, etc.; i.e. the data structure representing the received data stream is accessed and visualized/displayed in a graphical representation via a user interface); and
determine, using an intelligence monitoring module, a deviation classification of the graphical representation (e.g. paragraph 0023, scanning computer objects to detect anomaly such as data drift or model drift; paragraph 0025, indicating that updating of deployed models is performed depending on anomalies detected; detecting unique anomalies such as data drift or model drift; paragraph 0043, rendering notifications that specifically indicate the particular anomalies detected; paragraph 0046, detecting data drift anomaly; detecting a model drift anomaly; i.e. the system determines/detects particular anomalies, including an identification of a particular type/classification for the anomaly).
With respect to claims 4 and 13, Ramamurthy teaches all of the limitations of claims 3 and 12 as previously discussed, and further teaches wherein executing the instructions further causes the processing device to/ wherein the computer program product further comprises an executable portion configured to:
determine that the deviation classification is associated with a deviation in performance (e.g. paragraph 0047, anomaly detected is model performance; paragraph 0087, automatically identifying performance anomalies);
determine a retraining protocol in response to determining that the deviation classification is associated with the deviation in performance (e.g. paragraph 0025, continuously updating (retraining or tuning) deployed models depending on anomalies detected; paragraph 0045, model that is underperforming detected; retraining/redeploying model; paragraph 0063, based on detection of an anomaly, updating the model; passing newly received data back to deployed model which causes the model to retrain using the newly retrieved data; paragraph 0156, retraining model based on detection of anomaly); and
implement the retraining protocol on the one or more control automation modules (e.g. paragraph 0025, continuously updating (retraining or tuning) deployed models depending on anomalies detected; paragraph 0045, model that is underperforming detected; retraining/redeploying model; paragraph 0063, based on detection of an anomaly, updating the model; passing newly received data back to deployed model which causes the model to retrain using the newly retrieved data; paragraph 0156, retraining model based on detection of anomaly).
With respect to claims 5 and 14, Ramamurthy teaches all of the limitations of claims 3 and 12 as previously discussed, and further teaches wherein executing the instructions further causes the processing device to/wherein the computer program product further comprises an executable portion configured to:
determine that the deviation classification is associated with a deviation in procedure (e.g. paragraph 0045, detecting anomalies based on rules, policies or thresholds; paragraph 0046, anomaly refers to specific deviation in data beyond or outside of some threshold, such as provided in a ruleset; paragraph 0047, detected anomaly can be a latency of data flow in near real-time; latency affects usefulness of the prediction; if data used as inputs to model lagging beyond threshold quantity of time, detecting and notifying the user; compare with specification of the instant application at paragraph 0091, indicating that deviation in procedure may be determined based on analyzing the data stream with algorithms, analytical tools, expected trends, etc.; i.e. where receiving streaming data with a latency beyond what is expected appears to be analogous to a deviation in procedure (i.e. such as a procedure associated with receipt of the data within expected timeframes, based on analysis of the datastream));
determine an intelligent interpretation protocol in response to determining that the deviation classification is associated with the deviation in procedure (e.g. paragraph 0004, updating (retrain, or tune) deployed model depending on anomalies, including data latency, detected in stream data; paragraph 0025, continuously updating (retraining or tuning) deployed models depending on anomalies detected; paragraph 0063, based on detection of an anomaly, updating the model; passing newly received data back to deployed model which causes the model to retrain using the newly retrieved data; paragraph 0156, retraining model based on detection of anomaly; compare with specification of the instant application at paragraph 0095, indicating that implementing the intelligent interpretation protocol may include adjusting parameters, retraining, providing additional training data, etc. with respect to the control automation modules; i.e. where the retraining of the deployed model based on detected data latency appears to be analogous to an intelligent interpretation protocol, since this is described as including adjusting, retraining, etc. of the module); and
implement the intelligent interpretation protocol on the one or more control automation modules (e.g. paragraph 0004, updating (retrain, or tune) deployed model depending on anomalies, including data latency, detected in stream data; paragraph 0025, continuously updating (retraining or tuning) deployed models depending on anomalies detected; paragraph 0063, based on detection of an anomaly, updating the model; passing newly received data back to deployed model which causes the model to retrain using the newly retrieved data; paragraph 0156, retraining model based on detection of anomaly; compare with specification of the instant application at paragraph 0095, indicating that implementing the intelligent interpretation protocol may include adjusting parameters, retraining, providing additional training data, etc. with respect to the control automation modules; i.e. where the retraining of the deployed model based on detected data latency appears to be analogous to an intelligent interpretation protocol, since this is described as including adjusting, retraining, etc. of the module).
With respect to claims 6 and 15, Ramamurthy teaches all of the limitations of claims 1 and 10 as previously discussed, and further teaches wherein executing the instructions further causes the processing device to/wherein the computer program product further comprises an executable portion configured to determine, using the intelligence restoration module, a drift classification of the data stream (e.g. paragraph 0023, scanning computer objects to detect anomaly such as data drift; paragraph 0025, indicating that updating of deployed models is performed depending on anomalies detected; detecting unique anomalies such as data drift; paragraph 0043, rendering notifications that specifically indicate the particular anomalies detected; paragraph 0046, detecting data drift anomaly; i.e. the system determines/detects particular anomalies, including an identification of a particular type/classification for the anomaly such as of a data drift anomaly).
With respect to claims 7, and 16, Ramamurthy teaches all of the limitations of claims 6 and 15 as previously discussed, and further teaches wherein executing the instructions further causes the processing device to:
determine that the drift classification is associated with a data drift classification (e.g. paragraph 0023, scanning computer objects to detect anomaly such as data drift; paragraph 0025, indicating that updating of deployed models is performed depending on anomalies detected; detecting unique anomalies such as data drift; paragraph 0043, rendering notifications that specifically indicate the particular anomalies detected; paragraph 0046, detecting data drift anomaly; i.e. the system determines/detects particular anomalies, including an identification of a particular type/classification for the anomaly such as of a data drift anomaly);
determine a range of one or more individual features in response to determining that the drift classification is associated with the data drift classification (e.g. paragraph 0046, detecting whether there is a data drift by determining whether there is a discrepancy between data range used to train the machine learning model and the corresponding data range post model deployment outside of a threshold);
calculate a data feature importance threshold (e.g. paragraph 0046, discrepancy of input data outside of threshold; comparing input training data to streaming data against a threshold); and
create a suggested data drift model (e.g. paragraph 0025, continuously updating (retraining or tuning) deployed models depending on anomalies detected; paragraph 0062, determining model to used based on presence or absence of data drift at different time periods; paragraph 0063, based on detection of an anomaly, updating the model; passing newly received data back to deployed model which causes the model to retrain using the newly retrieved data; paragraph 0064, selecting between new or old machine learning models based on relative amounts of data drift; paragraph 0156, retraining model based on detection of anomaly).
With respect to claims 8 and 17, Ramamurthy teaches all of the limitations of claims 6 and 15 as previously discussed, and further teaches wherein executing the instructions further causes the processing device to/ wherein the computer program product further comprises an executable portion configured to:
determine that the drift classification is associated with a performance drift classification (e.g. paragraph 0022, model drift refers to model losing its predictive ability; model drift/concept drift is a change in target variable or model prediction over time; paragraph 0047, anomaly detected is model performance; paragraph 0048, anomaly detected may be bias in the machine learning model; paragraph 0050, detecting bias by quantifying amount of bias and identifying target category subspace that defines the bias; paragraph 0087, automatically identifying performance anomalies);
transmit the data stream to a bias correction module in response determining that the drift classification is associated with the performance drift classification (e.g. paragraph 0053, providing input to biasing/de-biasing component 210);
monitor the data stream for a concept drift (e.g. paragraph 0022, indicating that model drift and concept drift are interchangeable terms; paragraph 0023, scanning computer objects to detect an anomaly such as model drift; paragraph 0025, detecting unique anomaly such as model drift; paragraph 0046, model and dataset monitoring component detecting model drift; paragraph 0047, anomaly detected is model performance);
calculate a performance feature importance threshold (e.g. paragraph 0045, detecting anomalies based on thresholds, such as provided ranges of performance calculations and threshold provided along with the model; paragraph 0047, detecting latency of model inputs; inputs lagging beyond threshold quantity of time; detecting this as an anomaly and notifying user); and
create a suggested performance drift model (e.g. paragraph 0025, continuously updating (retraining or tuning) deployed models depending on anomalies detected; paragraph 0062, determining model to used based on presence or absence of data drift at different time periods; paragraph 0063, based on detection of an anomaly, updating the model; passing newly received data back to deployed model which causes the model to retrain using the newly retrieved data; paragraph 0064, selecting between new or old machine learning models based on relative amounts of data drift, based on comparative distance of predictions from ground truth such as using a loss function; paragraph 0156, retraining model based on detection of anomaly).
With respect to claims 9 and 18, Ramamurthy teaches all of the limitations of claims 8 and 17 as previously discussed, and further teaches wherein the bias correction module further comprises at least one of: a representational bias module configured to mitigate representational bias of the data stream; a confirmation bias module configured to mitigate confirmation bias of the data stream; a selection bias module configured to mitigate selection bias of the data stream; or a survivorship bias module configured to mitigate survivorship bias of the data stream (e.g. paragraph 0048-0056, describing components of the system configured to detect various types of bias, including in socioeconomic status, race, gender, sexual orientation, age, ethnic group, and/or the like in various settings including healthcare, banking, social media, etc.; monitoring component 114 detecting bias by quantifying/geometrically capturing amount of bias; providing inputs to biasing/de-biasing component 210, etc.).
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain,” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting in re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (GCPA 1968)). Further, a reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill the art, including nonpreferred embodiments. Merck & Co, v. Biocraft Laboratories, 874 F.2d 804, 10 USPQ2d 1843 (Fed. Cir.), cert, denied, 493 U.S. 975 (1989). See also Upsher-Smith Labs. v. Pamlab, LLC, 412 F,3d 1319, 1323, 75 USPQ2d 1213, 1215 (Fed. Cir, 2005): Celeritas Technologies Ltd. v. Rockwell International Corp., 150 F.3d 1354, 1361, 47 USPQ2d 1516, 1522-23 (Fed. Cir. 1998).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEREMY L STANLEY whose telephone number is (469)295-9105. The examiner can normally be reached on Monday-Friday from 9:00 AM to 5:00 PM CST.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Al Kawsar, can be reached at telephone number (571) 270-3169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center and Private PAIR for authorized users only. Should you have questions about access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/JEREMY L STANLEY/
Primary Examiner, Art Unit 2127