DETAILED ACTION
This office action is responsive to the above identified application filed 4/16/2026. The application contains claims 1-18, 21-22, all examined and rejected.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim 21 rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Examiner was not able to find a support for the claim’s requirement “wherein the first quantized inference model and the quantized second inference model are configured as inference models that are forbidden to be re-trained after an initial training and deployment has been completed”. The specification disclose the possibility to forbid the training of the first quantized inference model See at least ¶17, ¶65, ¶95, ¶125 that point to the original quantized model that intentionally kept fixed. However, the examiner was not able to find a support for second quantized model forbidden from training or a first and second quantized models forbidden from training.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 12-14, 17-18, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Olgiati et al. [US 2021/009,7433 A1, hereinafter Olgiati] in view of Green et al. [US 2022/0171863 A1, hereinafter Green] in view of ALLAHDADIAN et al [US 2022/0188410 A1, hereinafter ALLAHDADIAN].
With regard to Claim 1,
Olgiati teach a method of managing data (¶16, ¶27), the method comprising:
making a first identification that a first data drift has occurred in first data obtained from a data collector (¶24, “The data collection 154A may, for individual inference requests, collect inference data such as the inference input data, the resulting inference, and various elements of model metadata (e.g., a model identifier, a model version identifier, an endpoint identifier, a timestamp, a container identifier, and so on)”, ¶27, “the analysis may automatically detect problems or anomalies such as models that fail golden examples, outliers in input data, inference data distribution changes, label distribution changes, label changes for individual entities, ground truth discrepancies, and/or other forms of data drift or model drift”, ¶46);
obtaining, in response to the first identification, second data from the data collector (¶26, “inference data may be collected for particular windows of time“, ¶30, ” The analysis 170 of inference data may be initiated on a schedule, e.g., every twenty-four hours to analyze the previous days’ worth of inference data. The analysis 170 may be initiated on a manual and ad-hoc basis, e.g., by user input”, ¶45, “inference generation shown in 500 may be performed continuously or regularly without being impacted negatively by the data collection or analysis of the collected data. As shown in 540, if analysis is desired at this time, then the data may be retrieved from storage. The inference production may be decoupled from the storage and from the analysis in order to minimize the performance impact on the inference”);
classifying the second data using a continuous inference model and an anomaly threshold to obtain a first classification, the first classification indicating whether the second data is considered anomalous or non-anomalous (¶18, “machine learning inference system 140 may apply the tested model 135 to inference input data 116 from one or more data sources 110A and may produce inferences”.¶20, “machine learning model may be associated with a collection of weights”, ¶38, “analysis 170 may be performed according to thresholds and/or tiers of thresholds”);
classifying the second data using a second inference model and the anomaly threshold to obtain a second classification, the second classification indicating whether the second data is considered anomalous or non-anomalous (¶16, “The analysis may automatically detect problems or anomalies such as models that fail golden examples, outliers in input data, inference data distribution changes, label distribution changes, label changes for individual entities, ground truth discrepancies, and/or other forms of data drift or model drift”, ¶46, ¶¶49-50, ¶27, “ analysis may be performed according to thresholds, and thresholds”, ¶38, “analysis 170 may be performed according to thresholds and/or tiers of thresholds. For example, if a model is less accurate by a threshold percentage yesterday than the day before yesterday, then a problem may be detected and a notification generated accordingly. Tiers of thresholds may represent severity levels of detected problems”);
making a first determination, using the first classification and the second classification, that a second data drift has occurred in the second data (¶41, “analysis 170 may compare two versions of a model to checkpoint previous versions and provide improved alerts”, ¶27, “A machine learning analysis system 170 may use the collected inference data in the data store 160 to perform automated analysis of inference production 152A. The analysis system 170 may determine the correctness of inputs, outputs, and intermediate steps of the inference production and/or the quality of deployed machine learning models”, ¶38, “¶38, “analysis 170 may be performed according to thresholds and/or tiers of thresholds. For example, if a model is less accurate by a threshold percentage yesterday than the day before yesterday, then a problem may be detected and a notification generated accordingly. Tiers of thresholds may represent severity levels of detected problems”);
making a second determination, using the second data and a first quantized inference model, that the second data drift indicates that the first data drift is a transient data drift (¶¶49-50, “determine whether the predictions are staying the same or similar over time”, ¶38, “thresholds and/or tiers of thresholds. For example, if a model is less accurate by a threshold percentage yesterday than the day before yesterday, then a problem may be detected and a notification generated accordingly. Tiers of thresholds may represent severity levels of detected problems, and notifications may vary based (at least in part) on the tier in which a problem is placed”);
performing in response to the second determination, an action set (¶31, “ analysis system 170 may include a component for automated problem remediation 174 that attempts to remediate, correct, or otherwise improve a detected problem. The problem remediation 172 may initiate one or more actions to improve a model or its use in generating inferences …”, ¶37, “send notifications to a notification system “,¶46, “if a problem was detected, then one or more actions may be initiated by the analysis system to remediate the problem”).
Olgiati does not explicitly teach using a second quantized inference model, a first quantized inference model, the continuous inference model and the first quantized inference model being hosted using limited computing resources of a data processing system configured as an anomaly detector.
Green teach classifying a second quantized inference model, a first quantized inference model (¶23, “determine an appropriate quantization for data sampling, analysis, and output”, ¶85, “execute a machine learning model on instances of the generalized data structure; converting the function call to a set of instructions for the embedded device based on the input mapping; executing the set of instructions to generate an output of the machine learning model “, ¶86, “store a set of quantized weights for the recurrent neural network. The embedded device can then: execute the LSTM network on the vector of data points; and output a classification or other inference”);
the continuous inference model and the first quantized inference model being hosted using limited computing resources of a data processing system configured as an anomaly detector (¶15, “embedded devices (i.e., IoT devices, edge devices, very small devices) such as pressure and pump rate sensors in oil wells or temperature and humidity sensors in home and industrial applications—utilize low-compute low-memory microcontrollers“, ¶17, “the method S100 is executed by a computer system, such as a computational device, a set of computer servers, or a cloud-based computing platform, in order to install containerized applications on one or more embedded devices characterized by low-computing and low-memory resources”, ¶18, “computer system enables reconfigurable compute capabilities at embedded devices (e.g., microcontrollers) with limited on-board memory and processing bandwidth, … Thus, the computer system can minimize the memory footprint (e.g., to less than or equal to 256 kilobytes)”, ¶50, “ embedded devices such as 32-bit RISC devices include extremely limited on-board memory (e.g., 32-256 KB). More specifically, the computer system can generate a HAL and a CRE characterized by a memory footprint of less than 256 KB on the embedded device. Therefore, the computer system can automatically generate a CRE and HAL that is customized to the hardware of the embedded device (as defined by the set of hardware parameters), the set of selected container functions, and/or particular sensor types but excludes unneeded and/or unspecified functionalities, thereby reducing the size (e.g., memory footprint) of the CRE and permitting larger, containerized applications“, ¶86, “The embedded device can then: execute the LSTM network on the vector of data points; and output a classification or other inference based on the vector of data points via available hardware accelerators on the embedded device”).
Olgiati and Green are analogous art to the claimed invention because they are from a similar field of endeavor of machine learning based data analysis systems that generate inference outputs from collected data to detect anomalous or significant conditions and make determinations or take actions based on those inference results. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Olgiati resulting in resolutions as disclosed by Green with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Olgiati as described above to replace the inference model with a quantized inference model to reduce model size, speeding up inference, lowering memory usage, and decreasing power consumption, making complex AI deployable on resource constrained devices like edge hardware or cheaper cloud instances, all while maintaining most of the original model's accuracy by converting parameters to lower-precision numbers. In addition hosting the models using limited computing resources of a data processing system configured as an anomaly detector allow reducing the size (e.g., memory footprint) of the CRE and permitting larger, containerized applications (Green, ¶50). This is a simple substitution of one known element for another to obtain predictable results, combining prior art elements according to known methods to yield predictable results, use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
Olgiati-Green does not explicitly teach data drift indicates that the first data drift is a transient data drift that is temporary rather than permanent, comprising correcting the first data drift without re-training the continuous inference model and the first quantized inference model.
ALLAHDADIAN teach data drift indicates that the first data drift is a transient data drift that is temporary rather than permanent, comprising correcting the first data drift without re-training the continuous inference model and the first quantized inference model (¶19, “There is no need to change the anomaly detection model or retrain it, which saves much time and cost of retraining”, “This approach is reversible because it can be used in unidirectional or bidirectional ways to respectively have permanent or temporary effects in dealing with different types of concept drifts”, ¶52, “moving average 130 facilitates automatically distinguishing an anomalous input from concept drift”, ¶39, “… Seasonality may influence consumer preferences which may interfere with a predictive ML model for a supply chain or for behavioral advertisement targeting“, ¶¶40-41, “ If that anomaly detector repeatedly or continuously raises false alarms, the anomaly detector may become more or less useless for two reasons. First, system administrators waste much time with manual forensics and diagnostics to decide whether an alarm is correct or not. Second, system administrators learn to ignore the malfunctioning anomaly detector such that a true alarm for a real problem goes unnoticed” ¶55, “when moving average 130 of reconstruction error 120 for feature 111 exceeds feature suppression threshold 140, concept drift is detected”, ¶59, “downstream feature suppression serves two purposes. First, it prevents abnormally trending feature(s) from causing a false alarm for a mistakenly supposed anomaly. Second, it allows the anomaly detector to remain in reliable service and continue to inspect an ongoing series of inputs without retraining” ).
Olgiati-Green and ALLAHDADIAN are analogous art to the claimed invention because they are from a similar field of endeavor of anomaly detection. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Olgiati-Green resulting in resolutions as disclosed by ALLAHDADIAN with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Olgiati-Green as described above to prevent abnormally trending feature(s) from causing a false alarm for a mistakenly supposed anomaly and allows the anomaly detector to remain in reliable service and continue to inspect an ongoing series of inputs without retraining (ALLAHDADIAN, ¶59). This is a simple combining prior art elements according to known methods to yield predictable results, use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
With regard to Claim 2,
Olgiati-Green-ALLAHDADIAN teach the method of claim 1, wherein classifying the second data using the continuous inference model and the anomaly threshold comprises (Olgiati, ¶38, “analysis 170 may be performed according to thresholds and/or tiers of thresholds. For example, if a model is less accurate by a threshold percentage yesterday than the day before yesterday, then a problem may be detected and a notification generated accordingly. Tiers of thresholds may represent severity levels of detected problems”, ¶41):
obtaining a first inference using the continuous inference model and the second data (Olgiati, ¶41, “ analysis 170 may detect anomalous model drift. For example, if the difference between the prediction distributions of the current model and the previous model is typically 0.05 in squared distance, but for the current model the difference is 1.5 “, “ analysis 170 may plot the accuracy over time; if the accuracy is a straight line, then the analysis may recommend that training be performed less frequently to conserve resources. However, if the line is jagged, then the analysis 170 may recommend that training be performed more frequently to improve the quality of predictions”);
making a third determination regarding whether the first inference is within the anomaly threshold (Olgiati, ¶38, “Tiers of thresholds may represent severity levels of detected problems”, ¶41, “is typically 0.05 in squared distance, but for the current model the difference is 1.5 “, , ¶46, “determine whether a problem was detected. The analysis may be performed according to thresholds that determine whether a given observation about the model rises to the level of a problem that may require intervention”);
in a first instance of the third determination in which the first inference is within the anomaly threshold, classifying the second data as non-anomalous to obtain the first classification (Olgiati, ¶41, “ difference between the prediction distributions of the current model and the previous model is typically 0.05 in squared distance”, ¶46, “determine whether a problem was detected. The analysis may be performed according to thresholds that determine whether a given observation about the model rises to the level of a problem that may require intervention”); and
in a second instance of the third determination where the first inference is not within the anomaly threshold, classifying the second data as anomalous to obtain the first classification (Olgiati, ¶41, “ analysis 170 may detect anomalous model drift. For example, if the difference between the prediction distributions of the current model and the previous model is typically 0.05 in squared distance, but for the current model the difference is 1.5, the analysis 170 may report that the training data may be contaminated or otherwise problematic“, ¶46, “determine whether a problem was detected. The analysis may be performed according to thresholds that determine whether a given observation about the model rises to the level of a problem that may require intervention”).
The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 3,
Olgiati-Green-ALLAHDADIAN teach the method of claim 2, wherein classifying the second data using the second quantized inference model comprises:
quantizing the second data to obtain quantized second data (Green, ¶23, “ at a first quantization”, “at a second quantization”);
obtaining a second inference using the second quantized inference model and the quantized second data (Green, ¶23, “apply outputs from the first set of containerized applications at the first quantization in a first instance of a machine learning model; and apply outputs from the second set of containerized applications in a second instance of the machine learning model”, ¶86, “store a set of quantized weights for the recurrent neural network. The embedded device can then: execute the LSTM network on the vector of data points; and output a classification or other inference based on the vector of data points “);
making a fourth determination regarding whether the second inference is within the anomaly threshold (Olgiati, ¶38, “Tiers of thresholds may represent severity levels of detected problems”, , “green tier may indicate that the model is working as expected, a yellow tier may indicate that one or more problems should be investigated, and a red tier may indicate that a model is probably broken and producing faulty inferences. The thresholds and/or tiers may be specified by users or may represent defaults”, ¶41, “is typically 0.05 in squared distance, but for the current model the difference is 1.5 “, , ¶46, “determine whether a problem was detected. The analysis may be performed according to thresholds that determine whether a given observation about the model rises to the level of a problem that may require intervention”);
in a first instance of the fourth determination where the second inference is within the anomaly threshold, classifying the second data as non-anomalous to obtain the second classification (Olgiati, ¶38, “Tiers of thresholds may represent severity levels of detected problems”, “green tier may indicate that the model is working as expected”, ¶41, “is typically 0.05 in squared distance, but for the current model the difference is 1.5 “, ¶46, “determine whether a problem was detected. The analysis may be performed according to thresholds that determine whether a given observation about the model rises to the level of a problem that may require intervention”); and
in a second instance of the fourth determination where the second inference is not within the anomaly threshold, classifying the second data as anomalous to obtain the second classification (Olgiati, ¶38, “Tiers of thresholds may represent severity levels of detected problems”, , “green tier may indicate that the model is working as expected, a yellow tier may indicate that one or more problems should be investigated, and a red tier may indicate that a model is probably broken and producing faulty inferences. The thresholds and/or tiers may be specified by users or may represent defaults”, ¶41, “is typically 0.05 in squared distance, but for the current model the difference is 1.5 “, ¶46, “determine whether a problem was detected. The analysis may be performed according to thresholds that determine whether a given observation about the model rises to the level of a problem that may require intervention”). The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 12,
Claim 12 is similar in scope to claim 1; therefore it is rejected under similar rationale. Further Olgiati teach A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for managing data See at least claim 15, ¶¶54-56, ¶58.
With regard to Claim 13,
Claim 13 is similar in scope to claim 2; therefore it is rejected under similar rationale.
With regard to Claim 14,
Claim 14 is similar in scope to claim 3; therefore it is rejected under similar rationale.
With regard to Claim 17,
Claim 17 is similar in scope to claim 1; therefore it is rejected under similar rationale. Further Olgiati teach a processor and a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for managing data See at claim 1, least ¶¶54-56.
With regard to Claim 18,
Claim 18 is similar in scope to claim 2; therefore it is rejected under similar rationale.
With regard to Claim 22,
Olgiati-Green-ALLAHDADIAN teach the method of claim 1, wherein the first data drift is determined as the transient data drift when it is determined, as part of the second determination, that the second data drift shifts the second data is an opposite direction of a shift of the first data caused by the first data drift (ALLAHDADIAN, ¶19, ¶51, “ Anomaly detection alerts a sudden spike in reconstruction error of feature(s). Concept drift can be gradual or sudden depending on its cause”, ¶¶52-53, “each of features 111-113 has its own moving average of its own reconstruction error. Moving average 130 may serve two purposes. First, moving average 130 provides smoothing such that a sudden spike in current reconstruction error 120 of feature 111 should not by itself cause a reaction for concept drift detection as the spike should for anomaly detection. That is, moving average 130 facilitates automatically distinguishing an anomalous input from concept drift”). The same motivation to combine for claim 1 equally applies for current claim.
Claims 4-9, 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Olgiati et al. [US 2021/009,7433 A1, hereinafter Olgiati] in view of Green et al. [US 2022/0171863 A1, hereinafter Green] in view of ALLAHDADIAN et al [US 2022/0188410 A1, hereinafter ALLAHDADIAN] in view of Bialkowski et al. [US 2010/0008594 A1, hereinafter D1].
With regard to Claim 4,
Olgiati-Green-ALLAHDADIAN teach the method of claim 3.
Olgiati-Green-ALLAHDADIAN does not explicitly teach identifying a quantized data value corresponding to each data value of the second data using a schema for quantizing data and a set of quantized data values; and obtaining the quantized second data using the quantized data value corresponding to each data value of the second data.
D1 teach identifying a quantized data value corresponding to each data value of the second data using a schema for quantizing data and a set of quantized data values (¶8, “The quantization level indicates a number of amplitudes of data values which are summarized within a quantization interval to a reconstruction value. For example, with a quantization level of 15 the amplitudes from 0 to 14 or from 15 to 29 etc. are each summarized to a reconstruction value, e.g. 7, 23 etc.”, quantization level interval is schema); and
obtaining the quantized second data using the quantized data value corresponding to each data value of the second data (¶39, “a range of figures from 0 to 255 is separated out into eight first quantization intervals”, schema specify ranges, “a value is indicated on the lower and on the upper interval boundary for every first quantization interval Q11, as well as a first reconstruction value R1 corresponding to the respective first quantization interval “, “the uncoded data value X0=90 is quantized into the value 2, i.e. a first intermediate value X1=2”).
Olgiati-Green-ALLAHDADIAN and D1 are analogous art to the claimed invention because they are from a similar field of endeavor of machine learning inference systems. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Olgiati-Green-ALLAHDADIAN resulting in resolutions as disclosed by D1 with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Olgiati-Green-ALLAHDADIAN as described above to optimize performance and efficiency with a minimal loss in accuracy which provide faster inference speed, lower power consumption, enhance scalability and reduce operational cost. This is a simple substitution of one known element for another to obtain predictable results, combining prior art elements according to known methods to yield predictable results, use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
With regard to Claim 5,
Olgiati-Green-ALLAHDADIAN-D1 teach the method of claim 4, wherein the schema specifies a range of the second data uniquely corresponding to each quantized data value of the set of quantized data values (D1, ¶39, “a range of figures from 0 to 255 is separated out into eight first quantization intervals QI1 of equal size, i.e. a first quantization level of the first quantization comes to 32”, ¶42, “the interval boundaries of the second quantization intervals QI2 are shifted in such a way that each of them corresponds to the nearest-located interval boundaries of the first quantization intervals”, ¶11, “ for each of the third quantization intervals a third reconstruction value is established in such a way that the third reconstruction value is located within the associated third”, each interval has its own construction value). The same motivation to combine for claim 4 equally applies for current claim.
With regard to Claim 6,
Olgiati-Green-ALLAHDADIAN-D1 teach the method of claim 5, wherein the second quantized inference model is trained using training data obtained after the first data drift (Olgiati, ¶16, “The analysis may automatically detect problems or anomalies such as models that fail golden examples, outliers in input data, inference data distribution changes, label distribution changes, label changes for individual entities, ground truth discrepancies, and/or other forms of data drift or model drift”, ¶31, “the analysis system 170 may automatically initiate retraining of machine learning models based on problem detection”, ¶39, “the retraining 374 may include generating a new set of training data. The new set of training data may be consistent with one or more characteristics of the inference input data “, ¶41, “For frequently retrained models, the analysis 170 may detect anomalous model drift … automatically initiate model retraining once the predictions differ”, Green, ¶86, “store a set of quantized weights for the recurrent neural network. The embedded device can then: execute the LSTM network on the vector of data points; and output a classification or other inference based on the vector of data points “). The same motivation to combine for claim 4 equally applies for current claim.
With regard to Claim 7,
Olgiati-Green-ALLAHDADIAN-D1 teach the method of claim 6, wherein making the first determination comprises:
making a fifth determination (Olgiati, ¶46, “As shown in 560, the method may determine whether a problem was detected. The analysis may be performed according to thresholds that determine whether a given observation about the model rises to the level of a problem”) regarding whether the first classification specifies that the second data is considered non-anomalous and the second classification specifies that the second data is considered anomalous (Olgiati, ¶41, “The analysis 170 may compare two versions of a model to checkpoint previous versions and provide improved alerts. For frequently retrained models, the analysis 170 may detect anomalous model drift. For example, if the difference between the prediction distributions of the current model and the previous model is typically 0.05 in squared distance, but for the current model the difference is 1.5, the analysis 170 may report that the training data may be contaminated or otherwise problematic”); and
in a first instance of the fifth determination in which the first classification specifies that the second data is considered non-anomalous and the second classification specifies that the second data is considered anomalous (Olgiati, ¶41, “ analysis 170 may compare two versions of a model to checkpoint previous versions …”, ¶49, “inference data distribution change analysis …”, ¶50, “The label distribution change analysis 972 may compare inference data 960B from a recent window of time (e.g., the previous twenty-four hours) with inference data 960A from a prior window of time … if yesterday had 85% TRUE predictions but the day before yesterday had 15% TRUE predictions, then the label distribution change analysis 972 may identify this discrepancy as a problem” :
making a second identification that the second data drift has occurred in the second data (Olgiati, ¶16,”The analysis may automatically detect problems or anomalies such as models that fail golden examples, outliers in input data, inference data distribution changes, label distribution changes, label changes for individual entities, ground truth discrepancies, and/or other forms of data drift or model drift”). The same motivation to combine for claim 4 equally applies for current claim.
With regard to Claim 8,
Olgiati-Green-ALLAHDADIAN-D1 teach the method of claim 7, wherein making the second determination comprises:
obtaining the first quantized inference model, the first quantized inference model being trained using training data obtained prior to the first data drift (Olgiati, ¶41, “The analysis 170 may compare two versions of a model to checkpoint previous versions … difference between the prediction distributions of the current model and the previous model”, Green, ¶86, “store a set of quantized weights for the recurrent neural network”);
classifying the second data using the first quantized inference model and the anomaly threshold to obtain a third classification (Olgiati, ¶41, “The analysis 170 may compare two versions of a model to checkpoint previous versions and provide improved alerts. For frequently retrained models, the analysis 170 may detect anomalous model drift. For example, if the difference between the prediction distributions of the current model and the previous model is typically 0.05 in squared distance, but for the current model the difference is 1.5, the analysis 170 may report that the training data may be contaminated or otherwise problematic”, ¶38, “ analysis 170 may be performed according to thresholds and/or tiers of thresholds”, Green, ¶86, “store a set of quantized weights for the recurrent neural network”), the third classification indicating whether the second data is considered anomalous or non-anomalous (Olgiati, ¶41, “The analysis 170 may compare two versions of a model to checkpoint previous versions and provide improved alerts. For frequently retrained models, the analysis 170 may detect anomalous model drift. For example, if the difference between the prediction distributions of the current model and the previous model is typically 0.05 in squared distance, but for the current model the difference is 1.5, the analysis 170 may report that the training data may be contaminated or otherwise problematic”);
making a sixth determination regarding whether the third classification indicates that the second data is non-anomalous (Olgiati, ¶46, “As shown in 560, the method may determine whether a problem was detected. The analysis may be performed according to thresholds that determine whether a given observation about the model rises to the level of a problem”); and
in a first instance of the sixth determination in which the third classification indicates that the second data is non-anomalous: making a third identification that the first data drift is a transient data drift (Olgiati, ¶41, “The analysis 170 may compare two versions of a model to checkpoint previous versions and provide improved alerts. For frequently retrained models, the analysis 170 may detect anomalous model drift. For example, if the difference between the prediction distributions of the current model and the previous model is typically 0.05 in squared distance, but for the current model the difference is 1.5, the analysis 170 may report that the training data may be contaminated or otherwise problematic”). The same motivation to combine for claim 4 equally applies for current claim.
With regard to Claim 9,
Olgiati-Green-ALLAHDADIAN-D1 teach the method of claim 8, wherein the first determination is made, at least in part, using the first quantized inference model (Olgiati, ¶27, “analysis may automatically detect problems or anomalies such as models that fail golden examples, outliers in input data, inference data distribution changes, label distribution changes, label changes for individual entities, ground truth discrepancies, and/or other forms of data drift or model drift”, ¶46, ¶41, “the analysis 170 may detect anomalous model drift. For example, if the difference between the prediction distributions of the current model and the previous model is typically 0.05 in squared distance, but for the current model the difference is 1.5, the analysis 170 may report that the training data may be contaminated or otherwise problematic", ¶38, “analysis 170 may be performed according to thresholds and/or tiers of thresholds. For example, if a model is less accurate by a threshold percentage yesterday than the day before yesterday, then a problem may be detected”, ¶16, “The analysis may automatically detect problems or anomalies such as models that fail golden examples, outliers in input data, inference data distribution changes, label distribution changes, label changes for individual entities, ground truth discrepancies, and/or other forms of data drift or model drift”, ¶23, Green, ¶86, “store a set of quantized weights for the recurrent neural network. The embedded device can then: execute the LSTM network on the vector of data points; and output a classification or other inference based on the vector of data points”). The same motivation to combine for claim 4 equally applies for current claim.
With regard to Claim 15,
Claim 15 is similar in scope to claim 4; therefore it is rejected under similar rationale.
With regard to Claim 16,
Claim 16 is similar in scope to claim 5; therefore it is rejected under similar rationale.
Claim(s) 10-11 are rejected under 35 U.S.C. 103 as being unpatentable over Olgiati et al. [US 2021/009,7433 A1, hereinafter Olgiati] in view of Green et al. [US 2022/0171863 A1, hereinafter Green] in view of ALLAHDADIAN et al [US 2022/0188410 A1, hereinafter ALLAHDADIAN] in view of Bialkowski et al. [US 2010/0008594 A1, hereinafter D1] in view of Liu et al. [US 2021/0406796 A1, hereinafter Liu].
With regard to Claim 10,
Olgiati-Green-ALLAHDADIAN-D1 teach the method of claim 9, wherein performing the action set comprises one selected from a list of actions consisting of:
[taking action based on comparing] continuous inference model to a historical version of the continuous inference model (¶40, “ two different versions of a model may be trained, tested, and used to produce inferences in parallel or serially. For example, one version of a model may be represented using trained model 125A and tested model 135A, and another version of the model may be represented using trained model 125B and tested model 135B. One of the versions may represent a more recent version that is sought to be compared against an older version”, ¶41, “ analysis 170 may compare two versions of a model to checkpoint previous versions and provide improved alerts”, ¶47, “ golden example discrepancy analysis 672 may detect inadvertent deployment of a faulty model, detect changes in the production environment (e.g., changes in a dependency that have impacted the model), and/or ensure that new versions of a model do not break fundamental use cases“, ¶31, “ analysis system 170 may include a component for automated problem remediation 174 that attempts to remediate, correct, or otherwise improve a detected problem”), and reverting the first quantized inference model to a historical version of the first quantized inference model.
Olgiati teach the ability to compare models versions and identify if an older model has a better performance and the ability to use this determination to automatically take an action for remediation. Even though the action is understood to include the activation of the previous version, however as Olgiati does not explicitly teach that the activate action is to revert the model version and in effort to expedite persecution Liu teach reverting the continuous inference model to a historical version of the continuous inference model (¶11, “The determined anomalies and any helpful information related thereto may be employed to automate remediation (e.g., debugging, repairing, rolling back, restarting, updates, system environment adjustments, etc.)”, ¶23, “The model 1 may be a forecasting model configured to predict first total payment volumes for future periods of time “, ¶40, “the model 2 may be a machine learning model trained to predict second total payment volumes for the future periods of time”, ¶16, “systems and methods may include automatically performing a rollback of one or more recently released software versions and/or a rollout of a previous software version”, ¶44, “ the system 206 may automatically rollback one or more recently released software versions and rollout a previous version in response to detecting an anomaly”).
Olgiati-Green-ALLAHDADIAN-D1 and Liu are analogous art to the claimed invention because they are from a similar field of endeavor of monitoring production, detecting anomalies and taking remediation steps automatically based on the detected anomalies.
Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Olgiati-Green-ALLAHDADIAN-D1 resulting in resolutions as disclosed by Liu with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Olgiati-Green-ALLAHDADIAN-D1 as described above to provides a critical safety net, allowing teams to quickly recover from issues introduced by a new release which prevents a total system failure and minimizes negative impact on users or operations. This is a simple substitution of one known element for another to obtain predictable results, combining prior art elements according to known methods to yield predictable results, use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
With regard to Claim 11,
Olgiati-Green-ALLAHDADIAN-D1 teach the method of claim 10, wherein reverting the first quantized inference model to a historical version of the first quantized inference model comprises: replacing the first quantized inference model with the second quantized inference model (Because this limitation merely elaborates on a conditional limitation of a parent claim, the prior art of record is deemed to meet this limitation by virtue of meeting an alternative condition in the parent claim). The same motivation to combine for claim 10 equally applies for current claim.
Claim 21 rejected under 35 U.S.C. 103 as being unpatentable over Olgiati et al. [US 2021/009,7433 A1, hereinafter Olgiati] in view of Green et al. [US 2022/0171863 A1, hereinafter Green] in view of ALLAHDADIAN et al [US 2022/0188410 A1, hereinafter ALLAHDADIAN] in view of Wiemker et al. [US 2024/0005455 A1, hereinafter Wiemker].
With regard to Claim 21,
Olgiati-Green-ALLAHDADIAN teach the method of claim 1, wherein the first quantized inference model and the quantized second inference model (Green, ¶85, “execute machine learning classification and/or inference generation based on data sampled from a sensor“).
Olgiati-Green-ALLAHDADIAN does not teach are configured as inference models that are forbidden to be re-trained after an initial training and deployment has been completed.
Wiemker teach [model] are configured as inference models that are forbidden to be re-trained after an initial training and deployment has been completed (¶85, “the data-driven model is trained in a training phase, and the frozen data-driven model applied in the inference phase, i.e. deployment or application phase. In training mode, an initial model of the data-driven model is trained based on a set of training data to produce a trained data-driven model. In deployment mode, also referred to as inference mode, the pre-trained data-driven model is fed with non-training, new data, to operate during normal use”).
Olgiati-Green-ALLAHDADIAN and Wiemker are analogous art to the claimed invention because they are from a similar field of endeavor of machine learning inference systems.
Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Olgiati-Green-ALLAHDADIAN resulting in resolutions as disclosed by Wiemker with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Olgiati-Green-ALLAHDADIAN as described above in order to obtain the advantage of a large training set and reproducible performance (Wiemker, ¶85). This is combining prior art elements according to known methods to yield predictable results, use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143). The same motivation to combine for claim 10 equally applies for current claim.
Response to Arguments
Examiner respectfully withdraw the 35 USC 101 rejection based on applicant amendments persuasive arguments related to the current remarks regarding how the current invention provide an improvement to computer and inference model functionality (remarks P. 12-13).
Applicant’s arguments with respect to claim(s) 1-18 and 21-22 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
As to the remaining dependent claims, applicant argue that they are allowable due to their respective direct and indirect dependencies upon one of the aforementioned Independent claims. The examiner respectfully disagrees, Independent claims were not allowable as stated in the paragraph above in this “Response to Arguments” section in this office action.
Conclusion
The prior art made of record and not relied upon is considered pertinent to the applicant’s disclosure.
US Patent Application Publication No. 2021/0133632 A1 filed by Elprin et al. that disclose an automated and universal systems and methods for detecting model drift at large scale See at least ¶8, -15, ¶34, ¶47, ¶¶49-50
Examiner has pointed out particular references contained in the prior arts of record in the body of this action for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and Figures may apply as well. It is respectfully requested from the applicant, in preparing the response, to consider fully the entire references as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior arts or disclosed by the examiner. It is noted that any citation to specific pages, columns, figures, or lines in the prior art references any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331-33, 216 USPQ 1038-39 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHAMED ABOU EL SEOUD whose telephone number is (303)297-4285. The examiner can normally be reached Monday-Thursday 9:00am-6:00pm MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHAMED ABOU EL SEOUD/Primary Examiner, Art Unit 2148
Prosecution Insights