DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment and Arguments
Applicant’s amendment filed on November 21, 2025 has been entered and made of record. Claims 1-19 and 21 are pending and are being examined in this application.
In light of applicant’s amendments and remarks, the 101 rejection is withdrawn.
Applicant’s arguments with respect to the 103 rejections have been fully considered, but are unpersuasive for at least the following reasons:
Applicant argues that the cited references do not teach or suggest the amended limitations, particularly the newly added portions: “simulating effects of lag on a performance of the machine learning model when operated under real-world conditions by using the lag feature to restrict a query for historical data records from at least one of the data sources to only historical data records available at a predetermined time prior to a time of the test procedure of the machine learning model" and "generating one or more performance metrics of the machine learning model that are dependent on lag attributes based on the output of the machine learning model” [Remarks, pg. 10]. Please see the updated rejections below, which incorporate new mappings for the newly added portions of the amended limitations. It is noted that applicant’s arguments are not applicable to the new mappings.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7-10, 13-17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Fry et al. (US Pub. 20220222573) in view of Matar et al. (US Pat. 12306909).
Referring to claim 1, Fry discloses A method of analyzing the effectiveness of a machine learning model, using one or more processors [par. 10; computer-implemented techniques], comprising:
executing a test procedure of a machine learning model comprising a plurality of input features and a lag feature associated with at least one of the plurality of input features [pars. 42 and 49; an ML test system runs performance tests to test the quality of results based on features used, where the tests are used to optimize a machine learning model for lag], wherein executing the test procedure comprises:
simulating effects of lag on a performance of the machine learning model when operated under real-world conditions by using the lag feature... [pars. 14, 22, 42, 45, 49, and 53; the performance tests simulate the results with respect to source constraints (i.e., input features) comprising constraints related to available sensors (i.e., real-world conditions), performance of the available sensors (i.e., real-world conditions), compute power of the system running the performance tests (i.e., real-world conditions), and constraints such as lag];
generating an output of the machine learning model [par. 42; note the results]; and
generating one or more performance metrics of the machine learning model that are dependent on lag attributes based on the output of the machine learning model [pars. 14, 22, 42, 45, 49, and 53; note the quality of the results tested by the performance tests, which includes lag measurements based on the results as specified by the lag constraints].
Fry does not appear to explicitly disclose that executing the test procedure comprises: querying one or more data sources for historical data records associated with the plurality of input features; that the lag feature is used to restrict a query for historical data records from at least one of the data sources to only historical data records available at a predetermined time prior to a time of the text procedure of the machine learning model; and receiving the historical data records from the one or more data sources at the machine learning model.
However, Matar discloses that executing the test procedure comprises: querying one or more data sources for historical data records [col. 4, lines 32-35 and 58-67; a plurality of forecasting model features illustrate historical data of a time series (e.g., lag)] associated with the plurality of input features [col. 7, lines 13-25; instructions (i.e., a query) for generating a future forecast/prediction model for a time series dataset is received, including: information identifying the time series dataset and information identifying one or more forecasting model features in the time series data set]; that the lag feature is used to restrict a query for historical data records from at least one of the data sources to only historical data records available at a predetermined time prior to a time of the text procedure of the machine learning model [col. 9, lines 12-30; based on the instructions, a forecasting model feature is obtained from the time series dataset; the time series constituting the forecasting model feature is divided into time-based components, each associated with time series data associated with only a particular time period; information about the time period is included in the instructions for generating the future forecast/prediction model]; and receiving the historical data records from the one or more data sources at the machine learning model [col. 7, lines 13-25; col. 9, lines 12-30; the one or more forecasting model features are obtained from the time series dataset based on the instructions].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the ML test system taught by Fry so that the tests include obtaining historical data from a time series dataset based on instructions identifying a time period as taught by Matar, with a reasonable expectation of success. The motivation for doing so would have been to determine different effects that different features (e.g., lag) have on prediction results [Matar, col. 4, lines 58-67].
Referring to claim 2, Matar discloses The method of analyzing the effectiveness of a machine learning model of claim 1, further comprising: adjusting the lag feature to a different predetermined time; executing an additional test procedure of the machine learning model; and analyzing the output of the test procedure with an output of the additional test procedure to determine how impactful the adjustment in the lag feature was to a result of the machine learning model [fig. 4.1; col. 9, lines 36-46; col. 11, lines 13-44; the time period for generating the future forecast/prediction model may be set to a first period of time to analyze an effect of the forecasting model feature in the short-term, then set to a second period of time to analyze an effect of the forecasting model feature in the long-term].
Referring to claim 3, Matar discloses The method of analyzing the effectiveness of a machine learning model of claim 2, wherein analyzing the output includes: generating a plurality of metrics for the test procedure and the additional test procedure; and comparing the plurality of metrics from each test procedure to determine how impactful each test procedure was to the machine learning model [col. 11, lines 38-44; the short-term effect is indicated with a short-term importance value, and the long-term effect is indicated with a long-term importance value; see also Fry, par. 49, which discloses generating quality measurements from running the tests and optimizing the machine learning model based on the quality measurements].
Referring to claim 4, Fry discloses The method of analyzing the effectiveness of a machine learning model of 2 claim 3, further comprising: adjusting the machine learning model based on the metrics of the test procedure and the additional test procedure [par. 49; note the optimizing of the machine learning model based on the quality measurements].
Referring to claim 7, Fry discloses The method of analyzing the effectiveness of a machine learning model of claim 1, wherein: at least one of the plurality of input features is not associated with a lag feature [pars. 42 and 49; testing the quality of results based on features used includes testing for other factors such as accuracy, energy consumption, and result frequency, in addition to lag].
Referring to claim 8, see the rejection for claim 1 and 2.
Referring to claim 9, see the rejection for claim 3.
Referring to claim 10, see the rejection for claim 4.
Referring to claim 13, Matar discloses The method of analyzing the effectiveness of a machine learning model of claim 8, wherein: the plurality of test procedures comprise every permutation of a preset number of predetermined times for each of the plurality of input features for which lag is being tested [col. 11, lines 45-54; the time period is set to all sub-windows of size N of the time series to determine which features are important over different horizons of time].
Referring to claim 14, see at least the rejection for claim 1. Fry further discloses A system, comprising: one or more computing devices; and memory storing instructions, the instructions being executable by the one or more computing devices, wherein the one or more computing devices are configured to perform the claimed steps [par. 10; note the computer-implemented techniques].
Referring to claim 15, see the rejection for claim 2.
Referring to claim 16, see the rejection for claim 3.
Referring to claim 17, see the rejection for claim 4.
Referring to claim 20, see the rejection for claim 7.
Claims 5, 6, 11, 12, 18, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Fry and Matar in view of Pai et al. (US Pub. 20200193234).
Referring to claim 5, Fry and Matar do not appear to explicitly disclose The method of analyzing the effectiveness of a machine learning model of claim 1, wherein: the predetermined time is based on a known amount of lag associated with the at least one of the data sources.
However, Pai discloses The method of analyzing the effectiveness of a machine learning model of claim 1, wherein: the predetermined time is based on a known amount of lag associated with the at least one of the data sources [par. 47; once a time lag is determined, input data anomalies associated with the input data from the determined time period are used to determine a contribution to output data anomaly and/or performance anomaly].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the ML test system taught by the combination of Fry and Matar so that the time period is a determined time lag as taught by Pai, with a reasonable expectation of success. The motivation for doing so would have been to facilitate determining the impact of the input data reflected with a time lag with respect to the input data’s impact on the output data [Pai, par. 160].
Referring to claim 6, Fry and Matar do not appear to explicitly disclose The method of analyzing the effectiveness of a machine learning model of claim 1, wherein: the predetermined time is between about 3 hours and 48 hours.
However, Pai discloses The method of analyzing the effectiveness of a machine learning model of claim 1, wherein: the predetermined time is between about 3 hours and 48 hours [par. 73; the comparative period may be 24 hours as an example].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the ML test system taught by the combination of Fry and Matar so that the time period is a determined time lag as taught by Pai, with a reasonable expectation of success. The motivation for doing so would have been because the time period may be set to any length to allow clear representation of the effects of each element in the period [Matar, col. 9, lines 36-39].
Referring to claim 11, see the rejection for claim 5.
Referring to claim 12, see the rejection for claim 6.
Referring to claim 18, see the rejection for claim 5.
Referring to claim 19, see the rejection for claim 6.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GRACE PARK whose telephone number is (571)270-7727. The examiner can normally be reached M-F 8AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, TAMARA KYLE can be reached at (571)272-4241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Grace Park/Primary Examiner, Art Unit 2144