DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1-20 are pending for examination.
Claims 1, 8 and 15 are independent Claims.
Claims 1-20 are rejected under 35 U.S.C. §102.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Achin et al. (U.S. 2018/0060738 hereinafter Achin).
As Claim 1, Achin teaches a system for determining the importance of a feature to model predictions, comprising:
a processor that executes instructions (Achin (¶0026 line 2-5), memory configured to store processor-executable instructions and a processor); and
a non-transitory computer-readable medium having instructions executable by the
processor (Achin (¶0026 line 2-5), memory configured to store processor-executable instructions and a processor) for:
identifying a time-series data record describing a plurality of features for each timestep of a sequence of timesteps (Achin (¶0272 line 2-5, ¶0273 line 4-9), available data includes 3 years of previous daily sales data from 10,000 locations plus other variables);
generating an unmasked prediction describing a plurality of predictions for a timestep of the sequence of timesteps by applying a trained timeseries model to the time-series data record (Achin (¶0310 line 4-6, fig. 9 item 910), system fits the associated predictive model to the initial dataset), wherein the trained timeseries model generates the plurality of predictions for the timestep based on previous timesteps in the sequence of timesteps (Achin (¶0311 line 2-5), fitted model predicts one or more outcomes of the initial prediction problem. First accuracy score is obtained.);
generating a masked prediction for the timestep by applying the time-series model to a masked time-series data record comprising the time-series data record (Achin (¶0272 line 2-8), “includes a time series model that can predict the values of a target I at time t and optionally t+1, ..., t+i, given observations of I at times before t and optionally observations of other predictor variables P at times before t. In some embodiments, the predictive modeling system 100 partitions past observations to train a supervised learning model, measure its performance, and improve accuracy”) in which (A) a feature subset of the plurality of features is masked (Achin (¶0312 line 1-10, fig. 9 item 930, ¶0313 line 1-6, fig. 9 item 940), system shuffles the values of a feature F in in the initial dataset. Shuffling operation includes masking operations, such as assigning random values, reducing the predictive value or removing the feature F. The second accuracy score represents one or more outcomes of the modified predictive problem) for a window of timesteps in the sequence of timesteps (Achin (¶0280 line 1-6), “The user may indicate a "skip range" in the data, which is a gap between the end of a training window (e.g., a time range of data used for training) and the start of a validation window (e.g., a time range of data used for validation) or a holdout window (e.g., a time range of data used for holdout testing.” Achin (¶0286 line 1-4), “With the desired number of training and validation ranges and the length of these ranges plus the skip range, the engine 110 can divide up the dataset into a consistent series of training and validation ranges (time windows)”) and (B) other features are not masked in the window timesteps (Achin (¶0312 line 1-4), “the system 100 "shuffles" the values of a particular feature F across the observations in the initial dataset, thereby generating a modified dataset representing a modified prediction problem”, only feature F across the observations (all of the features) in the initial dataset is masked); and
determining a feature-window importance score describing an effect on
model prediction of the trained time-series model for the time series data record in the window of the feature subset based on a difference in the unmasked prediction and the masked prediction (Achin (¶0314 line 3-7 and last 4 lines, fig. 9 item 940, 940), “the predictive value of the feature F for a modeling procedure or model is calculated based on the change in accuracy (e.g., based on the difference between the first and second accuracy scores for model)”, “model-specific predictive values” (a feature-window importance score) are based on the first accuracy score and the second accuracy score).
As Claim 2, besides Claim 1, Achin teaches wherein the instructions are further executable for:
determining a second feature-window importance score based on a second masked prediction in which the feature subset is masked by the window except for an initial timestep of the window (Achin (¶0315 line 1-6, fig. 9 item 960), the system continues analyzing the predictive value of other features); and
determining a feature-step importance score describing the effect on model predictions in the window of the feature subset at the initial timestep based on a comparison of the feature window important score and the second feature window importance score (Achin (¶0319 line 5-10), system ranks the features by their predictive values).
As Claim 3, besides Claim 2, Achin teaches wherein the instructions are further executable for:
determining one or more additional feature-step importance scores (Achin (¶0319 line 5-10), system ranks the features by their predictive values) for the feature subset for timestep at windows of different lengths (Achin (¶0280 line 1-6), user defines “skip range” for a plurality of time windows) beginning at the initial time step (Achin (¶0315 line 1-6, fig. 9 item 960), the system continues analyzing the predictive value of other features); and
determining an aggregate feature importance score based on the feature-step importance score and the one or more additional feature-step importance scores (Achin (¶0318, ¶0319 line 5-10), system categorizes features as “more important” or “less important” if the features are on top highest N values), the aggregate feature importance score describing the importance of the feature subset at the timestep on model predictions at a plurality of time windows (Achin (¶0280 line 1-6), user defines “skip range” for a plurality of time windows).
As Claim 4, besides Claim 1, Achin teaches wherein the feature subset is masked with values sampled from a feature generator (Achin (¶0312 last 3 lines), system reduces predictive value of feature F below a threshold value).
As Claim 5, besides Claim 1, Achin teaches wherein the instructions are further executable for validating the model based on the feature-window importance score (Achin (¶0319 line 5-10), system ranks the features by their predictive values).
As Claim 6, besides Claim 1, Achin teaches wherein the instructions are further executable for retraining the time-series model based on the feature-window importance score (Achin (¶0286 last 3 lines, ¶0320 line 1-4), system reserves a time window for tuning model hyper-parameters and blended models. System selects models for blending based on the model-specific predictive values).
As Claim 7, besides Claim 1, Achin teaches wherein the instructions are further executable for determining a frequency to sample the feature subset based on the feature-window importance score (Achine (¶0325 line 3-17), system weights models based on model specific feature values).
As Claims 8-14, the Claims are rejected for the same reasons as Claims 1-7, respectively.
As Claims 15-20, the Claims are rejected for the same reasons as Claims 1-6, respectively.
Response to Arguments
Section 112 Rejection:
Applicant amended the Claims, therefore, 35 U.S.C. §112 rejections are respectfully withdrawn.
Section 102 Rejection:
As per Achin, Applicants argue that the reference does not disclose “generating a mask prediction” because the mappings use two different location (paragraph 312 relating to masking the data and paragraph 280 relating to “a window of time steps”) (last paragraph of page 10 in the remarks).
PNG
media_image1.png
173
661
media_image1.png
Greyscale
Applicants’ arguments are not persuasive. Achin (¶0280 line 1-6, “The user may indicate a "skip range" in the data, which is a gap between the end of a training window (e.g., a time range of data used for training) and the start of a validation window (e.g., a time range of data used for validation) or a holdout window (e.g., a time range of data used for holdout testing”) teaches that user can select a skip range or time windows within the whole set of time series data. Achin (¶0286 line 1-4, “With the desired number of training and validation ranges and the length of these ranges plus the skip range, the engine 110 can divide up the dataset into a consistent series of training and validation ranges (time windows)”) teaches that the system is trained using the user selected time windows (data outside of the skip range) as training dataset. Achin (¶0312 line 1-4, “the system 100 "shuffles" the values of a particular feature F across the observations in the initial dataset, thereby generating a modified dataset representing a modified prediction problem”) teaches that the training dataset is “shuffled” to created a masked training dataset. The two location references illustrates when the time windows are selected and when the training dataset is masked. However, only one training dataset is used throughout Achin’s disclosure.
As per Achin, Applicants argue that “skip range” cannot include one of the “window of timesteps in the sequence of time steps” (first paragraph of page 11 in the remarks).
PNG
media_image2.png
255
647
media_image2.png
Greyscale
Applicant’s arguments are not persuasive because “skip range” defines an exclusion of a “time window”. The other time windows surround this skip range are used as training dataset for the model. Achin (¶0280 line 1-6, “The user may indicate a "skip range" in the data, which is a gap between the end of a training window (e.g., a time range of data used for training) and the start of a validation window (e.g., a time range of data used for validation) or a holdout window (e.g., a time range of data used for holdout testing”) teaches that user can select a skip range or time windows within the whole set of time series data. Achin (¶0286 line 1-4, “With the desired number of training and validation ranges and the length of these ranges plus the skip range, the engine 110 can divide up the dataset into a consistent series of training and validation ranges (time windows)”) teaches that the system is trained using the user selected time windows (data outside of the skip range) as training dataset.
As per Achin, Applicants argue that current claims recite “other features are not masked in the window of time steps” (first paragraph of page 11 in the remarks).
Applicants’ arguments are not persuasive. Achin (¶0312 line 1-4, “the system 100 "shuffles" the values of a particular feature F across the observations in the initial dataset, thereby generating a modified dataset representing a modified prediction problem”, it is construed as the rest of the features are not “shuffled”) teaches that only feature F across the observations (all of the features) in the initial dataset is masked.
As per Achin, Applicants argue that Achin does not disclose “a feature-window importance score …” because (first) Achin does not evaluate the effect of features within a window (last paragraph of page 11 in the remarks).
Applicants’ arguments are not persuasive because Achin (¶0286 line 1-4, “With the desired number of training and validation ranges and the length of these ranges plus the skip range, the engine 110 can divide up the dataset into a consistent series of training and validation ranges (time windows)”) teaches that the system is trained using the user selected time windows (data outside of the skip range) as training dataset.
As per Achin, Applicants argue that Achin does not disclose “a feature-window importance score …” because (secondly) Achin does determine the extent to which masking of the feature in the window changes the model’s prediction (last paragraph of page 11 in the remarks).
Applicants’ arguments are not persuasive. Achin (¶0314 line 3-7 and last 4 lines, fig. 9 item 940, 940), “the predictive value of the feature F for a modeling procedure or model is calculated based on the change in accuracy (e.g., based on the difference between the first and second accuracy scores for model)”) teaches that “model-specific predictive values” (a feature-window importance score) are based on the first accuracy score and the second accuracy score. Achin (¶0286 line 1-4, “With the desired number of training and validation ranges and the length of these ranges plus the skip range, the engine 110 can divide up the dataset into a consistent series of training and validation ranges (time windows)”) teaches that the system is trained using the user selected time windows (data outside of the skip range) as training dataset. Therefore, the result of the evaluation is pertained to the specific selected time windows.
As per Achin, Applicants argue that Achin is contrast to the “claimed feature window important” which evaluate the significant of the window for the same model. Further on, Achin calculated accuracy scores are based on different information and used for different purposes (first paragraph of page 12 in the remarks)
Applicants’ arguments are not persuasive because Achin evaluates the effect of the feature within a user selected time windows for the inference model. Further clarifying on how claimed feature window important score are calculated might differentiate the claimed invention from Achin and advance the prosecution.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NHAT HUY T NGUYEN whose telephone number is (571)270-7333. The examiner can normally be reached M-F: 12:00-8:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at 571-270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NHAT HUY T NGUYEN/Primary Examiner, Art Unit 2147
Prosecution Insights