DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is responsive to claims filed 10/6/2025. Claims 1-8, 10-17, 19-20 are pending.
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 10/6/2025 has been entered.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The 35 U.S.C. 101 subject matter eligibility analysis first asks whether the claim is directed to one of the four statutory categories (Step 1). It next asks whether the claim is directed to an abstract idea (Step 2A), via Prong 1, whether an abstract idea (e.g., mathematical concept, mental process, certain methods of organizing human activity) is recited, and Prong 2, whether it is integrated into a practical application. It finally asks whether the claim as a whole includes additional elements that amount to significantly more than the judicial exception (Step 2B). See MPEP 2106.
STEP 1: The claims falls within one of the four statutory categories:
As all claims 1-20 are directed to methods, systems, and computer storage media that are not signals per se, the claims are statutory.
STEP 2A PRONG 1: The claims recite a judicial exception:
The claims are directed to a technique for analyzing data to determine concept drift by comparing various aspects of old and new data via mathematical calculations. As such, they are directed to a mathematical process. For example, for claim 1 (additional elements are underlined): a computer-implemented method, comprising:
obtaining, by one or more processors, a plurality of historical data items and a plurality of new data items, the plurality of historical data items being used for training a model, and the plurality of new data items being applied to the model (obtaining data for further analysis is a mental process, the fact that this data are historical data items and new data items or data for a training a model, such as a mental model that categorizes data, does not preclude this obtaining from occurring mentally or with the aid of pen and paper);
determining, by the one or more processors:
an overall difference determined by dividing a number of the new data items that are dissimilar from the historical data items by a total number of the new data items (determining that new data has substantially different properties by considering what proportion is different via determining a quotient is a mathematical process),
a structural difference based on a proportion of the new data items having a structure dissimilar from the historical data items, the structure being determined by projecting, respectively, the historical data items and the new data items from multiple dimensions into one dimension by applying a set of weights to each dimension to convert multi-dimensional data items into a single dimensional projection result that are target values, wherein the dimensions are the fields of the data items (Considering a weighted sum of tracked multivariate in determination of data drift is a mathematical concept), and
a confidence difference between the plurality of historical data items and the plurality of new data items, the confidence difference based on the difference between predicted values obtained by applying, respectively, the new data items and the historical data items to the model (comparing differences in model results, such as by determining a difference in output confidence levels by taking a difference, is a mathematical process); and
determining, by the one or more processors, an indication of whether to update the model based on a combination of the overall difference, the structural difference, and the confidence difference (combining various difference calculations in various ways, such as via a weighted sum, is a mathematical process).
Likewise, for claims 10 and 19: a system comprising:
one or more computer readable storage media with program instructions collectively stored on the one or more computer readable storage media; and
one or more processors configured to executed the program instructions to perform a method comprising:
obtaining a plurality of historical data items and a plurality of new data items, the plurality of historical data items being used for training a model, and the plurality of new data items being applied to the model (obtaining data for further analysis is a mental process, the fact that this data are historical data items and new data items or data for a training a model, such as a mental model that categorizes data, does not preclude this obtaining from occurring mentally or with the aid of pen and paper);
determining an overall difference based on a proportion of new data items that are dissimilar from the historical data items (determining that new data has substantially different properties by considering what proportion is different, such as via determining a quotient is a mathematical process),
determining a structural difference based on a proportion of the new data items having a structure dissimilar from the historical data items, the structure being determined by projecting, respectively, the historical data items and the new data items from multiple dimensions into one dimension by applying a set of weights to each dimension to determine target values (Considering a weighted sum of tracked multivariate in determination of data drift is a mathematical concept), and
determining a confidence difference between the plurality of historical data items and the plurality of new data items, the confidence difference based on the difference between predicted values obtained by applying, respectively, the new data items and the historical data items to the model (comparing differences in model results, such as by determining a difference in output confidence levels by taking a difference, is a mathematical process); and
determining an indication of whether to update the model based on the overall difference, the structural difference, and the confidence difference (Consideration of cumulative differences indication based on a combination of metrics, such as by taking a weighted sum, is a mathematical concept).
For claim 2: clustering, by the one or more processors, the plurality of historical data items into a first plurality of clusters (clustering data via clustering algorithms is a mathematical concept);
clustering, by the one or more processors, the plurality of new data items into a second plurality of clusters (As above, clustering is a mathematical algorithm);
determining, by the one or more processors, a first number of data items in the plurality of new data items (Selecting data for further computation is a mental process).
For claim 3: wherein determining the overall difference comprises:
selecting, by the one or more processors, a first set of new data items from the plurality of new data items, a cluster of each data item in the first set of new data items according to the second plurality of clusters being different from the first plurality of clusters (Selecting for consideration from clusters is a mental process);
determining, by the one or more processors, a second number of data items in the first set of data items (Selecting for consideration is a mental process); and
determining, by the one or more processors, the overall difference based on the first number and the second number (Comparing to determine a difference based on selected number is a mental process).
For claim 4: wherein determining the structural difference comprises:
selecting, by the one or more processors, a second set of new data items from the plurality of new data items, a cluster of each data item in the second set of new data items according to the second plurality of clusters being the same as a cluster of the first plurality of clusters (selecting for consideration from clustered data is a mental process);
selecting, by the one or more processors, a third set of new data items from the second set of new data items, a structure of each data item in the third set of new data items being different from structures of the plurality of historical data items (selecting from clustered data is a mental process);
determining, by the one or more processors, a third number of data items in the third set of new data items (selecting data is a mental process); and
determining, by the one or more processors, the structural difference based on the first number and the third number (determining difference based on evaluation of structure is a mental process, that of judgment or evaluation).
For claim 5: wherein selecting the third set of new data items comprises:
for each data item of the second set of new data items:
determining, by the one or more processors, a cluster of the data item (determining a clustering or grouping membership is a mental process);
selecting, by the one or more processors, a first set of historical data items from the plurality of historical data items, the first set of historical data items having been clustered into the cluster (selecting data from a group or cluster is a mental process);
determining, by the one or more processors, the target value of the data item and a target value distribution of the first set of historical data items (determining a desired value is a mental process, determining a desired distribution is a mental process); and
in accordance with a determination that the target value of the data item fails to fall within the target value distribution of the first set of historical data items, determining, by the one or more processors, that the data item is a data item in the third set of new data items (determining membership or accordance with some distribution is a mental process).
For claim 6: wherein determining the target value and the target value distribution comprises:
weighting, by the one or more processors, the first set of historical data items by the set of weights to determine the target value distribution of the first set of historical data items (Considering a weighted sum of tracked multivariate in determination of data drift is a mathematical concept).
For claim 7: wherein determining the confidence difference comprises:
selecting, by the one or more processors, a fourth set of new data items from the plurality of new data items, a cluster of each data item in the fourth set of new data items according to the second plurality of clusters being the same as a cluster of the first plurality of clusters (selecting items from a cluster is a mental process);
selecting, by the one or more processors, a fifth set of new data items from the fourth set of new data items, a confidence of each data item in the fifth set of new data items being different from confidences of the plurality of historical data items (selecting from a set based on a criteria, such as confidence, is a mental process, that of judgment, evaluation);
determining, by the one or more processors, a fourth number of data items in the fifth set of new data items (selecting from a set is a mental process); and
determining, by the one or more processors, the confidence difference based on the first number and the fourth number (determining a difference is a mental process, that of judgment, evaluation).
For claim 8: wherein selecting the fifth set of new data items comprises:
for each data item of the fourth set of new data items:
determining, by the one or more processors, a cluster of the data item (determining a grouping or cluster is a mental process);
selecting, by the one or more processors, a second set of historical data items from the plurality of historical data items, the second set of historical data items having been clustered into the cluster (selecting a subset from a group is a mental process);
determining, by the one or more processors, a confidence of the data item and a confidence interval of the second set of historical data items (determine confidence of a selection is evaluation or judgment, a mental process);
in accordance with a determination that the confidence of the data item fails to fall within the confidence interval of the second set of historical data items, determining, by the one or more processors, that the data item is a data item in the fifth set of new data items (selecting based on a confidence evaluation is a mental process).
STEP 2A PRONG 2: The claims do not integrate the exception into a practical application:
As shown above, for the claims 1-8, 10, 19, the additional elements comprise various computer hardware components: one or more processors, computer storage media containing instructions, etc. However, these consist of mere instructions to implement an abstract idea on a computer and as such does not meaningfully limit the practice of the abstract idea and hence does not constitute an integration into a practical application.
The remaining claims recite analogous systems and computer products and do not integration into a practical application for the same reasons.
STEP 2B: The claim as a whole do not include additional elements that amount to significantly more than the abstract idea:
For the independent claims 1-8, 10, 19, the cited computer hardware components (one or more processors, computer storage media containing instructions, etc.) are well-understood, routine and conventional (WURC) and hence does not constitute significantly more (2b).
The remaining claims recite analogous systems and computer products and do not amount to significantly more for the same reasons.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 10-15, 19-20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Nakano (US 20210365813 A1).
For claim 10, Nakano discloses: a system, comprising:
one or more computer readable storage media with program instructions collectively stored on the one or more computer readable storage media (fig.16:5400, 5500, 5800); and
one or more processors configured to execute the program instructions to perform a method (fig.16:5300) comprising:
obtaining a plurality of historical data items and a plurality of new data items, the plurality of historical data items being used for training a model, and the plurality of new data items being applied to the model (fig.1: 11,11D, 14, 14D, 0028, 0034: attributes of the historical training data and newly acquired training data are analyzed to determine whether retraining is necessary);
determining an overall difference based on a proportion of new data that are dissimilar from the historical data items (fig.11, 0078: determining number of data belonging to / within a standard deviation of new cluster O3, a drifted cluster of new data, is the same as the data of the clusters of the in operation model, then executing retraining, hence, determining a proportion of at least 1 between new data items dissimilar from historical data items),
determining a structural difference based on a proportion of the new data items having a structure dissimilar from the historical data items, the structure being determined by projecting, respectively, the historical data items and the new data items from multiple dimensions into one dimension by applying a set of weights to each dimension to determine target values (fig.13, 0080-84 contemplates comparing various metrics of the distribution of training data and new retraining data as a factor to determine if retraining is needed, the metrics including skewness, kurtosis, standard deviation, and variance; these metrics indicating that some proportion (e.g., 1 standard deviation of the data) of new data items is dissimilar from the historical data items, the structural difference being determined with regard to a particular feature, hence, the projection of the feature vector into the particular feature dimension, such as via a one-hot vector ([0 1 0 0 0 …]); see also fig.15 row 5, 0089 contemplating the application of the above technique (embodiment 4) to each cluster), and
determining a confidence difference between the plurality of historical data items and the plurality of new data items, the confidence difference based on the difference between predicted values obtained by applying, respectively, the new data items and the historical data items to the model (fig.2, 0029, 0034, 0037, fig.9, 0062: the difference between predicted value by applying the historical data items and by applying the new data items to the model is determined to determine a difference to determine a retraining necessity, hence, confidence difference); and
determining an indication of whether to update the model based on the overall difference, the structural difference, and the confidence difference (0086-88, 0079 considers determining retraining indication based on combination of above factors).
Regarding claim 11, Nakano discloses the method of claim 1, as described above. Nakano further discloses: clustering the plurality of historical data items into a first plurality of clusters (fig.11, 070-72, 0075-79: clustering data including historical data);
clustering the plurality of new data items into a second plurality of clusters (ibid: new data items are clustered, in N1-N2, O1-O3);
determining a first number of data items in the plurality of new data items (fig.11, 0078-79; fig.12, 0072-73: determining data items in new clusters O3 for various comparisons for accuracy prediction).
Regarding claim 12, Nakano discloses the method of claim 11, as described above. Nakano further discloses: wherein determining the overall difference comprises:
selecting a first set of new data items from the plurality of new data items, a cluster of each data item in the first set of new data items according to the second plurality of clusters being different from the first plurality of clusters (fig.11, 0078-79: selecting new data items in new drifted clusters for comparison);
determining a second number of data items in the first set of data items (0078-79: determining a number of data items in existing clusters of the in-operation model 101a for prediction model); and
determining the overall difference based on the first number and the second number (ibid: comparing the numbers to determine a difference metric for retraining considerations).
Regarding claim 13, Nakano discloses the method of claim 11, as described above. Nakano further discloses: wherein determining the structural difference comprises:
selecting a second set of new data items from the plurality of new data items, a cluster of each data item in the second set of new data items according to the second plurality of clusters being the same as a cluster of the first plurality of clusters (fig.13, fourth embodiment (0080-89) contemplates comparing distributions and features of retraining data (0036: retraining data may be newly collected data or a combination of new and old data) and old data (i.e., training data of the in-operation model) for each cluster (see fig.15, 0089), hence, selecting a cluster for comparison and Feature(s) for each cluster);
selecting a third set of new data items from the second set of new data items, a structure of each data item in the third set of new data items being different from structures of the plurality of historical data items (fig.13, 0081-85 contemplates selecting new data that may have different attributes such as statistical indices such as skewness, kurtosis, etc. (0083), or similarity / KL divergence (0084), hence, a selected third set may have different indices);
determining a third number of data items in the third set of new data items (ibid: a third number of data items is determined for calculating the above similarity or statistical indices); and
determining the structural difference based on the first number and the third number (0085-86: determining that retraining criteria is met, i.e., that structural difference is significant based on both third number and a first number comprising a number of retraining data).
Regarding claim 14, Nakano discloses the method of claim 13, as described above. Nakano further discloses: wherein selecting the third set of new data items comprises:
for each data item of the second set of new data items:
determining a cluster of the data item (fig.15 row 5, 0089: determining a cluster for distribution comparisons of fig.13, embodiment 4);
selecting a first set of historical data items from the plurality of historical data items, the first set of historical data items having been clustered into the cluster (fig.13, 0082-85: selecting historical data items for calculating distribution);
determining the target value of the data item and a target value distribution of the first set of historical data items (ibid: target indices or similarities metrics are determined for a new data item and for the historical distribution based on their respective values); and
in accordance with a determination that the target value of the data item fails to fall within the target value distribution of the first set of historical data items, determining, by the one or more processors, that the data item is a data item in the third set of new data items (ibid: based on a comparison between the new and the old indices or metrics, determining that the third set is part of the overall heuristics, including whether a first set of numbers of retraining data is within a predetermined range, to trigger a retraining).
Regarding claim 15, Nakano discloses the method of claim14, as described above. Nakano further discloses: wherein determining the target value and the target value distribution comprises:
weighting the first set of historical data items by the set of weights to determine the target value distribution of the first set of historical data items (fig.13, 0081-82: using one-hot weights (e.g., [0 1 0 0 ..]) to extract scalars values for features to determine value distribution).
Claims 10-15, 19-20 recite systems and products analogous to the above methods and hence are rejected under the same rationale.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 7-8, 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Nakano (US 20210365813 A1) in view of Raz (US 20200242505 A1).
Regarding claim 16, Nakano discloses the method of claim 2, as described above. Nakano further discloses: wherein determining the confidence difference comprises:
selecting a fourth set of new data items from the plurality of new data items, a cluster of each data item in the fourth set of new data items according to the second plurality of clusters being the same as a cluster of the first plurality of clusters (fig.13, fourth embodiment (0080-89) contemplates comparing distributions and features of retraining data (0036: retraining data may be newly collected data or a combination of new and old data) and old data (i.e., training data of the in-operation model), with fig.15 row 5, 0089 contemplating application to particular clusters, such as shown in fig.11);
selecting a fifth set of new data items from the fourth set of new data items, a metric of each data item in the fifth set of new data items being different from metrics of the plurality of historical data items (fig.13, 0081-85 contemplates selecting new data that may have different attributes such as statistical indices such as skewness, kurtosis, etc. (0083), or similarity / KL divergence (0084), hence, a selected set may have different indices);
determining a fourth number of data items in the fifth set of new data items (ibid: a fourth number of data items is determined for calculating similarity or statistical indices); and
determining the metric difference based on the first number and the fourth number (0085-86: determining that retraining criteria is met, i.e., that structural difference is significant based on both fourth number and a first number comprising a number of retraining data).
Nakano does not disclose: wherein the metric is a confidence; wherein the metric difference is a confidence difference.
Raz discloses: wherein the metric is a confidence; wherein the metric difference is a confidence difference (fig.1, 0016-18: determining a confidence level of new production data and old training data to determine whether a drift has occurred (0006)).
It would have been obvious before the effective filing date to one of ordinary skill in the art to modify the method of Nakano by incorporating the confidence drift detection technique of Raz. Both concern the art of drift detection in machine learning, and the incorporation would have, according to Naz, help improve statistical detection of drift based on already existing metrics (0016-17, 0003-4).
Regarding claim 17, Nakano modified by Raz discloses the method of claim 7, as described above. Nakano modified by Raz further discloses: wherein selecting the fifth set of new data items comprises:
for each data item of the fourth set of new data items:
determining a cluster of the data item (Nakano fig.15 row 5, 0089: determining a cluster for calculating metrics);
selecting a second set of historical data items from the plurality of historical data items, the second set of historical data items having been clustered into the cluster (Nakano fig.13, 0080-83: selecting historical data items of the same cluster for determining historical metrics; Raz 0020: determining distribution of the historical training data);
determining a confidence of the data item and a confidence interval of the second set of historical data items (Raz 0020-21 : determining confidence distribution and interval of new production data to determine quantitative difference);
in accordance with a determination that the confidence of the data item fails to fall within the confidence interval of the second set of historical data items, determining, by the one or more processors, that the data item is a data item in the fifth set of new data items (Raz 0020-21: determination of differences in confidence intervals, i.e., that the data does not fall within the same confidence intervals, determining a measure of drift; 0085-86: based on a comparison between the new and the old indices or metrics, determining that the fifth set is part of the overall heuristics, including whether a first set of numbers of retraining data is within a predetermined range, to trigger a retraining).
Response to Arguments
Applicant’s arguments have been fully considered. Applicant argues:
1. Regarding the 101 rejections:
a. The recited claims may not be practically performed in the mind.
Based on the amendments, Examiner submits that the claims are directed to a mathematical concept. In general, the claims are directed to a technique of determining whether to update a model based on a data drift analysis. This analysis may be performed via a proportion calculation of the new data, via a projection of features of the new data, and via a confidence difference calculation. As all these are mathematical concepts used to generate a mathematical evaluation of the data and the model, the claims are directed to a mathematical concept.
b. The updating of a model improves a computer or a technological field by not expending resources when a model update is not necessary.
Examiner respectfully disagrees. The claims are directed to the improvement of a technique for detecting data drift which is not tied to any particular model or field. Although such a process may perform more optimally on a computer, such an association comprise mere instructions to operate the improved detection technique on a computer and hence does not meaningfully limit the practice of the abstract idea and hence does not constitute an integration into a practical application.
2. Regarding the prior art rejections:
a. The art of record does not disclose the newly amended limitations.
Examiner agrees and the rejections are withdrawn for the newly amended claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Healy ("A note on multivariate CUSUM procedures", published 1987) eq.3-4 (p.410 col.2) discloses a CUSUM calculation that incorporates a linear combination of multivariate normal distributions.
Yamane (US 20220270115 A1) 0070-72 discloses determining concept drift via determination of outlier proportion.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LIANG LI whose telephone number is (303)297-4263. The examiner can normally be reached Mon-Fri 9-12p, 3-11p MT (11-2p, 5-1a ET).
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Jennifer Welch can be reached on (571)272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center or Private PAIR to authorized users only. Should you have questions about access to Patent Center or the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. The examiner is available for interviews Mon-Fri 6-11a, 2-7p MT (8-1p, 4-9p ET).
/LIANG LI/
Primary examiner AU 2143