DETAILED ACTION
This action is responsive to Applicant’s reply filed 19 May 2026. This action is made non-final.
Status of the Claims
Claims 1-18 and 20 are amended.
Claim 19 is canceled.
Claim 21 is added.
Claim status is currently pending and under examination for claims 1-18 and 20-21 of which independent claims are 1, 9 and 16.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on September 29, 2025 has been entered.
Response to Amendment
Applicant’s amendments to the Claims have overcome each and every 101 rejections previously set forth in the Final Office Action mailed February 19th 2026.
On Page 13 of Applicant’s response, applicant argues that the Examiner has failed to establish a
proper prima facie case of obviousness because Agarwal does not teach an inference data set or a validating data set. Applicant’s arguments are not persuasive. The Examiner notes that the applicant provides no meaningful definitions of these terms in their specification.
Accordingly, the Applicant’s written description at [0022] describes an inference data set “includes inference input data for input into the inferential model”, and a validating data set “includes validating input data for input into the inferential model”.
Applicant’s written description at [0120] describes an inference data set and validating data set are collected under different conditions, “the inference data set is collected under a first condition represented in the inference data set and the validating data set is collected under a second condition represented in the validating data set, the first condition and the second condition at least partially differ”.
In the previous office action, the Examiner mapped an inference data set as “training data” and a validating data set as “live data”. Under the broadest reasonable interpretation in view of Applicant’s specification at [0022, 0120], the Examiner interprets an “inference data set” as encompassing training data as disclosed by Agarwal. Training data is used to train a model and training data is collected under a different condition than live data since training data and live data have different distributions (see P. 125, Last Paragraph of Agarwal). Therefore, “training data” is inference input data used as input into a model that performs inference, and training data is collected under a different condition than live data (and therefore “training data” is an “inference data set”).
Under the broadest reasonable interpretation in view of Applicant’s specification at [0022, 0120], the Examiner interprets “validating data set” as encompassing live data as disclosed by Agarwal. Live data is used as input into a model and its distribution distance is compared to determine drifts (see P. 125, Last Paragraph), therefore live data is used to validate a model since drifts determine model degradation (model validation). Live data is collected under a different condition than training data since training data and live data have different distributions. Therefore, “live data” is validating input data for input into an inferential model, and “live data” is collected under a different condition than training data (and therefore “live data” is a “validating data set”).
On Pages 13-14 of Applicant’s response, applicant argues that the Examiner has failed to establish a proper prima facie case of obviousness because Agarwal does not teach a “validating data set previously used to validate, in a first validation operation, the trained inferential model”. Applicant’s argument is not persuasive since multiple distribution drift checks occur during a period of 30 days using the live data (validation data set), as depicted in Figure 7.2 on P. 128 by Agarwal. On each day, covariate drift is determined by calculating the distance between the distributions of live data and training data. Drift level is used to determine if significant drift occurred, therefore validating a trained model in a validation operation (see P. 127, Last Paragraph). Since distributions are checked daily (validation is iterative), distributions are compared before a validation operation (the previous day’s distribution check) and after a validation operation (the following day’s distribution check), therefore live data is a validating data set previously used to validate, in a first validation operation, the trained inferential model.
On Pages 13-14 of Applicant’s response, applicant argues that the Examiner has failed to establish a proper prima facie case of obviousness because Agarwal’s trained model is not “trained using a training data set different from the validating data set”. Applicant’s arguments are not persuasive since the training data (“inference data set”) used to the train the model is different than the live data (“validating data set”) used to compare distributions. The training data and the live data have different distributions (see P. 132, First Paragraph), therefore the training data and live data must be different.
On Page 15 of Applicant’s response, applicant argues that Agarwal does not teach “outputting, via a communication interface, a model-validity determination indicating whether the trained inferential model is validated for operation on the inference data set”. Applicant’s argument is not persuasive because generated alerts indicating drift level are model-validity determinations. Calculating the distance between distributions of live data and training data to determine drift level is model validation, as severe drift is indicative of model degradation (see P. 124, ¶ 3). Therefore, drift level determines if a model remains valid, and therefore generated alerts are model-validity determinations that indicate if a trained model is validated for operation on an inference data set.
Applicant’s remaining arguments regarding the art rejections are moot in view of the new grounds of rejection necessitated by Applicant’s amendment.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The following are the references relied upon in the rejections below:
Agarwal, S., Mishra, S. (2021). Accountability in AI. In: Responsible AI. Springer, Cham. https://doi.org/10.1007/978-3-030-76860-7_7
Lee, Wonju, et al. "Unsupervised model drift estimation with batch normalization statistics for dataset shift detection and model selection." arXiv preprint arXiv:2107.00191 (2021).
Yi, Zhengkun. "Discriminative dimensionality reduction for sensor drift compensation in electronic nose: A robust, low-rank, and sparse representation method." Expert systems with applications 148 (2020): 113238.
Phillippe et al. (US 20230132501 A1)
Claims 1-2, 5-6, 9-10, 13-14 and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Agarwal et al. (“Accountability in AI”), hereinafter Agarwal.
With respect to claim 1, Agarwal teaches:
A method of validating a trained inferential model for operation on an unlabeled inference data set, the method comprising (The Examiner interprets “inference data set” according to its broadest reasonable interpretation in view of the applicant’s specification as encompassing training data. This interpretation is consistent with the descriptions in the Applicant’s specification at [0022] and [0120], (see excerpts below).
Applicant’s written description at [0022] “The inference data set 116 includes inference input data for input into the inferential model”
Applicant’s written description at [0120] “the inference data set is collected under a first condition represented in the inference data set and the validating data set is collected under a second condition represented in the validating data set, the first condition and the second condition at least partially differ”
Agarwal discloses “to be able to monitor a model’s performance, the first and the foremost challenge is quantifying the model degradation. Identifying the parameters to track the model performance and defining the thresholds that if breached should raise an alert are fundamental components of model monitoring. … Eventually a model that exhibits any kind of degradation will need to be examined further either for recalibration, retraining or, in the worst case, replacement” (P. 124, ¶ 1-3).
Agarwal discloses “the data distribution of the live data shifts over a period of time. This is known as data drift (or model drift), and if left unattended, it leads to performance degradation of the model. … Another common reason for the poor performance on the production environment is production skew – which is the difference in the model performance between training and production environments. Production skew can happen because of errors in training, bugs in the production environment or because the training data and the live data do not follow the same distribution” (P. 123, ¶2).
Agarwal discloses “we just need to monitor the distribution of the live input data and compare it with the training data to identify any drift. This does not require us to know the ground truth (actual outcome)” (P. 125, Last Paragraph).
The distributions of training data (‘unlabeled inference data set’) and live data are compared for data shifts to determine if model retaining is required. Therefore, when comparing the training data and live data distributions to determine if a shift has occurred, a trained model is validated on an unlabeled training data set since ground truth (labels) is not known. The training data is different from the live data because they do not follow the same distribution, therefore training data is collected under a different condition than live data (validating data set), and therefore training data is an inference data set.):
extracting a first distribution of values of a first parameter from the unlabeled inference data set (Agarwal discloses a programming function used to calculate a covariate drift between a training data set (‘unlabeled inference data set’) and a live data set (‘validating data set’) on P. 126 (reproduced below). The function obtains (‘extracts’) distributions from the training data set of variables (‘parameters’) that are present in both the training data set and the live data set.
PNG
media_image1.png
483
1797
media_image1.png
Greyscale
Agarwal further discloses “each independent variable is binned to form i bins (commonly 20 equal bins) from both the actual distribution and live distribution taken together. Then, the shift in the variable contribution to each bin is calculated. The function below picks the common variables between the two datasets and calls the calculate_distance function to get the values for each variable” (P. 126, ¶2-3).);
extracting a second distribution of values of the first parameter from a validating data set previously used to validate, in a first validation operation, the trained inferential model (The Examiner interprets “validating data set” according to its broadest reasonable interpretation in view of the applicant’s specification as encompassing live data. This interpretation is consistent with the descriptions in the Applicant’s specification at [0022] and [0120], (see excerpts below).
Applicant’s written description at [0022] “The validating data set 114 includes validating input data for input into the inferential model”
Applicant’s written description at [0120] “the inference data set is collected under a first condition represented in the inference data set and the validating data set is collected under a second condition represented in the validating data set, the first condition and the second condition at least partially differ”
(P. 125, Last Paragraph) “we just need to monitor the distribution of the live input data and compare it with the training data to identify any drift”
Agarwal discloses the calculate_covariate_drift() programming function on P. 126. The function obtains (‘extracts’) distributions from the live data (‘validating data set’) of variables (‘parameters’) that are present in both the training data and the live data. The live data (that is used as input) has a different distribution than the training data (inference data set), therefore, the live data set is collected under a different condition than the training data set (and therefore the live data set is a validating data set since it used for detecting model drifts (model validation)).
(P. 127, Last Paragraph) “Continuing with the same dataset, for the feature “Applied Amount”, the covariate drift over 30 days (Fig. 7.2) using the above concept shows a need for investigation as the drift is close to the High Drift level on a large number of days”
Agarwal discloses Figure 7.2 on P. 128 (reproduced below) depicting covariate drifts calculated for the feature (‘first parameter’) Applied Amount for a period of 30 days. Each covariate drift measures the difference in distributions between the training data set (inference data set) and the live data set (‘validating data set’) for a specific day. Drift level is used to determine if significant drift occurred, therefore validating a trained model in a validation operation. Since distributions are checked daily (validation is iterative), distributions are compared before a validation operation (the previous day’s distribution check) and after a validation operation (the following day’s distribution check), therefore live data is a validating data set previously used to validate, in a first validation operation, the trained inferential model.
PNG
media_image2.png
636
1096
media_image2.png
Greyscale
),
the trained inferential model being trained using a training data set different from the validating dataset (Agarwal discloses “Validity of a model is highly dependent on the similarity between the data distribution on which it is trained and the live data on which it makes its predictions. As the live data distribution changes, the validity of the model can come under the scanner” (P. 132, First Paragraph).
A model is trained on a training data set that has a different data distribution than that of a live data set (‘validating dataset’). Therefore, since the training data set and the live data set have different data distributions, the datasets must be different from each other.);
determining a first parameter distance between the extracted first distribution and the extracted second distribution … (Agarwal discloses the programming function calculate_distance() on P. 126 (reproduced below) is used to calculate a distance (‘first parameter distance’) between two distributions, “the function [calculate_covariate_drift] picks the common variables between the two datasets and calls the calculate_distance function to get the values for each variable. We begin by sorting the distributions by data rank and then creating bins of equal sizes. For a given feature, the percentage of observations falling into each bin is computed separately for the training and live data. The distance calculation after that is straightforward – the sum of the minimum percentage across each bin is calculated and then subtracted from 1” (P. 126, Last Two Paragraphs).
PNG
media_image3.png
695
1763
media_image3.png
Greyscale
);
and validating, in a second validation operation without using labeled ground-truth output data for the unlabeled inference data set, the trained inferential model for operation on the unlabeled inference data set based on satisfaction of a validation condition, (Agarwal discloses programming function cal_threshold() on P. 217 (reproduced below) determines if an alert should be raised to indicate the severity of a data drift.
PNG
media_image4.png
389
1412
media_image4.png
Greyscale
See (P. 125, Last Paragraph) describing comparing distributions between training data and live data does not require knowing the ground truth of the training data (‘unlabeled inference data set’). Therefore, distances obtained from calculate_covariate_drift() do not use labeled ground-truth output data.
Agarwal further discloses “using a predefined threshold, alerts are generated by passing the distance calculated above for each batch of new or live data.” (P. 127, First Paragraph).
Agarwal discloses covariate drifts (‘validation operations’) close to the High Drift level indicate a model investigation is needed, “Continuing with the same dataset, for the feature “Applied Amount”, the covariate drift over 30 days (Fig. 7.2) using the above concept shows a need for investigation as the drift is close to the High Drift level on a large number of days” (P. 127, Last Paragraph).),
including outputting, via a communication interface, a model-validity determination indicating whether the trained inferential model is validated for operation on the inference data set ((P. 124, ¶ 3) “You may choose to leverage most of these techniques as part of model monitoring, but at the minimum, it is imperative to have a process for detecting, alerting and addressing any kind of drift that may occur post go live. Eventually a model that exhibits any kind of degradation will need to be examined further either for recalibration, retraining or, in the worst case, replacement”
(P. 127, First Paragraph) “using a predefined threshold, alerts are generated by passing the distance calculated above for each batch of new or live data”
Calculating the distance between distributions of live data and training data to determine drift level is model validation, as severe drift is indicative of model degradation. Therefore, drift level determines if a model remains valid, and therefore generated alerts depicting No Drift / Low Drift / Medium Drift / High Drift are model-validity determinations that indicate if a trained model is validated for operation on an inference data set. A communication interface is implied by generating alerts.),
the satisfaction of the validation condition being based on the determined first parameter distance (The cal_threshold() function uses distances obtained from comparing distributions between training data and live data, and compares the distances against thresholds to determine data shift severity.).
However, Agarwal does not teach statistically measuring a similarity between the extracted first distribution and the extracted second distribution, which in a different embodiment, Agarwal teaches:
determining a first parameter distance between the extracted first distribution and the extracted second distribution by statistically measuring a similarity between the extracted first distribution and the extracted second distribution (The Examiner interprets “parameter distance” according to its broadest reasonable interpretation in view of the applicant’s specification as encompassing covariate drift. This interpretation is consistent with the descriptions in the Applicant’s specification at [0011] and [0033], (see excerpts below).
Applicant’s written description at [0011] “One source of drift is a covariate shift (also called input or feature drift). Covariate shift is a deviation within input variables of time-series data.”
Applicant’s written description at [0033] “The data set distance model validator 118 uses a Kolmogorov-Smirnoff test (K-S test) to determine distances between distributions of parameter values with continuous values (or values of continuous variables sampled discretely).”
(P. 136, Last Paragraph) “KS stats or Kolmogorov-Smirnov divergence is one of those metrics which is used to determine the ability of a model to distinguish between events and non-events, by comparing the two cumulative distributions (event and non-event) and returns the maximum difference (distance) between them”
(P. 137, ¶2) “For monitoring purpose, the model should have similar KS divergence for actual and predicted distributions. The p-value, returned by the KS stats, of more than 0.1 is statistically significant and indicates that the model performance needs to be investigated”).
Agarwal teaches using Kolmogorov-Smirnov divergence to determine distribution similarity is a known method in the art. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to combine the method of Agarwal with the Kolmogorov-Smirnov divergence disclosed by Agarwal to use Kolmogorov-Smirnov divergence to determine distribution similarity. By using Kolmogorov-Smirnov divergence to determine distribution similarity, it can be determined if distribution similarity is statistically significant, thereby ensuring that differences between distributions are not due to random chance.
With respect to claim 2, Agarwal teaches:
the method of claim 1, wherein the unlabeled inference data set is collected under a first condition represented in the unlabeled inference data set and the validating data set is collected under a second condition represented in the validating data set, the first condition and the second condition at least partially differ (Agarwal discloses “Covariate drift is the change in the distribution of the input feature set … This is one of the most common causes of model drifts, and the distribution drift can happen due to many reasons. The drifts can happen slowly over a period of time … A lot of time drift in data distribution occur because of latent changes, macro-economic changes or demographic changes … As the covariate drift happens, the distribution of the live data shifts from that of the data used for the training and testing. The distance between the non-intersection of the two distributions is a very good measure of the drift. … P and Q are training and live distributions, respectively. The larger the distance, the bigger is the drift” (P. 126, ¶ 1-2).
It is implied training data (‘inference data set’) and live data (‘validating data set’) are collected during different periods of times (‘conditions’) since the live data consists of new incoming data. Comparing two datasets with differing distributions also implies that data sets were collected during different conditions.),
and satisfaction of the validation condition is based on the first condition and the second condition (The Examiner interprets the satisfaction of the validation condition as encompassing the exact same step as the validation step in Claim 1.).
With respect to claim 5, Agarwal teaches the method of claim 1. Furthermore, Agarwal also teaches the further limitations in a different embodiment:
wherein the first parameter represents output from the trained inferential model (Agarwal discloses “the same concept can be even replicated for the outcome – the predicted probabilities. The same techniques applied on the predicted probabilities is called prior probability shift (Fig. 7.3). This refers to the change in the distribution of the target variable in the training data and the live data. The target variable is binned to form 𝑖 bins from both the distribution. Then, the shift in the variable contribution to each bin is calculated” (P. 128, Last Paragraph).),
the extracted first distribution including a distribution of inference output parameter values output from the trained inferential model responsive to input of inference input data from the unlabeled inference data set into the trained inferential model (A distribution is obtained from the predicted probabilities (‘inference output parameter values’) generated using training data (‘inference data set’) for a target variable (‘first parameter’). Agarwal discloses “the same concept can be even replicated for the outcome – the predicted probabilities. The same techniques applied on the predicted probabilities is called prior probability shift (Fig. 7.3). This refers to the change in the distribution of the target variable in the training data and the live data. The target variable is binned to form 𝑖 bins from both the distribution. Then, the shift in the variable contribution to each bin is calculated” (P. 128, Last Paragraph).),
the extracted second distribution including a distribution of validating output parameter values output from the trained inferential model responsive to input of validating model input data from the validating data set into the trained inferential model (A distribution is obtained from the predicted probabilities (‘validating output parameter values’) generated using live data (‘validating data set’) for a target variable (‘first parameter’). Agarwal discloses “the same concept can be even replicated for the outcome – the predicted probabilities. The same techniques applied on the predicted probabilities is called prior probability shift (Fig. 7.3). This refers to the change in the distribution of the target variable in the training data and the live data. The target variable is binned to form 𝑖 bins from both the distribution. Then, the shift in the variable contribution to each bin is calculated” (P. 128, Last Paragraph).).
Agarwal teaches detecting for drifts in the distribution of predicted probabilities (‘output parameter values’) is a known method in the art. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the method of Agarwal with the method disclosed by Agarwal for detecting drifts in the distribution of outcomes because predicted outcomes can be biased and unfair. Predictions that do not reflect the real-world prevalence of outcomes are biased and unfair, and by detecting these biases, machine learning engineers can develop ways to remove them and make their model more fair.
With respect to claim 6, Agarwal teaches:
the method of claim 1, wherein the first parameter represents a metadata parameter, the extracted first distribution including a distribution of values of the metadata parameter from the unlabeled inference data set, and the extracted second distribution including a distribution of values of the metadata parameter of the validating data set (The Examiner interprets “metadata parameter” according to its broadest reasonable interpretation (in view of the Applicant’s specification at Paragraph 0055) as being a Kullback-Leibler divergence and a Covariate drift describing the distributions of two datasets being compared.
Agarwal discloses “mathematically, given two population sets P and Q (indicating the trained data and the live data, respectively), the stability index can be defined using Kullback-Leibler divergence (DKL). To over the issue of its being non-symmetric, the sum of K-L divergence … is used as the base metric” (P. 133, First Paragraph).
To compute the Kullback-Leibler divergence, the distributions of populations sets P (‘inference data set’) and Q (‘validating data set’) must be known, therefore the descriptions that describe these distributions are the metadata for these population sets.
Agarwal further discloses “Covariate drift is the change in the distribution of the input feature set … As the covariate drift happens, the distribution of the live data shifts from that of the data used for the training and testing. The distance between the non-intersection of the two distributions is a very good measure of the drift” (P. 126, ¶ 1-2).
To compute the Covariate drift, the distributions of training data (‘inference data set’) and live data (‘validating data set’) must be known, therefore the descriptions that describe these distributions are the metadata for these data sets.).
With respect to claim 9, the rejection of claim 1 is incorporated. The difference in scope being:
A system, comprising: one or more hardware processors configured to execute instructions stored in memory (A computer is implied by calculating the covariate drift for a data set as disclosed in Figure 7.2 on P. 128 by Agarwal. A hardware processor configured to execute instructions stored in memory is further implied by the use of a computer.);
and a data set distance model validator executable by the one or more hardware processors, the data set distance model validator including (The Examiner interprets a data set distance model validator according to its broadest reasonable interpretation as encompassing the programming instructions used to detect covariate drifts disclosed above in claim 1.).
With respect to claims 10 and 17, the claims recite similar limitations corresponding to claim 2, therefore the same rationale of rejection is applicable.
With respect to claim 13, the claim recites similar limitations corresponding to claim 5, therefore the same rationale of rejection is applicable.
With respect to claim 14, the claim recites similar limitations corresponding to claim 6, therefore the same rationale of rejection is applicable.
With respect to claim 16, the rejection of claim 1 is incorporated. The difference in scope being:
One or more tangible processor-readable storage media embodied with instructions for executing on one or more processors and circuits of a computing device a process of validating a trained inferential model for operation on an inference data set, the process comprising (A computer is implied by calculating the covariate drift for a data set as disclosed in Figure 7.2 on P. 128 by Agarwal. A tangible processor-readable storage media embodied with instructions for executing on one or more processors and circuits of a computing device is further implied by the use of a computer.).
Claims 3-4, 11-12, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Agarwal in view of Lee et al. (“Unsupervised model drift estimation with batch normalization …”), hereinafter Lee.
With respect to claim 3, Agarwal teaches: the method of claim 1, however Agarwal does not teach normalizing a distance or applying a weight, which is taught by Lee:
further comprising: normalizing the determined first parameter distance to generate a normalized first parameter distance (Lee discloses Equation 4 on P. 4 (reproduced below) for calculating a drift score (‘first parameter distance’) between a target data set (‘validation set;of a batch normalization layer.
PNG
media_image5.png
311
1432
media_image5.png
Greyscale
Lee discloses “Dist(a, b) is a distance metric between vectors a and b. From the Equation 4, the right element of distance metric contains the information of the source in the BN layer and the left element that contains the information of the target, and hence we can implicitly compute the discrepancy of the source dataset from target dataset using only model parameters” (P. 4, Sec. 3.2, First Paragraph).
Lee further discloses a drift score can be normalized by using a cosine distance, “Gaussian random variable, the drift score (4) of BN layer l can be computed with conventional distance metrics such as cosine distance
C
o
s
D
i
s
t
(
a
,
b
)
=
(
1
-
a
·
b
/
|
|
a
|
|
|
|
b
|
|
)
/
2
, which is bounded in [0, 1]” (P. 4, Sec. 3.2, Last Paragraph).),
the normalized first parameter distance normalized to a predetermined range of values (Lee discloses “Gaussian random variable, the drift score (4) of BN layer l can be computed with conventional distance metrics such as cosine distance
C
o
s
D
i
s
t
(
a
,
b
)
=
(
1
-
a
·
b
/
|
|
a
|
|
|
|
b
|
|
)
/
2
, which is bounded in [0, 1]” (P. 4, Sec. 3.2, Last Paragraph).);
and applying a relative weight to the normalized first parameter distance to generate a weighted first parameter distance (Lee discloses Equation 5 (reproduced below) on P.4 for calculating overall model discrepancy. The equation applies a weight to a calculated drift score (‘first parameter distance’).
PNG
media_image6.png
315
1428
media_image6.png
Greyscale
Lee discloses “overall model discrepancy can be calculated by taking average of all layers … where weight
w
(
l
)
∈
[
0
,
1
]
indicates the relative importance of BN layer l compared to others. The weights can be set proportional to magnitude of gradient during training phase [17] or ratio the L2-norm of weight and gradient” (P. 4, Sec. 3.2, First Paragraph).),
the relative weight based on a predetermined correlative value of the first parameter relative to a second parameter represented in the unlabeled inference data set and the validating data set (Lee discloses a weight can be set proportional to magnitude of gradient during training (‘predetermined correlative value of the first parameter’), “overall model discrepancy can be calculated by taking average of all layers … where weight
w
(
l
)
∈
[
0
,
1
]
indicates the relative importance of BN layer l compared to others. The weights can be set proportional to magnitude of gradient during training phase [17] or ratio the L2-norm of weight and gradient” (P. 4, Sec. 3.2, First Paragraph). A weight is then applied to a drift score calculated using model parameters that represents the discrepancy (relationship) between a source data set (‘inference data set’) and a target data set (‘validating data set’).),
wherein the satisfaction of the validation condition is based on the weighted first parameter distance (Lee discloses that drift scores are used to select a model with the smallest drift, “Figure 1 shows an example of practical uses of the MDE [(model drift estimation)] where the dataset shift occurs every 30 epoch. When dataset shift happens, the drift score of MDE bounces upward and we select the model having the smallest drift from 20 different model candidates, which are trained with different datasets, for automatic recovery from dataset shift” (P. 2, Sec. 1, Last Paragraph). See also Figure 4 on P. 8 depicting the drift scores calculated for each model by using a target data set.).
Lee teaches calculating a normalized and weighted drift score (‘parameter distance’) to determine dataset drift is a known method in the art. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the method of Agarwal with the drift score disclosed by Lee to remove biases. Normalizing and weighing a score would give more importance to parameters that significantly affect drift and ensures that all scores are on the same scale, thereby creating fair comparisons and reducing biases. By reducing or removing biases, more accurate and fair machine learning models can be developed and used to make fair decisions.
With respect to claim 4, Agarwal teaches:
the method of claim 1, further comprising: extracting a third distribution of values of a second parameter from the unlabeled inference data set (Agarwal discloses the programming function calculate_covariate_drift() used to calculate a covariate drift between a training data set (‘inference data set’) and a live data set (‘validating data set’) on P. 126 (reproduced above). The function obtains (‘extracts’) distributions from the training data set of each variable (‘parameter’) that is present in both the training data set and the live data set.);
extracting a fourth distribution of values of the second parameter from the validating data set (Agarwal discloses the calculate_covariate_drift() programming function on P. 126 (reproduced above). The function obtains (‘extracts’) distributions from the live data set (‘validating data set’) of each variable that is present in both the training data set and the live data set.);
determining a second parameter distance between the extracted third distribution and the extracted fourth distribution (Agarwal discloses the programming function calculate_distance() on P. 126 (reproduced above) is used to calculate a distance (‘second parameter distance’) between two distributions, “the function [calculate_covariate_drift] picks the common variables between the two datasets and calls the calculate_distance function to get the values for each variable. We begin by sorting the distributions by data rank and then creating bins of equal sizes. For a given feature, the percentage of observations falling into each bin is computed separately for the training and live data. The distance calculation after that is straightforward – the sum of the minimum percentage across each bin is calculated and then subtracted from 1” (P. 126, Last Two Paragraphs). (Emphasis added).);
However, Agarwal does not teach determining an aggregate distance, which is taught by Lee:
and determining an aggregate distance based on the determined first parameter distance and the determined second parameter distance (Lee discloses Equation 5 (reproduced above) on P.4 for calculating overall model discrepancy. The equation averages (‘aggregates’) all the drift scores (‘parameter distances’) calculated for each layer to find the overall model drift. Lee discloses “overall model discrepancy can be calculated by taking average of all layers … where weight
w
(
l
)
∈
[
0
,
1
]
indicates the relative importance of BN layer l compared to others” (P. 4, Sec. 3.2, First Paragraph).),
wherein the satisfaction of the validation condition is based on the aggregate distance (Lee discloses that drift scores are used to select a model with the smallest drift, “Figure 1 shows an example of practical uses of the MDE [(model drift estimation)] where the dataset shift occurs every 30 epoch. When dataset shift happens, the drift score of MDE bounces upward and we select the model having the smallest drift from 20 different model candidates, which are trained with different datasets, for automatic recovery from dataset shift” (P. 2, Sec. 1, Last Paragraph).).
Lee teaches calculating an aggregate drift score (‘aggregate distance’) to determine dataset drift is a known method in the art. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the method of Agarwal with the aggregate drift score disclosed by Lee to determine overall model drift. By focusing on multiple parameters, an aggregate drift score is able to reflect real-world data drifts that affect multiple parameters in a model simultaneously, thus leading to a more comprehensive and accurate understanding of distribution changes that can assist in model development.
With respect to claims 11 and 18, the claims recite similar limitations corresponding to claim 3, therefore the same rationale of rejection is applicable.
With respect to claim 12, the rejection of claim 4 is incorporated. The difference in scope being:
wherein operation of the trained inferential model on the validating data set generates validated data results (Agarwal discloses predicted probabilities (‘validated data results’), “the same concept can be even replicated for the outcome – the predicted probabilities. The same techniques applied on the predicted probabilities is called prior probability shift (Fig. 7.3). This refers to the change in the distribution of the target variable in the training data and the live data. The target variable is binned to form 𝑖 bins from both the distribution. Then, the shift in the variable contribution to each bin is calculated” (P. 128, Last Paragraph).).
Claims 7-8, 15 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Agarwal in view of Yi (“Discriminative dimensionality reduction for sensor drift …”).
With respect to claim 7, Agarwal teaches the method of claim 1, however Agarwal does not teach a first parameter that represents a sensor data parameter, which Yi does:
wherein the first parameter represents a sensor data parameter (Yi discloses “the UCSD sensor drift dataset was collected by Vergara et al. (2012). There are 13,910 samples in total collected using an electronic nose with 16 gas sensors. The collection period lasted for 36 months starting from January 2018 … According to the sample acquisition time, the sample set was split into ten batches. … Eight Features were extracted from sensor signals for each sensor” (P. 5-6, Sec. 4.1, First Paragraph).),
the extracted first distribution including a distribution of values of the sensor data parameter from the unlabeled inference data set (Yi discloses “the sample set was split into ten batches. … Eight Features were extracted from sensor signals for each sensor. Thus, the feature vector was 128-dimensional for each sample … batch 1 is adopted as the samples in the source domain. The label information in the source domain is available. Other batches are adopted as the samples in the target domains whose labels need to be predicted. Fig. 1 shows the 2D projection of the samples in batch 1~10. It is easy to observe that the sensor signals are time-varying, i.e., the distribution difference between the source domain and target domain is time-dependent” (P. 5-6, Sec. 4.1, First Paragraph). See Figure 1 on P. 4 depicting samples (‘distribution of values’) of Batch 1 (‘first distribution’) projected into a 2D subspace.
See Figure 3 on P. 7 depicting classification performance obtained by tuning model parameters and using the UCSD dataset that makes up the source (‘inference data set’) and target data.),
and the extracted second distribution including a distribution of values of the sensor data parameter of validating model input data of the validating data set configured to be input into the trained inferential model (Yi discloses “the sample set was split into ten batches. … Eight Features were extracted from sensor signals for each sensor. Thus, the feature vector was 128-dimensional for each sample … batch 1 is adopted as the samples in the source domain. The label information in the source domain is available. Other batches are adopted as the samples in the target domains whose labels need to be predicted. Fig. 1 shows the 2D projection of the samples in batch 1~10. It is easy to observe that the sensor signals are time-varying, i.e., the distribution difference between the source domain and target domain is time-dependent” (P. 5-6, Sec. 4.1, First Paragraph). See Figure 1 on P. 4 depicting samples (‘distribution of values’) of a target domain (‘second distribution’) projected into a 2D subspace.
See Figure 3 on P. 7 depicting classification performance obtained by tuning model parameters and using the UCSD dataset that makes up the source and target data (‘validating data set’).).
Yi teaches obtaining (‘extracting’) samples (‘distributions’) from sensor data to detect distribution drifts is a known method in the art. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the method of Agarwal with the sensor data samples disclosed by Yi because sensor data is prone to drifts. By detecting for drifts in sensor data, machine learning engineers can determine if a sensor is degraded or if environmental changes have occurred, and can therefore, take steps to replace sensors or update a model to reflect real-world measurements.
With respect to claim 8, Agarwal teaches the method of claim 1, however, Agarwal does not teach a reduced representation parameter of raw sensor data, which is taught by Yi:
wherein the first parameter includes a reduced representation parameter of raw sensor data output by a sensor (Yi discloses “employ a robust, low-rank, and sparse representation of sensor signals to address the sensor drift problem. The low-rank property guarantees that each target sample can be approximately represented by its neighbors in the common subspace, while the sparse property ensures that each target sample can be represented by a few samples in the source domain. Specifically, both the source and target data are projected into a common subspace, where each target sample is assumed to be represented by a linear combination of all the source samples via a reconstruction coefficient matrix. The distribution discrepancy between source data and target data is alleviated” (P. 2, Sec. 1, Second Paragraph).
Yi further discloses “the UCSD sensor drift dataset was collected by Vergara et al. (2012). There are 13,910 samples in total collected using an electronic nose with 16 gas sensors. The collection period lasted for 36 months starting from January 2018 … According to the sample acquisition time, the sample set was split into ten batches. … Eight Features were extracted from sensor signals for each sensor” (P. 5-6, Sec. 4.1, First Paragraph).),
the extracted first distribution including a distribution of values of the reduced representation parameter of raw sensor data of the unlabeled inference data set (Yi discloses “the sample set was split into ten batches. … Eight Features were extracted from sensor signals for each sensor. Thus, the feature vector was 128-dimensional for each sample … batch 1 is adopted as the samples in the source domain. The label information in the source domain is available. Other batches are adopted as the samples in the target domains whose labels need to be predicted. Fig. 1 shows the 2D projection of the samples in batch 1~10. It is easy to observe that the sensor signals are time-varying, i.e., the distribution difference between the source domain and target domain is time-dependent” (P. 5-6, Sec. 4.1, First Paragraph). See Figure 1 on P. 4 depicting samples (‘distribution of values’) of Batch 1 (‘first distribution’) projected into a 2D subspace.
See Figure 3 on P. 7 depicting classification performance obtained by tuning model parameters and using the UCSD dataset that makes up the source (‘inference data set’) and target data.),
the extracted second distribution including a distribution of values of the reduced representation parameter of raw sensor data of validating model input data of the validating data set configured to be input into the trained inferential model (Yi discloses “the sample set was split into ten batches. … Eight Features were extracted from sensor signals for each sensor. Thus, the feature vector was 128-dimensional for each sample … batch 1 is adopted as the samples in the source domain. The label information in the source domain is available. Other batches are adopted as the samples in the target domains whose labels need to be predicted. Fig. 1 shows the 2D projection of the samples in batch 1~10. It is easy to observe that the sensor signals are time-varying, i.e., the distribution difference between the source domain and target domain is time-dependent” (P. 5-6, Sec. 4.1, First Paragraph). See Figure 1 on P. 4 depicting samples (‘distribution of values’) of a target domain (‘second distribution’) projected into a 2D subspace.
See Figure 3 on P. 7 depicting classification performance obtained by tuning model parameters and using the UCSD dataset that makes up the source and target data (‘validating data set’).).
Yi teaches projecting sensor data into a common subspace to create a low-rank and sparse representation (‘reduced representation parameter’) of sensor signals (‘raw sensor data’) is a known method in the art. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the method of Agarwal with the sensor data samples disclosed by Yi because high-dimensional sensor data is complex. By reducing sensor data dimensionality, machine learning engineers can make models less complex resulting in a more interpretable and accurate model.
With respect to claims 15 and 20, the claims recite similar limitations corresponding to claim 7, therefore the same rationale of rejection is applicable.
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Agarwal in view of Phillippe et al. (US 20230132501 A1), hereinafter Phillippe.
With respect to claim 21, Agarwal teaches:
The method of claim 1, wherein the validating operation further includes determining whether a validity condition is satisfied, and further comprising (Agarwal discloses “using a predefined threshold, alerts are generated by passing the distance calculated above for each batch of new or live data.” (P. 127, First Paragraph).):
providing, responsive to the second validation operation, a responsive action including (Agarwal discloses “Eventually a model that exhibits any kind of degradation will need to be examined further either for recalibration, retraining or, in the worst case, replacement” (P. 124, ¶ 1-3).
Agarwal discloses covariate drifts close to the High Drift level indicate a model investigation is needed, see (P. 127, Last Paragraph).).
However, Agarwal does not teach responsive to a validation condition, transmitting over a network a notification or an instruction, which is taught by Phillippe.
responsive to determining that the validation condition is satisfied, transmitting over a network a notification indicating that the trained inferential model is or remains validated for operation on the inference data set (Phillippe discloses “The validation report may, for example, provide information about the validation checks performed, which checks passed, …The validation report may then be provided to the subscriber submitting the model artifact validation request” [0044].
Phillippe discloses “Subscribers to this service may submit a request for a model artifact corresponding to a model to be validated to a system (model validation-as-a-service system (MVS)) providing the service” [0043].);
and responsive to determining that the validation condition is not satisfied, transmitting over the network an instruction that the trained inferential model should be retrained (Phillippe discloses “for a validation check that failed or got a low score, the validation system may also suggest one or more actions/operations to pass that validation check or to improve the score for that validation check. The suggested actions when performed may enable an architect of the model and its corresponding model artifact to improve or refine the model artifact and the model so as to reduce model artifact deployment problems, improve model efficiency and performance of the model upon deployment, ensure consistent model execution, and in general improve the stability and consistency of model execution” [0053].
Phillippe discloses “A model artifact may comprise multiple components related to the trained model including related code or instructions, parameters (e.g., value of trained parameters)” [0002].).
Phillippe teaches validating a model using a dataset and displaying pass and fail validation results to a user is a known method in the art. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the method of Agarwal with the technique disclosed by Phillippe to recommend actions to users based on model validation checks. By recommending actions to users based on model validation checks, users can either confidently deploy their models if validation checks pass, or update their models to pass future validation checks if validation checks fail.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PEDRO J MORALES whose telephone number is (571)272-6106. The examiner can normally be reached 8:30 AM - 6:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MIRANDA M HUANG can be reached at (571)270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PEDRO J MORALES/Examiner, Art Unit 2124
/VINCENT GONZALES/Primary Examiner, Art Unit 2124
Prosecution Insights