CTNF 18/490,999 CTNF 101806 Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Specification 07-29 AIA The disclosure is objected to because of the following informalities: in paragraph [0028], “ based on hard (exact) or soft ( withing a threshold deviation) standards ”, withing should be within . Appropriate correction is required. Claim Rejections - 35 USC § 101 07-04-01 AIA 07-04 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1-5, 7-15, and 17-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract ideas without significantly more. The analysis of the claims will follow the 2019 Revised Patent Subject Matter Eligibility Guidelines (“2019 PEG”). Step 1: Independent claims 1 ( A method, comprising: ), and 11 ( A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising: ) are directed towards a method and a manufacture respectively. Therefore, these claims, as well as their dependent claims, are directed towards one of the four statutory categories (process, machine, manufacture, or composition of matter). Claim 1 Step 2A, Prong 1: The claim states, inter alia: determining a data distribution difference between the unknown dataset and one of the plurality of shifted datasets that is closest to the unknown dataset; This limitation recites a mental process, using evaluation, judgement and opinion, with aid of pen and paper, and mathematical calculation of a difference of distributions of two datasets. determining if the data distribution difference is less than or equal to a first known threshold; This limitation recites a mental process, using evaluation, judgement and opinion, with aid of pen and paper, and mathematical calculation of comparing the data distribution difference and a threshold and determining if the difference is less than or equal to the threshold. in response to determining that the data distribution difference is less than or equal to the first known threshold, applying the data distribution difference to a correlation model to determine an estimated test metric difference; This limitation recites a mathematical calculation of inputting an input into a mathematical model and observing a predicted output. determining a test metric difference between the first test metric and a second test metric associated with the one of the plurality of shifted datasets that is closest to the unknown dataset; and This limitation recites a mental process, using evaluation, judgement and opinion, with aid of pen and paper, and mathematical calculation of a difference of two values determining if a difference between the test metric difference and the estimated test metric difference is less than or equal to a second known threshold. This limitation recites a mental process, using evaluation, judgement and opinion, with aid of pen and paper, and mathematical calculation of calculating another difference between the test metric difference and the estimated test metric difference and comparing said another difference with a threshold and determining if said another difference is less than or equal to the threshold. Step 2A, Prong 2: The additional elements recited in the claim do not integrate the judicial exception into a practical application. Additional elements: A method, comprising: This limitation is recited a high level of generality and recites use of generic computer equipment to apply abstract idea limitations. Mere recitation that a judicial exception is to be performed using generic computer equipment in their ordinary capacity cannot meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f). generating a first test metric from a machine learning model using an unknown dataset; This limitation is recited at a high level of generality and recites use of a generic machine learning model to apply abstract idea limitations. Mere recitation that a judicial exception is to be performed using a generic machine learning model in their ordinary capacity cannot meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f). generating a plurality of second test metrics from the machine learning model using a plurality of shifted datasets, the plurality of shifted datasets being shifted versions of a known dataset; This limitation is recited at a high level of generality and recites use of a generic machine learning model to apply abstract idea limitations. Mere recitation that a judicial exception is to be performed using a generic machine learning model in their ordinary capacity cannot meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f). Step 2B: The additional elements recited in the claim do not integrate the judicial exception into a practical application. Additional elements: A method, comprising: This limitation is recited at a high level of generality and recites use of generic computer equipment to apply abstract idea limitations. Mere recitation that a judicial exception is to be performed using generic computer equipment in their ordinary capacity cannot meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f). generating a first test metric from a machine learning model using an unknown dataset; This limitation is recited at a high level of generality and recites use of a generic machine learning model to apply abstract idea limitations. Mere recitation that a judicial exception is to be performed using a generic machine learning model in their ordinary capacity cannot meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f). Claim 2 Step 2A, Prong 1: This claim states, inter alia: wherein determining that the data distribution difference is greater than the first known threshold is indicative of a false positive or false negative and that retraining, or revalidation of the machine learning model is to be performed. This limitation recites a mental process, using evaluation, judgement and opinion, with aid of pen and paper, to judge based on a prior determination, that there is a false positive, or a false negative and to judge, based on that prior determination, that retraining or revalidation should be performed. Step 2A, Prong 2: There are no additional elements in this claim. Step 2B: There are no additional elements in this claim. Claim 3 Step 2A, Prong 1: This claim states, inter alia: wherein determining that the difference between the test metric difference and the estimated test metric difference is greater than the second known threshold is indicative of a false positive or false negative and that retraining, or revalidation of the machine learning model is to be performed. This limitation recites a mental process, using evaluation, judgement and opinion, with aid of pen and paper, to judge, based on a prior determination, that there is a false positive or a false negative and to judge, based on that prior determination, that retraining or revalidation should be performed. Step 2A, Prong 2: There are no additional elements in this claim. Step 2B: There are no additional elements in this claim. Claim 4 Step 2A, Prong 1: The claim states, inter alia: wherein determining that the difference between the test metric difference and the estimated test metric difference is less than or equal to a second known threshold is indicative that an underlying data pipeline of the machine learning model is operating in an expected manner. This limitation recites a mental process, using evaluation, judgement and opinion, with aid of pen and paper, to judge, based on a prior determination, that a machine learning model is working correctly. Step 2A, Prong 2: There are no additional elements in this claim Step 2B: There are no additional elements in this claim Claim 5 Step 2A, Prong 1: The claim states, inter alia: determining a data distribution difference between the unknown dataset and each of the plurality of shifted datasets; and This limitation recites a mental process, using evaluation, judgement and opinion, with aid of pen and paper, and mathematical calculation of a difference of distributions of two datasets, for each of a plurality of shifted datasets. selecting the one of the plurality of shifted datasets that is closest to the unknown dataset based on the one of the plurality of shifted datasets having a smallest data distribution difference with the unknown dataset. This limitation recites a mental process, using evaluation, judgement and opinion, with aid of pen and paper, to judge which difference is the smallest, judge the shifted dataset that resulted in that difference to be the closest dataset to the unknown dataset, and to use that dataset as the closest. Step 2A, Prong 2: The additional elements recited in the claim do not integrate the judicial exception into a practical application. Additional elements: wherein determining a data distribution difference between the unknown dataset and one of the plurality of shifted datasets that is closest to the unknown dataset comprises: This limitation represents an insignificant extra-solution activity of selecting a particular type of data to be manipulated performed by generic computer equipment. See MPEP 2106.05(g); Step 2B: The additional elements recited in the claim do not integrate the judicial exception into a practical application. Additional elements: wherein determining a data distribution difference between the unknown dataset and one of the plurality of shifted datasets that is closest to the unknown dataset comprises: MPEP 2106.05(d)(II)(ii) indicates that performing repetitive calculations is a well-understood, routine, and conventional function when it is claimed in a merely generic manner (as it is in the present claim); Claim 7 Step 2A, Prong 1: There are no additional abstract ideas recited in this claim Step 2A, Prong 2: The additional elements recited in the claim do not integrate the judicial exception into a practical application. Additional elements: wherein the machine learning model is a compressed model that acts as a proxy for another machine learning model. This limitation is recited a high level of generality and recites use of a generic compressed model to apply abstract idea limitations. Mere recitation that a judicial exception is to be performed using a generic compressed model in their ordinary capacity cannot meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f). Step 2B: The additional elements recited in the claim do not integrate the judicial exception into a practical application. Additional elements: wherein the machine learning model is a compressed model that acts as a proxy for another machine learning model. This limitation is recited a high level of generality and recites use of a generic compressed model to apply abstract idea limitations. Mere recitation that a judicial exception is to be performed using a generic compressed model in their ordinary capacity cannot meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f). Claim 8 Step 2A, Prong 1: The claim states, inter alia: applying a plurality of perturbation functions to the known dataset to generate the plurality of shifted datasets. This limitation recites a mathematical concept of using mathematical functions to generate shifted datasets. Step 2A, Prong 2: There are no additional elements in this claim Step 2B: There are no additional elements in this claim Claim 9 Step 2A, Prong 1: The claim states, inter alia: wherein the first known threshold is based on an average of a data distribution difference between the unknown dataset and each of the plurality of shifted datasets. This limitation recites a mathematical concept of using averages to define a threshold. Step 2A, Prong 2: There are no additional elements in this claim Step 2B: There are no additional elements in this claim Claim 10 Step 2A, Prong 1: The claim states, inter alia: wherein the second known threshold is based on an average of second test metric differences between each of the second test metrics. This limitation recites a mathematical concept of using averages to define a threshold. Step 2A, Prong 2: There are no additional elements in this claim Step 2B: There are no additional elements in this claim Claim 11 Step 2A, Prong 1: The claim states, inter alia: determining a data distribution difference between the unknown dataset and one of the plurality of shifted datasets that is closest to the unknown dataset; This limitation recites a mental process, using evaluation, judgement and opinion, with aid of pen and paper, and mathematical calculation of a difference of distributions of two datasets. determining if the data distribution difference is less than or equal to a first known threshold; This limitation recites a mental process, using evaluation, judgement and opinion, with aid of pen and paper, and mathematical calculation of comparing the data distribution difference and a threshold and determining if the difference is less than or equal to the threshold. in response to determining that the data distribution difference is less than or equal to the first known threshold, applying the data distribution difference to a correlation model to determine an estimated test metric difference; This limitation recites a mathematical calculation of inputting an input into a mathematical model and observing a predicted output. determining a test metric difference between the first test metric and a second test metric associated with the one of the plurality of shifted datasets that is closest to the unknown dataset; and This limitation recites a mental process, using evaluation, judgement and opinion, with aid of pen and paper, and mathematical calculation of a difference of two values determining if a difference between the test metric difference and the estimated test metric difference is less than or equal to a second known threshold. This limitation recites a mental process, using evaluation, judgement and opinion, with aid of pen and paper, and mathematical calculation of calculating another difference between the test metric difference and the estimated test metric difference and comparing said another difference with a threshold and determining if said another difference is less than or equal to the threshold. Step 2A, Prong 2: The additional elements recited in the claim do not integrate the judicial exception into a practical application. Additional elements: A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising: This limitation is recited a high level of generality and recites use of generic computer equipment to apply abstract idea limitations. Mere recitation that a judicial exception is to be performed using generic computer equipment in their ordinary capacity cannot meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f). generating a first test metric from a machine learning model using an unknown dataset; This limitation is recited at a high level of generality and recites use of a generic machine learning model to apply abstract idea limitations. Mere recitation that a judicial exception is to be performed using a generic machine learning model in their ordinary capacity cannot meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f). generating a plurality of second test metrics from the machine learning model using a plurality of shifted datasets, the plurality of shifted datasets being shifted versions of a known dataset; This limitation is recited at a high level of generality and recites use of a generic machine learning model to apply abstract idea limitations. Mere recitation that a judicial exception is to be performed using a generic machine learning model in their ordinary capacity cannot meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f). Step 2B: The additional elements recited in the claim do not integrate the judicial exception into a practical application. Additional elements: A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising: This limitation is recited a high level of generality and recites use of generic computer equipment to apply abstract idea limitations. Mere recitation that a judicial exception is to be performed using generic computer equipment in their ordinary capacity cannot meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f). generating a first test metric from a machine learning model using an unknown dataset; This limitation is recited at a high level of generality and recites use of a generic machine learning model to apply abstract idea limitations. Mere recitation that a judicial exception is to be performed using a generic machine learning model in their ordinary capacity cannot meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f). Claim 12 Step 2A, Prong 1: This claim states, inter alia: wherein determining that the data distribution difference is greater than the first known threshold is indicative of a false positive or false negative and that retraining, or revalidation of the machine learning model is to be performed. This limitation recites a mental process, using evaluation, judgement and opinion, with aid of pen and paper, to judge based on a prior determination, that there is a false positive, or a false negative and to judge, based on that prior determination, that retraining or revalidation should be performed. Step 2A, Prong 2: There are no additional elements in this claim. Step 2B: There are no additional elements in this claim. Claim 13 Step 2A, Prong 1: This claim states, inter alia: wherein determining that the difference between the test metric difference and the estimated test metric difference is greater than the second known threshold is indicative of a false positive or false negative and that retraining, or revalidation of the machine learning model is to be performed. This limitation recites a mental process, using evaluation, judgement and opinion, with aid of pen and paper, to judge, based on a prior determination, that there is a false positive or a false negative and to judge, based on that prior determination, that retraining or revalidation should be performed. Step 2A, Prong 2: There are no additional elements in this claim. Step 2B: There are no additional elements in this claim. Claim 14 Step 2A, Prong 1: The claim states, inter alia: wherein determining that the difference between the test metric difference and the estimated test metric difference is less than or equal to a second known threshold is indicative that an underlying data pipeline of the machine learning model is operating in an expected manner. This limitation recites a mental process, using evaluation, judgement and opinion, with aid of pen and paper, to judge, based on a prior determination, that a machine learning model is working correctly. Step 2A, Prong 2: There are no additional elements in this claim Step 2B: There are no additional elements in this claim Claim 15 Step 2A, Prong 1: The claim states, inter alia: determining a data distribution difference between the unknown dataset and each of the plurality of shifted datasets; and This limitation recites a mental process, using evaluation, judgement and opinion, with aid of pen and paper, and mathematical calculation of a difference of distributions of two datasets, for each of a plurality of shifted datasets. selecting the one of the plurality of shifted datasets that is closest to the unknown dataset based on the one of the plurality of shifted datasets having a smallest data distribution difference with the unknown dataset. This limitation recites a mental process, using evaluation, judgement and opinion, with aid of pen and paper, to judge which difference is the smallest, judge the shifted dataset that resulted in that difference to be the closest dataset to the unknown dataset, and to use that dataset as the closest. Step 2A, Prong 2: The additional elements recited in the claim do not integrate the judicial exception into a practical application. Additional elements: wherein determining a data distribution difference between the unknown dataset and one of the plurality of shifted datasets that is closest to the unknown dataset comprises: This limitation represents an insignificant extra-solution activity of selecting a particular type of data to be manipulated performed by generic computer equipment. See MPEP 2106.05(g); Step 2B: The additional elements recited in the claim do not integrate the judicial exception into a practical application. Additional elements: wherein determining a data distribution difference between the unknown dataset and one of the plurality of shifted datasets that is closest to the unknown dataset comprises: MPEP 2106.05(d)(II)(ii) indicates that performing repetitive calculations is a well-understood, routine, and conventional function when it is claimed in a merely generic manner (as it is in the present claim); Claim 17 Step 2A, Prong 1: There are no additional abstract ideas recited in this claim Step 2A, Prong 2: The additional elements recited in the claim do not integrate the judicial exception into a practical application. Additional elements: wherein the machine learning model is a compressed model that acts as a proxy for another machine learning model. This limitation is recited a high level of generality and recites use of a generic compressed model to apply abstract idea limitations. Mere recitation that a judicial exception is to be performed using a generic compressed model in their ordinary capacity cannot meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f). Step 2B: The additional elements recited in the claim do not integrate the judicial exception into a practical application. Additional elements: wherein the machine learning model is a compressed model that acts as a proxy for another machine learning model. This limitation is recited a high level of generality and recites use of a generic compressed model to apply abstract idea limitations. Mere recitation that a judicial exception is to be performed using a generic compressed model in their ordinary capacity cannot meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f). Claim 18 Step 2A, Prong 1: The claim states, inter alia: applying a plurality of perturbation functions to the known dataset to generate the plurality of shifted datasets. This limitation recites a mathematical concept of using mathematical functions to generate shifted datasets. Step 2A, Prong 2: There are no additional elements in this claim Step 2B: There are no additional elements in this claim Claim 19 Step 2A, Prong 1: The claim states, inter alia: wherein the first known threshold is based on an average of a data distribution difference between the unknown dataset and each of the plurality of shifted datasets. This limitation recites a mathematical concept of using averages to define a threshold. Step 2A, Prong 2: There are no additional elements in this claim Step 2B: There are no additional elements in this claim Claim 20 Step 2A, Prong 1: The claim states, inter alia: wherein the second known threshold is based on an average of second test metric differences between each of the second test metrics. This limitation recites a mathematical concept of using averages to define a threshold. Step 2A, Prong 2: There are no additional elements in this claim Step 2B: There are no additional elements in this claim Claims 1 and 11 are rejected under 35 U.S.C. 101 because the claimed invention lacks patentable utility. Claims 1 and 11 by themselves do not disclose any practical purposes for the claimed material. Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-23-aia AIA The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. 07-20-02-aia AIA This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention. 07-21-aia AIA Claim (s) 1, 3-6, 8, 11, 13-16, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Using Generative Adversarial Networks to Validate Discrete Event Simulation Models by Montevechi et al., hereafter Montevechi, in view of US 12585995 B2 by Sodhi et al., hereafter Sodhi . Regarding claim 1, Montevechi teaches: A method, comprising: generating a first test metric from a machine learning model using an unknown dataset; ((Montevechi) page 5 paragraph 4-5, “…the Compared Data (CD) are inserted into the program to be prepared …An Equivalence Test for the Difference of Two Proportions (ETDTP) is carried out in the final step, where the proportion of two populations is given by p1 and p2.” Proportion p1 is a test metric for the Compared Data which is an unknown dataset. ) generating a plurality of second test metrics from the machine learning model using a plurality of shifted datasets, the plurality of shifted datasets being shifted versions of a known dataset; ((Montevechi) page 5 paragraph 1, “Then, the Generator is also trained to generate batches of synthetic data that confuse the Discriminator. This interaction of the adversarial learning process is repeated until there are no remaining batches of real data, completing a learning epoch. The maximum number of the epoch is 10000 .”; (Montevechi) page 5 paragraph 2, “If the synthetic data are perfectly realistic, the k-NN algorithm classifies each observation at random, and the accuracy (AC) is 50%.” Each epoch is a shifted dataset. The accuracy for each epoch is a test metric. ) […] using a correlation model to determine an estimated test metric difference; ((Montevechi) page 5 paragraph 4, “As soon as the data (Compared Data) are inserted, they are rescaled based on the trained data, and then the Discriminator judges them .” The Discriminator is a correlation model and judging the compared data is equivalent to determining an estimated test metric difference. ) determining a test metric difference between the first test metric and a second test metric associated with the one of the plurality of shifted datasets that is closest to the unknown dataset; and ((Montevechi) page 5 paragraph 5, “An Equivalence Test for the Difference of Two Proportions (ETDTP) is carried out in the final step, where the proportion of two populations is given by p1 and p2 .” Proportion p2 is a test metric for a data from a known dataset ) determining if a difference between the test metric difference and the estimated test metric difference is less than or equal to a second known threshold. ((Montevechi) page 5 paragraph 3, “The condition happens if the Discriminator discriminates the data by around 50.0% . However, we are aware that the situation sometimes is not possible. In this sense, the GANs are trained until the Discriminator discriminates data between 45.0% and 55.0% , because the judgment can assume a tolerance.” An estimated test metric difference could be 50%. A Discriminator output could be a test metric difference. The difference between the Discriminator output and 50% is checked to be less than or equal to 5%, which is a threshold. ) Montevechi does not explicitly disclose: determining a data distribution difference between the unknown dataset and one of the plurality of shifted datasets that is closest to the unknown dataset; determining if the data distribution difference is less than or equal to a first known threshold; in response to determining that the data distribution difference is less than or equal to the first known threshold, applying the data distribution difference […] Sodhi does teach: determining a data distribution difference between the unknown dataset and one of the plurality of shifted datasets that is closest to the unknown dataset; ((Sodhi) Column 10 lines 52-54, “The similarity scores may be calculated with at least one of Jaccard similarity, cosine similarity, Fast Fourier Transform, or Euclidean similarity” Similarity scores are reverse data distributions difference so is equivalent to a data distribution difference. ) determining if the data distribution difference is less than or equal to a first known threshold; in response to determining that the data distribution difference is less than or equal to the first known threshold, applying the data distribution difference […] ((Sodhi) Column 10 lines 58-61, “ Responsive to a determination that the closest match machine learning model exceeds a similarity threshold , process 400 generates predictions for the new dataset with the closest match machine learning model (step 414)” Exceeding a similarity threshold is being less than a difference threshold. Process 400 generating predictions with the closest match is equivalent to applying the data distribution difference to a correlation model to determine an estimated test metric difference. ) Sodhi and Montevechi are in the same area of invention, that being generation of shifted data and model validation. Thus, it would have been obvious for a person having ordinary skill in the art before the effective filing date to have combined determining of data distribution difference, determining if that difference is less than or equal to a threshold, and, if it is less than a threshold, applying that difference, as taught by Sodhi, to a correlation model to determine an estimate as taught by Montevechi, in order to make sure the data distributions are similar enough to accurately be compared. This application of a known technique to the applicable method disclosed by Montevechi would produce the predictable result of the method claimed in claim 1. Regarding claim 3, Montevechi, in view of Sodhi, teaches the material disclosed in claim 1, and Montevechi additionally teaches: wherein determining that the difference between the test metric difference and the estimated test metric difference is greater than the second known threshold is indicative of a false positive or false negative and that retraining, or revalidation of the machine learning model is to be performed. ((Montevechi) page 5 paragraphs 2-3, “If the synthetic data are perfectly realistic, the k-NN algorithm classifies each observation at random, and the accuracy (AC) is 50%. On the other hand, if synthetic data are not realistic at all, the classifier can easily separate the observations, and the expected AC reaches 100% (David and Oquab 2017). … Moreover, the GANs only stop training if another condition is reached. … The condition happens if the Discriminator discriminates the data by around 50.0% . However, we are aware that the situation sometimes is not possible. In this sense, the GANs are trained until the Discriminator discriminates data between 45.0% and 55.0% ,” Being not realistic is indicative of a false positive or a false negative. Continuing to train until an accuracy difference within 5% is reached is equivalent to indicating that re-validation is to be performed if the difference is greater than that threshold of 5%. ) Regarding claim 4, Montevechi, in view of Sodhi, teaches the material disclosed in claim 1, and Montevechi additionally teaches: wherein determining that the difference between the test metric difference and the estimated test metric difference is less than or equal to a second known threshold is indicative that an underlying data pipeline of the machine learning model is operating in an expected manner. ((Montevechi) page 5 paragraph 3, “The Discriminator evaluates the TD randomly if the Generator can trick the Discriminator (Brownlee 2020). The condition happens if the Discriminator discriminates the data by around 50.0% . However, we are aware that the situation sometimes is not possible. In this sense, the GANs are trained until the Discriminator discriminates data between 45.0% and 55.0% , because the judgment can assume a tolerance.” The difference between the test metric difference and estimated test metric difference being less than or equal to 5% is indicative that the Generator can trick the Discriminator which is indicative that it is operating in an expected manner. ) Regarding claim 5, Montevechi, in view of Sodhi, teaches the material disclosed in claim 1, and Sodhi additionally teaches: wherein determining a data distribution difference between the unknown dataset and one of the plurality of shifted datasets that is closest to the unknown dataset comprises: determining a data distribution difference between the unknown dataset and each of the plurality of shifted datasets; and ((Sodhi) column 10 lines 48-52, “Upon receiving a new dataset (step 408), process 400 calculates similarity scores between the new dataset and the machine learning models in the model catalog according to the properties of the datasets in the metadata of the machine learning models (step 410).” Similarity scores are inverse data distribution differences so are equivalent to data distribution differences. The new dataset is the unknown dataset. The machine learning models in the model catalog are the plurality of shifted datasets. ) selecting the one of the plurality of shifted datasets that is closest to the unknown dataset based on the one of the plurality of shifted datasets having a smallest data distribution difference with the unknown dataset. ((Sodhi) column 10 lines 55-57, “Process 400 identifies a closest match machine learning model from the model catalog for the new dataset according to similarity score (step 412).” Identifying a closest match according to similarity score is selecting one of the shifted datasets that is closest based on having a smallest data distribution difference. ) Sodhi and Montevechi are in the same area of invention, that being generation of shifted data and model validation. Thus, it would have been obvious for a person having ordinary skill in the art before the effective filing date to have combined the selection of one of the shifted datasets having a greatest similarity and thus smallest distribution difference, as taught by Sodhi, with the determining a data distribution difference between an unknown dataset and a shifted data set as previously taught by Montevechi, in view of Sodhi, in order to reduce ambiguity of which shifted dataset is the closest dataset. This improvement using a known technique, disclosed within a method being improved in the same way as the claimed invention, to the applicable method disclosed by Montevechi, in view of Sodhi, would produce the predictable result of the method claimed in claim 5. Regarding claim 6, Montevechi, in view of Sodhi, teaches the material disclosed in claim 1, and Montevechi additionally teaches: generating a plurality of second data distributions between each of the plurality of shifted datasets; ((Montevechi) page 5 paragraph 1, “Then, the Generator is also trained to generate batches of synthetic data that confuse the Discriminator .” The batches of synthetic data are a plurality of second data distributions. ) generating a plurality of second test metric differences between each of the second test metrics; and ((Montevechi) page 5 equation 1, “H1: -δ ≤ p1 - p2 ≤ δ” p1 and p2 are proportions that can be second test metrics between each of the second test metrics, and the difference between them would be second test metric differences. ) generating the correlation model based on the plurality of second data distributions and the plurality of second test metric differences. ((Montevechi) page 4 paragraph 1, “The GANs are based on two adversarial Artificial Neural Networks (ANN) that are trained iteratively and compete against each other . The two networks are the data Generator (G) and the data Discriminato r (D) (Pan et al. 2019).” A Discriminator is a correlation model and is generated based on the Generator which includes the second data distributions and second test metric differences. ) Regarding claim 8, Montevechi, in view of Sodhi teaches the material disclosed in claim 1, and Montevechi additionally teaches: applying a plurality of perturbation functions to the known dataset to generate the plurality of shifted datasets. ((Montevechi) page 4 paragraph 3, “On the other hand, G is trained in order to minimize the probability of D identifying the synthetic data, minimizing log(1–D(G(z, θg),θd) .” Generator G generates synthetic data by applying the perturbation function of minimizing a log function to a known real data. ) Regarding claim 11, Montevechi teaches a non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising: generating a first test metric from a machine learning model using an unknown dataset; ((Montevechi) page 5 paragraph 4-5, “…the Compared Data (CD) are inserted into the program to be prepared …An Equivalence Test for the Difference of Two Proportions (ETDTP) is carried out in the final step, where the proportion of two populations is given by p1 and p2.” Proportion p1 is a test metric for the Compared Data which is an unknown dataset. ) generating a plurality of second test metrics from the machine learning model using a plurality of shifted datasets, the plurality of shifted datasets being shifted versions of a known dataset; ((Montevechi) page 5 paragraph 1, “Then, the Generator is also trained to generate batches of synthetic data that confuse the Discriminator. This interaction of the adversarial learning process is repeated until there are no remaining batches of real data, completing a learning epoch. The maximum number of the epoch is 10000 .”; (Montevechi) page 5 paragraph 2, “If the synthetic data are perfectly realistic, the k-NN algorithm classifies each observation at random, and the accuracy (AC) is 50%.” Each epoch is a shifted dataset. The accuracy for each epoch is a test metric. ) […] using a correlation model to determine an estimated test metric difference; ((Montevechi) page 5 paragraph 4, “As soon as the data (Compared Data) are inserted, they are rescaled based on the trained data, and then the Discriminator judges them .” The Discriminator is a correlation model and judging the compared data is equivalent to determining an estimated test metric difference. ). determining a test metric difference between the first test metric and a second test metric associated with the one of the plurality of shifted datasets that is closest to the unknown dataset; and ((Montevechi) page 5 paragraph 5, “An Equivalence Test for the Difference of Two Proportions (ETDTP) is carried out in the final step, where the proportion of two populations is given by p1 and p2 .” Proportion p2 is a test metric for a data from a known dataset ) determining if a difference between the test metric difference and the estimated test metric difference is less than or equal to a second known threshold. ((Montevechi) page 5 paragraph 3, “The condition happens if the Discriminator discriminates the data by around 50.0% . However, we are aware that the situation sometimes is not possible. In this sense, the GANs are trained until the Discriminator discriminates data between 45.0% and 55.0% , because the judgment can assume a tolerance.” An estimated test metric difference could be 50%. A Discriminator output could be a test metric difference. The difference between the Discriminator output and 50% is checked to be less than or equal to 5%, which is a threshold. ) Montevechi does not explicitly disclose: A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising: ((Sodhi) column 4 lines 11-17, “These computer readable program instructions are stored in various types of computer readable storage media , such as cache 121 and the other storage media discussed below. The program instructions, and associated data, are accessed by processor set 110 to control and direct performance of the inventive methods.”) determining a data distribution difference between the unknown dataset and one of the plurality of shifted datasets that is closest to the unknown dataset; determining if the data distribution difference is less than or equal to a first known threshold; in response to determining that the data distribution difference is less than or equal to the first known threshold, applying the data distribution difference […]; Sodhi does teach: A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising: ((Sodhi) column 4 lines 11-17, “These computer readable program instructions are stored in various types of computer readable storage media , such as cache 121 and the other storage media discussed below. The program instructions, and associated data, are accessed by processor set 110 to control and direct performance of the inventive methods.”) determining a data distribution difference between the unknown dataset and one of the plurality of shifted datasets that is closest to the unknown dataset; ((Sodhi) Column 10 lines 52-54, “The similarity scores may be calculated with at least one of Jaccard similarity, cosine similarity, Fast Fourier Transform, or Euclidean similarity” Similarity scores are reverse data distributions difference so is equivalent to a data distribution difference. ) determining if the data distribution difference is less than or equal to a first known threshold; in response to determining that the data distribution difference is less than or equal to the first known threshold, applying the data distribution difference […]; ((Sodhi) Column 10 lines 58-61, “ Responsive to a determination that the closest match machine learning model exceeds a similarity threshold , process 400 generates predictions for the new dataset with the closest match machine learning model (step 414)” Exceeding a similarity threshold is being less than a difference threshold. Process 400 generating predictions with the closest match is equivalent to applying the data distribution difference to a correlation model to determine an estimated test metric difference. ) Sodhi and Montevechi are in the same area of invention, that being generation of shifted data and model validation. Thus, it would have been obvious for a person having ordinary skill in the art before the effective filing date to have combined using a storage medium to hold the instructions to be executed by a processor, determining of data distribution difference, determining if that difference is less than or equal to a threshold, and, if it is less than a threshold, applying that difference to a correlation model to determine an estimate, as taught by Sodhi, into the teachings and suggestions of Montevechi, in order to provide a structure necessary to run the method on, and to make sure the data distributions are similar enough to accurately be compared. This application of a known technique to the applicable storage medium instructions disclosed by Montevechi would produce the predictable result claimed in claim 11. Regarding claim 13, Montevechi, in view of Sodhi, teaches the material disclosed in claim 11, and Montevechi additionally teaches: wherein determining that the difference between the test metric difference and the estimated test metric difference is greater than the second known threshold is indicative of a false positive or false negative and that retraining, or revalidation of the machine learning model is to be performed. ((Montevechi) page 5 paragraphs 2-3, “If the synthetic data are perfectly realistic, the k-NN algorithm classifies each observation at random, and the accuracy (AC) is 50%. On the other hand, if synthetic data are not realistic at all, the classifier can easily separate the observations, and the expected AC reaches 100% (David and Oquab 2017). … Moreover, the GANs only stop training if another condition is reached. … The condition happens if the Discriminator discriminates the data by around 50.0% . However, we are aware that the situation sometimes is not possible. In this sense, the GANs are trained until the Discriminator discriminates data between 45.0% and 55.0% ,” Being not realistic is indicative of a false positive or a false negative. Continuing to train until an accuracy difference within 5% is reached is equivalent to indicating that re-validation is to be performed if the difference is greater than that threshold of 5%. ) Regarding claim 14, Montevechi, in view of Sodhi, teaches the material disclosed in claim 11, and Montevechi additionally teaches: wherein determining that the difference between the test metric difference and the estimated test metric difference is less than or equal to a second known threshold is indicative that an underlying data pipeline of the machine learning model is operating in an expected manner. ((Montevechi) page 5 paragraph 3, “The Discriminator evaluates the TD randomly if the Generator can trick the Discriminator (Brownlee 2020). The condition happens if the Discriminator discriminates the data by around 50.0% . However, we are aware that the situation sometimes is not possible. In this sense, the GANs are trained until the Discriminator discriminates data between 45.0% and 55.0% , because the judgment can assume a tolerance.” The difference between the test metric difference and estimated test metric difference being less than or equal to 5% is indicative that the Generator can trick the Discriminator which is indicative that it is operating in an expected manner. ) Regarding claim 15, Montevechi, in view of Sodhi, teaches the material disclosed in claim 11, and Sodhi additionally teaches: wherein determining a data distribution difference between the unknown dataset and one of the plurality of shifted datasets that is closest to the unknown dataset comprises: determining a data distribution difference between the unknown dataset and each of the plurality of shifted datasets; and ((Sodhi) column 10 lines 48-52, “Upon receiving a new dataset (step 408), process 400 calculates similarity scores between the new dataset and the machine learning models in the model catalog according to the properties of the datasets in the metadata of the machine learning models (step 410).” Similarity scores are inverse data distribution differences so are equivalent to data distribution differences. The new dataset is the unknown dataset. The machine learning models in the model catalog are the plurality of shifted datasets. ) selecting the one of the plurality of shifted datasets that is closest to the unknown dataset based on the one of the plurality of shifted datasets having a smallest data distribution difference with the unknown dataset. ((Sodhi) column 10 lines 55-57, “Process 400 identifies a closest match machine learning model from the model catalog for the new dataset according to similarity score (step 412).” Identifying a closest match according to similarity score is selecting one of the shifted datasets that is closest based on having a smallest data distribution difference. ) Sodhi and Montevechi are in the same area of invention, that being generation of shifted data and model validation. Thus, it would have been obvious for a person having ordinary skill in the art before the effective filing date to have combined the selection of one of the shifted datasets having a greatest similarity and thus smallest distribution difference, as taught by Sodhi, with the determining a data distribution difference between an unknown dataset and a shifted data set as previously taught by Montevechi, in view of Sodhi, in order to reduce ambiguity of which shifted dataset is the closest dataset. This improvement using a known technique, disclosed within a method being improved in the same way as the claimed invention, to the applicable storage medium instructions disclosed by Montevechi, in view of Sodhi, would produce the predictable result claimed in claim 15. Regarding claim 16, Montevechi, in view of Sodhi, teaches the material disclosed in claim 11, and Montevechi additionally teaches: generating a plurality of second data distributions between each of the plurality of shifted datasets; ((Montevechi) page 5 paragraph 1, “Then, the Generator is also trained to generate batches of synthetic data that confuse the Discriminator .” The batches of synthetic data are a plurality of second data distributions. ) generating a plurality of second test metric differences between each of the second test metrics; and ((Montevechi) page 5 equation 1, “H1: -δ ≤ p1 - p2 ≤ δ” p1 and p2 are proportions that can be second test metrics between each of the second test metrics, and the difference between them would be second test metric differences. ) generating the correlation model based on the plurality of second data distributions and the plurality of second test metric differences. ((Montevechi) page 4 paragraph 1, “The GANs are based on two adversarial Artificial Neural Networks (ANN) that are trained iteratively and compete against each other . The two networks are the data Generator (G) and the data Discriminato r (D) (Pan et al. 2019).” A Discriminator is a correlation model and is generated based on the Generator which includes the second data distributions and second test metric differences. ) Regarding claim 18, Montevechi, in view of Sodhi teaches the material disclosed in claim 11, and Montevechi additionally teaches: applying a plurality of perturbation functions to the known dataset to generate the plurality of shifted datasets. ((Montevechi) page 4 paragraph 3, “On the other hand, G is trained in order to minimize the probability of D identifying the synthetic data, minimizing log(1–D(G(z, θg),θd) .” Generator G generates synthetic data by applying the perturbation function of minimizing a log function to a known real data. ) 07-21-aia AIA Claim (s) 2 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Montevechi, in view of Sodhi, in further view of US 20160283861 A1 by Gerard, hereafter Gerard . Regarding claim 2, Montevechi, in view of Sodhi, teaches the material disclosed in claim 1, and Montevechi additionally teaches: [...] A difference not being within a threshold indicates a false positive or a false negative. […] ((Montevechi) page 5 paragraph 2, “If the synthetic data are perfectly realistic, the k-NN algorithm classifies each observation at random, and the accuracy (AC) is 50%. On the other hand, if synthetic data are not realistic at all, the classifier can easily separate the observations, and the expected AC reaches 100% (David and Oquab 2017” Being not realistic is indicative of a false positive or a false negative. ) Montevechi, in view of Sodhi, does not explicitly disclose: wherein determining that the data distribution difference is greater than the first known threshold is indicative […] that retraining, or revalidation of the machine learning model is to be performed. Gerard does teach: wherein determining that the data distribution difference is greater than the first known threshold is indicative […] that retraining, or revalidation of the machine learning model is to be performed. ((Gerard) paragraph [0045], “When machine-learning model subsystem 320 determines that the distribution difference between the baseline distribution and the updated distribution reaches a distribution difference threshold , machine-learning model subsystem 320 generates an indicator to retrain the machine-learning model due to the shift in the feature vector distribution.” Reaching a threshold implies being equal to or greater than the threshold. ) Gerard and Montevechi are in the same area of invention, that being machine learning model training including updating when a difference threshold is reached. Thus, it would have been obvious for a person having ordinary skill in the art before the effective filing date to have combined retraining the machine learning model when a threshold is reached, as taught by Gerard, into the indication that reaching a difference threshold is indicative of a false positive or false negative, as taught by Montevechi, and the rest of the teachings of Montevechi, in order make sure data includes time-dated information due to changing world conditions. This application of a known technique to the applicable method disclosed by Montevechi, in view of Sodhi, would produce the predictable result of the method claimed in claim 2. Regarding claim 12, Montevechi, in view of Sodhi, teaches the material disclosed in claim 11, and Montevechi additionally teaches: […] A difference not being within a threshold indicates a false positive or a false negative [...] ((Montevechi) page 5 paragraph 2, “If the synthetic data are perfectly realistic, the k-NN algorithm classifies each observation at random, and the accuracy (AC) is 50%. On the other hand, if synthetic data are not realistic at all, the classifier can easily separate the observations, and the expected AC reaches 100% (David and Oquab 2017” Being not realistic is indicative of a false positive or a false negative. ) Montevechi, in view of Sodhi, does not explicitly disclose: wherein determining that the data distribution difference is greater than the first known threshold is indicative […] that retraining, or revalidation of the machine learning model is to be performed. ((Gerard) paragraph [0045], “When machine-learning model subsystem 320 determines that the distribution difference between the baseline distribution and the updated distribution reaches a distribution difference threshold , machine-learning model subsystem 320 generates an indicator to retrain the machine-learning model due to the shift in the feature vector distribution.” Reaching a threshold implies being equal to or greater than the threshold .) Gerard does teach: wherein determining that the data distribution difference is greater than the first known threshold is indicative […] that retraining, or revalidation of the machine learning model is to be performed. ((Gerard) paragraph [0045], “When machine-learning model subsystem 320 determines that the distribution difference between the baseline distribution and the updated distribution reaches a distribution difference threshold , machine-learning model subsystem 320 generates an indicator to retrain the machine-learning model due to the shift in the feature vector distribution.” Reaching a threshold implies being equal to or greater than the threshold .) Gerard and Montevechi are in the same area of invention, that being machine learning model training including updating when a difference threshold is reached. Thus, it would have been obvious for a person having ordinary skill in the art before the effective filing date to have combined retraining the machine learning model when a threshold is reached, as taught by Gerard, into the indication that reaching a difference threshold is indicative of a false positive or false negative, as taught by Montevechi, and the rest of the teachings of Montevechi, in order make sure data includes time-dated information due to changing world conditions. This application of a known technique to the applicable storage medium instructions disclosed by Montevechi, in view of Sodhi, would produce the predictable result claimed in claim 12 . 07-21-aia AIA Claim (s) 7 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Montevechi, in view of Sodhi, in further view of US 20230177326 A1 by Wang et al., hereafter Wang . Regarding claim 7, Montevechi, in view of Sodhi, teaches the material disclosed in claim 1. Montevechi, in view of Sodhi, does not explicitly disclose: wherein the machine learning model is a compressed model that acts as a proxy for another machine learning model. Wang does teach: wherein the machine learning model is a compressed model that acts as a proxy for another machine learning model. ((Wang) paragraph [0022], “…in this embodiment, the neural network model is compressed by the determined first bit width, second bit width and target thinning rate, thereby ensuring that the obtained compression result has higher precision, simplifying compression steps of the neural network model, and improving a compression efficiency of the neural network model.” The obtained compression result is a compression model that acts as a proxy for another machine learning model. ) Wang and Montevechi are in the same area of invention, that being machine learning model training including updating when a difference threshold is reached. Thus, it would have been obvious for a person having ordinary skill in the art before the effective filing date to have combined obtaining a compression result to act as a compressed model proxy, as taught by Wang, for the machine learning model as taught by Montevechi, in view of Sodhi, in order to reduce the cost while still guaranteeing the precision of the model . This application of a known technique to the applicable method disclosed by Montevechi, in view of Sodhi, would produce the predictable result of the method claimed in claim 7. Regarding claim 17, Montevechi, in view of Sodhi, teaches the material disclosed in claim 11. Montevechi, in view of Sodhi, does not explicitly disclose: wherein the machine learning model is a compressed model that acts as a proxy for another machine learning model. Wang does teach: wherein the machine learning model is a compressed model that acts as a proxy for another machine learning model. ((Wang) paragraph [0022], “…in this embodiment, the neural network model is compressed by the determined first bit width, second bit width and target thinning rate, thereby ensuring that the obtained compression result has higher precision, simplifying compression steps of the neural network model, and improving a compression efficiency of the neural network model .” The obtained compression result is a compression model that acts as a proxy for another machine learning model. ) Wang and Montevechi are in the same area of invention, that being machine learning model training including updating when a difference threshold is reached. Thus, it would have been obvious for a person having ordinary skill in the art before the effective filing date to have combined obtaining a compression result to act as a compressed model proxy, as taught by Wang, for the machine learning model as taught by Montevechi, in view of Sodhi, in order to reduce the cost while still guaranteeing the precision of the model . This application of a known technique to the applicable storage medium instructions disclosed by Montevechi, in view of Sodhi, would produce the predictable result claimed in claim 17 . 07-21-aia AIA Claim (s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Montevechi, in view of Sodhi, in further view of US 20240428157 A1 by Makhijani, hereafter Makhijani . Regarding claim 9, Montevechi, in view of Sodhi, teaches the material disclosed in claim 1. Montevechi, in view of Sodhi, does not explicitly disclose: wherein the first known threshold is based on an average of a data distribution difference between the unknown dataset and each of the plurality of shifted datasets. Makhijani does teach: wherein the first known threshold is based on an average of a data distribution difference between the unknown dataset and each of the plurality of shifted datasets. ((Makhijani) paragraph [0055], “To determine the threshold values, the supply state prediction module 315 may first determine a difference between trends of the measured metric signal 307 and the predicted metric signal 312 over a longer time period (e.g., over a 24-hour period, a week, a month, etc.) and then calculate the threshold values as corresponding percentile values (i.e., P-values ) of the difference (e.g., P-70 value, and P-50 value ).” A percentile of 50 is an average. ) Makhijani and Montevechi are in the same area of invention, that being machine learning model training including calculating threshold values and calculating differences between related datasets. Thus, it would have been obvious for a person having ordinary skill in the art before the effective filing date to have combined setting a threshold for a data distribution difference as the 50 th percentile, or average, of the differences, as taught by Makhijani, as the first known threshold for a data distribution difference as taught by Montevechi, in view of Sodhi, in order to create a more dynamic threshold so that large sudden changes do not cause incorrect false positive or false negative determinations. This application of a known technique to the applicable method disclosed by Montevechi, in view of Sodhi, would produce the predictable result of the method claimed in claim 9. Regarding claim 19, Montevechi, in view of Sodhi, teaches the material disclosed in claim 11. Montevechi, in view of Sodhi, does not explicitly disclose: wherein the first known threshold is based on an average of a data distribution difference between the unknown dataset and each of the plurality of shifted datasets. Makhijani does teach: wherein the first known threshold is based on an average of a data distribution difference between the unknown dataset and each of the plurality of shifted datasets. ((Makhijani) paragraph [0055], “To determine the threshold values, the supply state prediction module 315 may first determine a difference between trends of the measured metric signal 307 and the predicted metric signal 312 over a longer time period (e.g., over a 24-hour period, a week, a month, etc.) and then calculate the threshold values as corresponding percentile values (i.e., P-values) of the difference (e.g., P-70 value, and P-50 value ).” A percentile of 50 is an average .) Makhijani and Montevechi are in the same area of invention, that being machine learning model training including calculating threshold values and calculating differences between related datasets. Thus, it would have been obvious for a person having ordinary skill in the art before the effective filing date to have combined setting a threshold for a data distribution difference as the 50th percentile, or average, of the differences, as taught by Makhijani, as the first known threshold for a data distribution difference as taught by Montevechi, in view of Sodhi, in order to create a more dynamic threshold so that large sudden changes do not cause incorrect false positive or false negative determinations. This application of a known technique to the applicable storage medium instructions disclosed by Montevechi, in view of Sodhi, would produce the predictable result claimed in claim 19 . 07-21-aia AIA Claim (s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Montevechi, in view of Sodhi, in further view of US 20230016464 A1 by OConnor et al., hereafter OConnor . Regarding claim 10, Montevechi, in view of Sodhi, teaches the material disclosed in claim 1. Montevechi, in view of Sodhi, does not explicitly disclose: wherein the second known threshold is based on an average of second test metric differences between each of the second test metrics. OConnor does teach: wherein the second known threshold is based on an average of second test metric differences between each of the second test metrics. ((OConnor) paragraph [0054], “As an example, a comparison metric threshold may be determined based on a moving average of the comparison metric through the image sequence , such as by setting a comparison metric threshold to 120% of a moving average of the comparison metrics determined for previous images in the image sequence.” A comparison metric threshold is a second known threshold. A comparison metric is a second test metric difference. ) OConnor and Montevechi are in the same area of invention, that being machine learning model training including observing threshold values and comparing metrics. Thus, it would have been obvious for a person having ordinary skill in the art before the effective filing date to have combined setting a threshold for a metric difference based on the average of the differences, as taught by OConnor, as the second known threshold for a test metric difference as taught by Montevechi, in view of Sodhi, in order to create a more dynamic threshold so that large sudden changes do not cause incorrect false positive or false negative determinations. This application of a known technique to the applicable method disclosed by Montevechi, in view of Sodhi, would produce the predictable result of the method claimed in claim 10. Regarding claim 20, Montevechi, in view of Sodhi, teaches the material disclosed in claim 11. Montevechi, in view of Sodhi, does not explicitly disclose: wherein the second known threshold is based on an average of second test metric differences between each of the second test metrics. OConnor does teach: wherein the second known threshold is based on an average of second test metric differences between each of the second test metrics. ((OConnor) paragraph [0054], “As an example, a comparison metric threshold may be determined based on a moving average of the comparison metric through the image sequence , such as by setting a comparison metric threshold to 120% of a moving average of the comparison metrics determined for previous images in the image sequence.” A comparison metric threshold is a second known threshold. A comparison metric is a second test metric difference. ) OConnor and Montevechi are in the same area of invention, that being machine learning model training including observing threshold values and comparing metrics. Thus, it would have been obvious for a person having ordinary skill in the art before the effective filing date to have combined setting a threshold for a metric difference based on the average of the differences, as taught by OConnor, as the second known threshold for a test metric difference as taught by Montevechi, in view of Sodhi, in order to create a more dynamic threshold so that large sudden changes do not cause incorrect false positive or false negative determinations. This application of a known technique to the applicable storage medium instructions disclosed by Montevechi, in view of Sodhi, would produce the predictable result claimed in claim 20 . Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Patents and/or related publications are cited in the Notice of References Cited (Form PTO-892) attached to this action to further show the state of the art with respect to calculating differences in data distribution and metrics, determining and comparing thresholds, retraining models, generating datasets, validating models, and compression models. Any inquiry concerning this communication or earlier communications from the examiner should be directed to DYLAN H LAI whose telephone number is (571)272-8628. The examiner can normally be reached Monday - Friday 7:30am-5:00pm . Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tamara Kyle can be reached at 5712524241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. / D. H. L./ Examiner Art Unit 2144 /TAMARA T KYLE/ Supervisory Patent Examiner, Art Unit 2144 Application/Control Number: 18/490,999 Page 2 Art Unit: 2144 Application/Control Number: 18/490,999 Page 3 Art Unit: 2144 Application/Control Number: 18/490,999 Page 4 Art Unit: 2144 Application/Control Number: 18/490,999 Page 5 Art Unit: 2144 Application/Control Number: 18/490,999 Page 6 Art Unit: 2144 Application/Control Number: 18/490,999 Page 7 Art Unit: 2144 Application/Control Number: 18/490,999 Page 8 Art Unit: 2144 Application/Control Number: 18/490,999 Page 9 Art Unit: 2144 Application/Control Number: 18/490,999 Page 10 Art Unit: 2144 Application/Control Number: 18/490,999 Page 11 Art Unit: 2144 Application/Control Number: 18/490,999 Page 12 Art Unit: 2144 Application/Control Number: 18/490,999 Page 13 Art Unit: 2144 Application/Control Number: 18/490,999 Page 14 Art Unit: 2144 Application/Control Number: 18/490,999 Page 15 Art Unit: 2144 Application/Control Number: 18/490,999 Page 16 Art Unit: 2144 Application/Control Number: 18/490,999 Page 17 Art Unit: 2144 Application/Control Number: 18/490,999 Page 18 Art Unit: 2144 Application/Control Number: 18/490,999 Page 19 Art Unit: 2144 Application/Control Number: 18/490,999 Page 20 Art Unit: 2144 Application/Control Number: 18/490,999 Page 21 Art Unit: 2144 Application/Control Number: 18/490,999 Page 22 Art Unit: 2144 Application/Control Number: 18/490,999 Page 23 Art Unit: 2144 Application/Control Number: 18/490,999 Page 24 Art Unit: 2144 Application/Control Number: 18/490,999 Page 25 Art Unit: 2144 Application/Control Number: 18/490,999 Page 26 Art Unit: 2144 Application/Control Number: 18/490,999 Page 27 Art Unit: 2144 Application/Control Number: 18/490,999 Page 28 Art Unit: 2144 Application/Control Number: 18/490,999 Page 29 Art Unit: 2144 Application/Control Number: 18/490,999 Page 30 Art Unit: 2144 Application/Control Number: 18/490,999 Page 31 Art Unit: 2144 Application/Control Number: 18/490,999 Page 32 Art Unit: 2144 Application/Control Number: 18/490,999 Page 33 Art Unit: 2144 Application/Control Number: 18/490,999 Page 34 Art Unit: 2144 Application/Control Number: 18/490,999 Page 35 Art Unit: 2144 Application/Control Number: 18/490,999 Page 36 Art Unit: 2144 Application/Control Number: 18/490,999 Page 37 Art Unit: 2144 Application/Control Number: 18/490,999 Page 38 Art Unit: 2144 Application/Control Number: 18/490,999 Page 39 Art Unit: 2144 Application/Control Number: 18/490,999 Page 40 Art Unit: 2144 Application/Control Number: 18/490,999 Page 41 Art Unit: 2144 Application/Control Number: 18/490,999 Page 42 Art Unit: 2144 Application/Control Number: 18/490,999 Page 43 Art Unit: 2144 Application/Control Number: 18/490,999 Page 44 Art Unit: 2144 Application/Control Number: 18/490,999 Page 45 Art Unit: 2144 Application/Control Number: 18/490,999 Page 46 Art Unit: 2144
Prosecution Insights