DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This action is responsive to remarks filed 12/22/2025. Claims 1, 7, 9, 15, 16, 18, and 20 are amended. Claims 3, 13, 17, and 19 have been cancelled, and there are no new claims.
Claims 1–2, 4–12, 14–16, 18, and 20 are pending for examination.
Response to Arguments
In reference to 35 USC § 112(b)
Applicant’s arguments and amendments, filed on 12/22/2025, with respect to the § 112(b) rejections have been fully considered and are persuasive. Thus, the § 112(b) rejections are withdrawn.
In reference to 35 USC § 101
Applicant’s arguments, filed on 12/22/2025, with respect to the § 101 rejections have been fully considered and are persuasive.
Examiner notes that while the claims recite several limitations that are mathematical concepts (including mathematical relationships, mathematical formulas or equations, mathematical calculations), the claims as a whole are not directed to an abstract idea. Applicant amended the claims, which collectively now recite a detailed algorithm directed toward testing machine learning models using test sample uncertainty. The newly amended independent claims now include "wherein the trained proxy model outputs a mean and a variance, and wherein the uncertainty score is computed by subtracting a predetermined constant from the mean and dividing a result of the subtracting by the variance." These additional limitations are not abstract ideas (see MPEP 2106.04(a)). Thus, these limitations must be considered additional elements to the abstract idea. Examiner notes that these additional elements integrate the abstract idea into a practical application because the entire claim amounts to a detailed system that requires implementing a specific combination of hardware with the methods determining, selecting, computing, and using (as opposed to a broad recitation at a high level of generality), and the specific combination of hardware and instructions recited in the additional element amounts to an improvement to the functioning of a computer/field, as set forth by MPEP 2106.05(a)), which states “the claim must include the components or steps of the invention that provide the improvement described in the specification.” Pursuant to this requirement set forth by the MPEP, examiner points out that the Specification states in at least [0012–0020] as provided by Applicant in the Remarks on Pg. 12: “When evaluated as a whole and in light of the specification, the claims are directed to a specific technical solution: using a trained proxy model to compute quantitative uncertainty scores for outputs of a target model, selecting testing data based on those uncertainty scores, and conditionally retesting the target model using distinct subsets of testing data. See, e.g., Spec. ¶¶[0012]-[0020]. This constitutes a practical application that improves the testing and evaluation of machine-learning systems by enabling uncertainty-guided testing workflows-an improvement.” Examiner agrees. Therefore, the additional elements reflect the improvement set forth and explains what the resulting improvement is.
Thus, the additional limitations do amount to significantly more, and the § 101 rejections are withdrawn.
In reference to 35 USC § 102
Applicant’s arguments and amendments, filed on 12/22/2025, with respect to the § 102 rejections and the amended claims have been considered but are not persuasive.
Applicant argues, beginning on pg. 21 of the Remarks, that “Varadarajan's approach continues adding data until success is achieved (i.e., a score tolerance threshold is met). This forward-progressing stopping rule is not the same as selecting a second subset responsive to failure, as expressly required by the amended claim.” Examiner respectfully disagrees. Examiner notes claim 7’s amended language:
selecting, responsive to a second result of the testing failing to satisfy a test success criterion, a second subset of the set of target model testing data, the second subset comprising a plurality of portions of target model testing data not included in the subset of the set of target model testing data; and retesting, using the second subset of the set of target model testing data, the trained target model.
Examiner contends that Varadarajan indeed teaches each feature of the amended claim because Varadarajan teaches, in at least figures 4 – 5 and paragraph [0062], its proxy selectively tests differently sized subsets of the testing data to find a minimum set that meets the overall threshold (i.e., failing to satisfy the success criterion), and adds rows (i.e., data not present in the previous subset). Varadarajan, in at least figures 5 – 6 and paragraph [0062, 0076], teach the rows are added until increasing the sample size no longer improves the model score, which requires retesting of the model as a person having ordinary skill in the art (PHOSITA) would recognize by the passages and the figures cited. Examiner notes that the cited portions and figures of Varadarajan explicitly teach retesting the model using specific subsets of data in at least paragraph [0062, 0076] which states “PANI-ML application 110 retains the version of the ML model (generated during hyperparameter selection stage 240) that was trained, using the final tuned set of hyperparameters, on the strict subset of rows of the training dataset that were identified during ADR stage 230. In this example, this retained version of the ML model is used as the final trained ML model.” Examiner notes that Varadarajan uses a proxy model that is tested more than one time. These concepts are further evinced by Varadarajan in at least paragraphs [0060–0063] which states in part:
Techniques described herein generate multiple subset sizes, from the features ranked according to each of the multiple ranking functions, by using an exponential growth function. The multiple subset sizes resulting from each ranking function are evaluated on the proxy model (P*) representing the selected algorithm (A*). All of these evaluations are ranked, and the feature subset that produces the highest score on the proxy model is selected.
Examiner also notes Varadarajan’s high level discussion on supervised machine learning in at least paragraph [0092–0094] which states in part:
In supervised training, training data is used by a supervised training algorithm to train a machine learning model. The training data includes input and a “known” output, as described above. In an embodiment, the supervised training algorithm is an iterative procedure. In each iteration, the machine learning algorithm applies the model artifact and the input to generate a predicated output. An error or variance between the predicated output and the known output is calculated using an objective function. In effect, the output of the objective function indicates the accuracy of the machine learning model based on the particular state of the model artifact in the iteration. By applying an optimization algorithm based on the objective function, the theta values of the model artifact are adjusted. An example of an optimization algorithm is gradient descent. The iterations may be repeated until a desired accuracy is achieved or some other criteria is met.
Examiner contends a PHOSITA would clearly understand Varadarajan as teaching the aforementioned limitations of amended claims 7 and 15 because Varadarajan teaches iteratively selecting, tuning, and validating machine learning models until a final model is produced. Furthermore, Applicant argues that the amended claim “explicitly excludes data used in the prior subset.” Examiner respectfully disagrees. The claim requires “a second subset of target model testing data … the second subset comprising a plurality of portions of target model testing data not included in the subset of the set of target model testing data” which does not explicitly or implicitly exclude all data used in the prior subset, it merely requires that some of the data not be the same. Thus, Examiner maintains the Primary Reference Varadarajan when rejecting the amended claims in the § 102 and § 103 rejections below.
In reference to 35 USC § 103
Applicant’s arguments, filed on 12/22/2025, with respect to the § 103 rejections and the newly amended limitations have been considered but are not persuasive.
Applicant argues, beginning on pg. 23 of the Remarks, that “there must be some articulated reasonings with some rational underpinning to support the legal conclusion of obviousness.” More specifically, Applicant argues that “the above-identified criteria and rationales are not met.” Examiner respectfully disagrees. Examiner points to the § 103 rejections below for a complete analysis. As an initial matter, examiner relied on Zhang to teach the limitation “wherein the uncertainty score is computed by subtracting a predetermined constant from the mean and dividing a result of the subtracting by the variance” because Zhang teaches normalization techniques that include subtracting a constant (e.g., grayscale) from a mean and then dividing by the variance. This type of 3 term algebraic equation is known in the art as normalization and more specifically, centering. Examiner contends Zhang indeed teaches this type of normalization because Zhang teaches “a mean value and a variance value of input samples may be determined. As another example, the mean value may be subtracted from a grayscale of each pixel point of each sample, and then the subtracted grayscale of each pixel point of each sample may be divided by the variance,” (emphasis added, Zhang paragraph [0151]). As a further example, Examiner notes that Varadarajan, at least in paragraph [0047], discloses “pre-processing stage 210” and “normalization.” Examiner noted that the BRI of the process of subtracting a predetermined constant from a mean and dividing a result by the variance (i.e., the algebraic equation) is normalization. Varadarajan’s recitation of normalization may not explicitly be this type of normalization rising to the level of a § 102 rejection, which is why Examiner relies on Zhang. As a further side note and completely independent, Examiner noted that it would have been obvious for Varadarajan to use this type of normalization because it would have been a simple algebraic sign swap “without loss of generality” and/or “obvious to try.” Examiner contends that the plain meaning of Zhang’s paragraph [0151] meets the § 103 Obviousness rejection standards because Examiner specifically pointed to the section of the identified prior art, included rationale with some articulated underpinnings, and provided some teaching, suggestion, or motivation to combine as originally stated in the Non-Final Rejection mailed 10/01/2025, hereinafter “The Action”.
Applicant argues, beginning on pg. 25 of the Remarks, that “Examiner's position that Zhang teaches or suggests the disputed limitation (which Applicant respectfully submits it does not), the Office Action nevertheless fails to provide a proper rationale for combining Varadarajan and Zhang under KSR.” Examiner respectfully disagrees. Examiner points to the § 103 rejections below for a complete analysis. First, Examiner reminds Applicant that the plain meaning of the cited passages of the prior art teaches all the limitations of claim 1. With regard to the additional “obvious to try” rationale included as an additional piece of evidence, Examiner notes the “Federal Circuit cautioned that an obviousness inquiry based on an obvious to try rationale must always be undertaken in the context of the subject matter in question.” MPEP § 2143.I.E. Here, the context includes artificial intelligence and machine learning techniques i.e., advanced mathematics. Examiner established that the BRI of the process of subtracting a predetermined constant from the mean and dividing a result of the subtracting by the variance (i.e., the algebraic equation) is normalization. Although Varadarajan indeed teaches normalization in at least in paragraph [0047], Examiner did not rely on Varadarajan to teach this more specific type of normalization. Examiner identified this deficiency in The Action and relied on Zhang to cure the deficiencies of Varadarajan. Examiner noted that One would be motivated to do so to improve data quality before inputting the data into a model (Zhang ¶0152):
In some embodiments, after the normalization operation, at least one predicted region identifier probability map output by the first branch network layer, at least one predicted landmark identifier probability map output by the second branch network layer, and at least one predicted plane identifier probability map output by the third branch network layer may be obtained, respectively, by inputting the normalized training sample set (i.e., each training sample image) into the initial neural network model.
Applicant continues to argue that “the Office Action does not identify a finite, identified set of predictable solutions from which the claimed approach would be selected.” Examiner respectfully disagrees. As noted above, the BRI of the amended limitation is normalization using an algebraic equation including 3 terms (e.g., a mean, constant, and variance). Examiner notes that this simple formula has 23 = 8 possible combinations of sign changes (i.e., simple algebraic manipulation). Examiner also noted that this would include a simple sign change from a positive number to a negative number i.e., a specific number of finite solutions. Examiner contends that a PHOSITA of advanced mathematics including machine learning techniques could have easily concluded this fact. Moreover, a PHOSITA would have known that these methods have a reasonable expectation of success because they are mathematically equivalent algebraic manipulations without loss of generality. Finally, examiner already noted some teaching, suggestion, or motivation to combine as originally stated in The Action and below in the § 103 rejections.
Thus, Examiner maintains the § 103 rejections.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 7–8, 10, and 12 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Varadarajan et al., (US-20210390466-A1), hereinafter “Varadarajan.”
Regarding claim 7, Varadarajan teaches:
a computer program product comprising one or more computer readable storage medium, and program instructions collectively stored on the one or more computer readable storage medium, the program instructions executable by a processor to cause the processor to perform operations comprising (Varadarajan ¶0131: “One or more of the functions attributed to any process described herein, may be performed any other logical entity that may or may not be depicted in FIG. 1, according to one or more embodiments. In an embodiment, each of the techniques and/or functionality described herein is performed automatically and may be implemented using one or more computer programs, other software elements, and/or digital logic in any of a general-purpose computer or a special-purpose computer, while performing data retrieval, transformation, and storage operations that involve interacting with and transforming the physical state of memory of the computer” and Varadarajan ¶0134: “Computer system 1100 also includes a main memory 1106, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 1102 for storing information and instructions to be executed by processor 1104. Main memory 1106 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 1104. Such instructions, when stored in non-transitory storage media accessible to processor 1104, render computer system 1100 into a special-purpose machine that is customized to perform the operations specified in the instructions”),
training … a proxy model to (Varadarajan ¶0047: “To illustrate, algorithm selection uses proxy models to quickly rank algorithms, adaptive data reduction uses a proxy model to identify a relevant segment of the dataset for model training, and hyperparameter selection uses a proxy model to bootstrap the process of selecting hyperparameters for the final model”—[(emphasis added)]),
using encoded representations of portions of target model training data and a label corresponding to each portion of the target model training data (Varadarajan ¶0047: “According to an embodiment, the sequence of stages for pipeline 200 starts with a data pre-processing stage 210, during which PANI-ML application 110 performs, on training dataset 122, a set of pre-processing operations, such as missing value imputation, label encoding, and normalization”—[(emphasis added)]),
determine an uncertainty score corresponding to an output of a trained target model, the training resulting in a trained proxy model (Varadarajan Fig. 5, ¶0062: “As depicted in FIG. 5, training dataset 124 is sampled iteratively from a small subset to the full dataset size and each sample is scored by the proxy model (P*) representing the algorithm (A*) selected by algorithm selection. According to an embodiment, the selected subset of data rows is the smallest sample of the dataset that does not sacrifice the quality of the model”);
computing, using the trained proxy model, a set of uncertainty scores, each uncertainty score in the set of uncertainty scores corresponding to a portion of target model testing data in a set of target model testing data (Varadarajan Fig. 5, ¶0062: “As depicted in FIG. 5, training dataset 124 is sampled iteratively from a small subset to the full dataset size and each sample is scored”]);
selecting a subset of the set of target model testing data, the subset comprising a plurality of portions of target model testing data each having an uncertainty score above a threshold uncertainty score (Varadarajan Figs. 3, 5, ¶¶0051–0052, ¶0062: “As depicted in FIG. 5, training dataset 124 is sampled iteratively from a small subset to the full dataset size and each sample is scored by the proxy model (P*) representing the algorithm (A*) selected by algorithm selection. According to an embodiment, the selected subset of data rows is the smallest sample of the dataset that does not sacrifice the quality of the model. Consecutive sample sizes, with increasing size, are tried on P* until a score tolerance threshold is met, wherein increasing the sample size no longer improves the model score by more than a threshold (e.g., 1%). If all of the sample sizes are exhausted without meeting the threshold, all rows of the feature-selected dataset are used for hyperparameter selection”—[wherein the BRI of selecting is any action that separates the target model testing data (see present disclosure ¶0021), and wherein the data is system selects the samples with increasing size until the score threshold is met]);
testing, using the subset of the set of target model testing data, the trained target model (Varadarajan Fig. 10, ¶0085, ¶0127: “The trained model is evaluated using the test and validation datasets, as described above”);
selecting, responsive to a second result of the testing failing to satisfy a test success criterion, a second subset of the set of target model testing data, the second subset comprising a plurality of portions of target model testing data not included in the subset of the set of target model testing data (Varadarajan Figs. 2–6, ¶0056, ¶¶0060–0062: “FIG. 5 depicts a row sampling block diagram comprising dataset sampling, proxy model evaluation, and ranking of obtained cross-validation scores used to select the best dataset sample and class distribution. As depicted in FIG. 5, training dataset 124 is sampled iteratively from a small subset to the full dataset size and each sample is scored by the proxy model (P*) representing the algorithm (A*) selected by algorithm selection. According to an embodiment, the selected subset of data rows is the smallest sample of the dataset that does not sacrifice the quality of the model. Consecutive sample sizes, with increasing size, are tried on P* until a score tolerance threshold is met, wherein increasing the sample size no longer improves the model score by more than a threshold (e.g., 1%). If all of the sample sizes are exhausted without meeting the threshold, all rows of the feature-selected dataset are used for hyperparameter selection”—[wherein the proxy selectively tests differently sized subsets of the testing data to find a minimum set that meets the overall threshold (i.e., failing to satisfy the success criterion), and adds rows (i.e., data not present in the previous subset)]); and
retesting, using the second subset of the set of target model testing data, the trained target model (Varadarajan Figs. 2–6, ¶0056, ¶¶0060–0062: “FIG. 5 depicts a row sampling block diagram comprising dataset sampling, proxy model evaluation, and ranking of obtained cross-validation scores used to select the best dataset sample and class distribution. As depicted in FIG. 5, training dataset 124 is sampled iteratively from a small subset to the full dataset size and each sample is scored by the proxy model (P*) representing the algorithm (A*) selected by algorithm selection. According to an embodiment, the selected subset of data rows is the smallest sample of the dataset that does not sacrifice the quality of the model. Consecutive sample sizes, with increasing size, are tried on P* until a score tolerance threshold is met, wherein increasing the sample size no longer improves the model score by more than a threshold (e.g., 1%). If all of the sample sizes are exhausted without meeting the threshold, all rows of the feature-selected dataset are used for hyperparameter selection” and Varadarajan ¶0076: “As another example, to conserve resources and training time, PANI-ML application 110 causes the ML model to be trained on a strict subset of rows of the training dataset that were identified during ADR stage 230. Alternatively, PANI-ML application 110 retains the version of the ML model (generated during hyperparameter selection stage 240) that was trained, using the final tuned set of hyperparameters, on the strict subset of rows of the training dataset that were identified during ADR stage 230. In this example, this retained version of the ML model is used as the final trained ML model”—[wherein the rows are added until increasing the sample size no longer improves the model score, i.e., retesting of the model]).
Regarding claim 8, Varadarajan teaches all the limitations of claim 7.
Varadarajan teaches:
wherein the stored program instructions are stored in a computer readable storage device in a data processing system (Varadarajan ¶0131: “One or more of the functions attributed to any process described herein, may be performed any other logical entity that may or may not be depicted in FIG. 1, according to one or more embodiments. In an embodiment, each of the techniques and/or functionality described herein is performed automatically and may be implemented using one or more computer programs, other software elements, and/or digital logic in any of a general-purpose computer or a special-purpose computer, while performing data retrieval, transformation, and storage operations that involve interacting with and transforming the physical state of memory of the computer”), and
wherein the stored program instructions are transferred over a network from a remote data processing system (Varadarajan ¶¶0143–0144: “Computer system 1100 can send messages and receive data, including program code, through the network(s), network link 1120 and communication interface 1118. In the Internet example, a server 1130 might transmit a requested code for an application program through Internet 1128, ISP 1126, local network 1122 and communication interface 1118. The received code may be executed by processor 1104 as it is received, and/or stored in storage device 1110, or other non-volatile storage for later execution”).
Regarding claim 10, Varadarajan teaches all the limitations of claim 7.
Varadarajan teaches:
generating, using a third trained neural network model, an encoded representation of a portion of the target model training data (Varadarajan ¶¶0118–0119, ¶0124: “Sophisticated analysis may be achieved by a so-called stack of MLPs. An example stack may sandwich an RNN between an upstream encoder ANN and a downstream decoder ANN, either or both of which may be an autoencoder”).
Regarding claim 12, Varadarajan teaches all the limitations of claim 7.
Varadarajan teaches:
deploying, responsive to a first result of the testing satisfying a test success criterion, the trained target model (Varadarajan ¶0083: “The benefits of an AutoML optimizer are measured using two main metrics: (1) the goodness of the final tuned model, called model performance or score; and (2) the amount of time and resources used to achieve that score, called speed or efficiency” and Varadarajan ¶¶0155–0158: “Generally, a cloud computing model enables some of those responsibilities which previously may have been provided by an organization's own information technology department, to instead be delivered as service layers within a cloud environment, for use by consumers (either within or external to the organization, according to the cloud's public/private nature)”—[wherein cloud computing is used to deploy (see present disclosure ¶¶0031–0045) the final tuned model (i.e., the trained target model) after determining the “goodness” of the final tuned model and the efficiency (i.e., test success criterion)]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1–2, 4–5, 11, 15–16, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Varadarajan (US-20210390466-A1) in view of Zhang et al., (US 20230419499 A1), hereinafter “Zhang”.
Regarding claim 1, Varadarajan teaches:
Training … a proxy model to (Varadarajan ¶0047: “To illustrate, algorithm selection uses proxy models to quickly rank algorithms, adaptive data reduction uses a proxy model to identify a relevant segment of the dataset for model training, and hyperparameter selection uses a proxy model to bootstrap the process of selecting hyperparameters for the final model”—[(emphasis added)]),
using encoded representations of portions of target model training data and a label corresponding to each portion of the target model training data (Varadarajan ¶0047: “According to an embodiment, the sequence of stages for pipeline 200 starts with a data pre-processing stage 210, during which PANI-ML application 110 performs, on training dataset 122, a set of pre-processing operations, such as missing value imputation, label encoding, and normalization”—[(emphasis added)]),
determine an uncertainty score corresponding to an output of a trained target model, the training resulting in a trained proxy model (Varadarajan Fig. 5, ¶0062: “As depicted in FIG. 5, training dataset 124 is sampled iteratively from a small subset to the full dataset size and each sample is scored by the proxy model (P*) representing the algorithm (A*) selected by algorithm selection. According to an embodiment, the selected subset of data rows is the smallest sample of the dataset that does not sacrifice the quality of the model”),
wherein the trained proxy model outputs a mean and a variance (Varadarajan ¶0081: “For CV, the mean of K-fold scores is maximized” and Varadarajan ¶0125: “Predictions for the time-series are calculated based on the mean of the predictions from the different decision trees” and Varadarajan ¶0094: “An error or variance between the predicated output and the known output is calculated using an objective function”),
computing, using the trained proxy model, a set of uncertainty scores, each uncertainty score in the set of uncertainty scores corresponding to a portion of target model testing data in a set of target model testing data (Varadarajan Fig. 5, ¶0062: “As depicted in FIG. 5, training dataset 124 is sampled iteratively from a small subset to the full dataset size and each sample is scored”]);
selecting a subset of the set of target model testing data, the subset comprising a plurality of portions of target model testing data each having an uncertainty score above a threshold uncertainty score (Varadarajan Figs. 3, 5, ¶¶0051–0052, ¶0062: “As depicted in FIG. 5, training dataset 124 is sampled iteratively from a small subset to the full dataset size and each sample is scored by the proxy model (P*) representing the algorithm (A*) selected by algorithm selection. According to an embodiment, the selected subset of data rows is the smallest sample of the dataset that does not sacrifice the quality of the model. Consecutive sample sizes, with increasing size, are tried on P* until a score tolerance threshold is met, wherein increasing the sample size no longer improves the model score by more than a threshold (e.g., 1%). If all of the sample sizes are exhausted without meeting the threshold, all rows of the feature-selected dataset are used for hyperparameter selection”—[wherein the BRI of selecting is any action that separates the target model testing data (see present disclosure ¶0021), and wherein the data is system selects the samples with increasing size until the score threshold is met]); and
testing, using the subset of the set of target model testing data, the trained target model (Varadarajan Fig. 10, ¶0085, ¶0127: “The trained model is evaluated using the test and validation datasets, as described above”).
Varadarajan does not appear to explicitly teach:
wherein the uncertainty score is computed by subtracting a predetermined constant from the mean and dividing a result of the subtracting by the variance.
However, Zhang teaches:
wherein the uncertainty score is computed by subtracting a predetermined constant from the mean and dividing a result of the subtracting by the variance (Zhang ¶0151: “The normalization operation may be performed in various manners, which are not limited in the present disclosure. For example, a mean value and a variance value of input samples may be determined. As another example, the mean value may be subtracted from a grayscale of each pixel point of each sample, and then the subtracted grayscale of each pixel point of each sample may be divided by the variance”—[(emphasis added) Examiner notes that Varadarajan, at least in ¶0047, discloses “pre-processing stage 210” and “normalization” and wherein the BRI of the process of subtracting a predetermined constant from a mean and dividing a result by the variance is normalization. Varadarajan’s recitation of normalization may not explicitly be this type of normalization as there are many types. However, Zhang teaches normalization techniques that include subtracting a constant (e.g., grayscale) from a mean and then dividing by the variance. Examiner notes that Zhang’s normalization subtracts the mean from the constant which would have been an “obvious to try” variant and/or a simple, and without loss of generality, reversed sign]).
The systems of Varadarajan, the teachings of Zhang, and the instant application are analogous art because they pertain statistical analysis using predictive machine learning models.
It would be obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the systems of Varadarajan with the teachings of Zhang to provide for a specific normalization technique to normalize data. One would be motivated to do so to approximate the sampling distribution to better fit the distribution in order to significantly improve the predictive performance (Zhang ¶0055–0057).
Regarding claim 2, Varadarajan in view of Zhang teaches all the limitations of claim 1.
Varadarajan teaches:
generating, using a third trained neural network model, an encoded representation of a portion of the target model training data (Varadarajan ¶¶0118–0119, ¶0124: “Sophisticated analysis may be achieved by a so-called stack of MLPs. An example stack may sandwich an RNN between an upstream encoder ANN and a downstream decoder ANN, either or both of which may be an autoencoder”).
Regarding claim 4, Varadarajan in view of Zhang teaches all the limitations of claim 1.
Varadarajan teaches:
deploying, responsive to a first result of the testing satisfying a test success criterion, the trained target model (Varadarajan ¶0083: “The benefits of an AutoML optimizer are measured using two main metrics: (1) the goodness of the final tuned model, called model performance or score; and (2) the amount of time and resources used to achieve that score, called speed or efficiency” and Varadarajan ¶¶0155–0158: “Generally, a cloud computing model enables some of those responsibilities which previously may have been provided by an organization's own information technology department, to instead be delivered as service layers within a cloud environment, for use by consumers (either within or external to the organization, according to the cloud's public/private nature)”—[wherein cloud computing is used to deploy (see present disclosure ¶¶0031–0045) the final tuned model (i.e., the trained target model) after determining the “goodness” of the final tuned model and the efficiency (i.e., test success criterion)]).
Regarding claim 5, Varadarajan in view of Zhang teaches all the limitations of claim 1.
Varadarajan teaches:
selecting, responsive to a second result of the testing failing to satisfy a test success criterion, a second subset of the set of target model testing data, the second subset comprising a plurality of portions of target model testing data not included in the subset of the set of target model testing data (Varadarajan Figs. 2–6, ¶0056, ¶¶0060–0062: “FIG. 5 depicts a row sampling block diagram comprising dataset sampling, proxy model evaluation, and ranking of obtained cross-validation scores used to select the best dataset sample and class distribution. As depicted in FIG. 5, training dataset 124 is sampled iteratively from a small subset to the full dataset size and each sample is scored by the proxy model (P*) representing the algorithm (A*) selected by algorithm selection. According to an embodiment, the selected subset of data rows is the smallest sample of the dataset that does not sacrifice the quality of the model. Consecutive sample sizes, with increasing size, are tried on P* until a score tolerance threshold is met, wherein increasing the sample size no longer improves the model score by more than a threshold (e.g., 1%). If all of the sample sizes are exhausted without meeting the threshold, all rows of the feature-selected dataset are used for hyperparameter selection”—[wherein the proxy selectively tests differently sized subsets of the testing data to find a minimum set that meets the overall threshold (i.e., failing to satisfy the success criterion), and adds rows (i.e., data not present in the previous subset)]); and
retesting, using the second subset of the set of target model testing data, the trained target model (Varadarajan Figs. 2–6, ¶0056, ¶¶0060–0062: “FIG. 5 depicts a row sampling block diagram comprising dataset sampling, proxy model evaluation, and ranking of obtained cross-validation scores used to select the best dataset sample and class distribution. As depicted in FIG. 5, training dataset 124 is sampled iteratively from a small subset to the full dataset size and each sample is scored by the proxy model (P*) representing the algorithm (A*) selected by algorithm selection. According to an embodiment, the selected subset of data rows is the smallest sample of the dataset that does not sacrifice the quality of the model. Consecutive sample sizes, with increasing size, are tried on P* until a score tolerance threshold is met, wherein increasing the sample size no longer improves the model score by more than a threshold (e.g., 1%). If all of the sample sizes are exhausted without meeting the threshold, all rows of the feature-selected dataset are used for hyperparameter selection” and Varadarajan ¶0076: “As another example, to conserve resources and training time, PANI-ML application 110 causes the ML model to be trained on a strict subset of rows of the training dataset that were identified during ADR stage 230. Alternatively, PANI-ML application 110 retains the version of the ML model (generated during hyperparameter selection stage 240) that was trained, using the final tuned set of hyperparameters, on the strict subset of rows of the training dataset that were identified during ADR stage 230. In this example, this retained version of the ML model is used as the final trained ML model”—[wherein the rows are added until increasing the sample size no longer improves the model score, i.e., the retesting of the model]).
Regarding claim 11, Varadarajan teaches all the limitations of claim 7.
Varadarajan teaches:
wherein the trained proxy model outputs a mean and a variance (Varadarajan ¶0081: “For CV, the mean of K-fold scores is maximized” and Varadarajan ¶0125: “Predictions for the time-series are calculated based on the mean of the predictions from the different decision trees” and Varadarajan ¶0094: “An error or variance between the predicated output and the known output is calculated using an objective function”),
Varadarajan does not appear to explicitly teach:
wherein the uncertainty score is computed by subtracting a predetermined constant from the mean and dividing a result of the subtracting by the variance.
However, Zhang teaches:
wherein the uncertainty score is computed by subtracting a predetermined constant from the mean and dividing a result of the subtracting by the variance (Zhang ¶0151: “The normalization operation may be performed in various manners, which are not limited in the present disclosure. For example, a mean value and a variance value of input samples may be determined. As another example, the mean value may be subtracted from a grayscale of each pixel point of each sample, and then the subtracted grayscale of each pixel point of each sample may be divided by the variance”—[(emphasis added) Examiner notes that Varadarajan, at least in ¶0047, discloses “pre-processing stage 210” and “normalization” and wherein the BRI of the process of subtracting a predetermined constant from a mean and dividing a result by the variance is normalization. Varadarajan’s recitation of normalization may not explicitly be this type of normalization as there are many types. However, Zhang teaches normalization techniques that include subtracting a constant (e.g., grayscale) from a mean and then dividing by the variance. Examiner notes that Zhang’s normalization subtracts the mean from the constant which would have been an “obvious to try” variant and/or a simple, and without loss of generality, reversed sign]).
The systems of Varadarajan, the teachings of Zhang, and the instant application are analogous art because they pertain statistical analysis using predictive machine learning models.
It would be obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the systems of Varadarajan with the teachings of Zhang to provide for a specific normalization technique to normalize data. One would be motivated to do so to approximate the sampling distribution to better fit the distribution in order to significantly improve the predictive performance (Zhang ¶0055–0057).
Regarding claim 15, Varadarajan teaches:
a computer system comprising a processor and one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by the processor to cause the processor to perform operations comprising (Varadarajan ¶0134: “Computer system 1100 also includes a main memory 1106, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 1102 for storing information and instructions to be executed by processor 1104. Main memory 1106 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 1104. Such instructions, when stored in non-transitory storage media accessible to processor 1104, render computer system 1100 into a special-purpose machine that is customized to perform the operations specified in the instructions”).
Regarding the remaining limitations of claim 15, although varying in scope, the remaining limitations of claim 15 are substantially the same as the limitations of claims 1 and 5. Thus, the remaining limitations of claim 15 are rejected using the same reasoning and analysis as claims 1 and 5, above.
Regarding claims 16 and 18, although varying in scope, the limitations of claims 16 and 18 are substantially the same as the limitations of claims 2 and 4, respectively. Thus, claims 16 and 18 are rejected using the same reasoning and analysis as claims 2 and 4 above, respectively.
Claims 6, 14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Varadarajan in view of Zhang, and further in view of Nickolov et al., (US 20170034023 A1), hereinafter “Nickolov”.
Regarding claim 6, Varadarajan in view of Zhang teaches all the limitations of claim 1.
Varadarajan in view of Zhang does not appear to explicitly teach:
rejecting, responsive to the testing failing to satisfy a test success criterion, the trained target model for deployment.
However, Nickolov teaches:
rejecting, responsive to the testing failing to satisfy a test success criterion, the trained target model for deployment (Nickolov Col. 23, lines 9–41: “The Subscriber DevOps systems 418 may query the DataGrid API 425 (directly or via anonymization gateway 419) in order to make deployment and operations decisions (similar to what Config Automation 412 does), including whether to upgrade packages/change configuration, or even deploy new versions of the software and systems CVE 430 (Common Vulnerabilities and Exploits) info database (e.g., third party) Existing database of known/reported vulnerabilities. The DataGrid Analytics Service may use these to provide supplementary information about packages and configurations (not gathered directly through DataGrid telemetry collection of the Tracked Servers). Example Input: usually provided by vendors and security researchers; the DataGrid Analytics Service have provided input as well (esp. regarding inconsistencies in the database necessitating corrections). Example Output: list of vulnerabilities, what software/configurations are affected, and what software versions are affected. Vendor Info 440 (or Community Info) (e.g., third party). Existing database of known/reported vulnerabilities provided by vendors and/or community for OS or other software package and configuration artifact. In addition to vulnerabilities, those may include quality information about configurations and package versions, such as: Known bugs; Test code coverage; Test pass/failure details; Discussion forums where issues are discussed; Etc.”—[(emphasis added) wherein the system uses the test pass/failure details to determine whether or not (e.g., not, i.e., reject) deployment of the software package (i.e., model)]).
The systems of Varadarajan, the teachings of Nickolov, and the instant application are analogous art because they pertain evaluating software using machine learning models.
It would be obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the systems of Varadarajan with the teachings of Nickolov to provide for a mechanism to reject the non-passing models. One would be motivated to do so in order to make decisions on whether or not to deploy the model (Nickolov Col. 23, lines 9–41: “The Subscriber DevOps systems 418 may query the DataGrid API 425 (directly or via anonymization gateway 419) in order to make deployment and operations decisions (similar to what Config Automation 412 does), including whether to upgrade packages/change configuration, or even deploy new versions of the software and systems”).
Regarding claims 14 and 20, although varying in scope, the limitations of claims 14 and 20 are substantially the same as the limitations of claim 6. Thus, claims 14 and 20 are rejected using the same reasoning and analysis as claim 6 above.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Varadarajan (US-20210390466-A1) as applied above in § 102, and further in view of Gan et al., (US 11263003 B1), hereinafter “Gan”.
Regarding claim 9, Varadarajan teaches all the limitations of claim 7.
Varadarajan teaches:
wherein the stored program instructions are stored in a computer readable storage device in a server data processing system (Varadarajan ¶0131: “One or more of the functions attributed to any process described herein, may be performed any other logical entity that may or may not be depicted in FIG. 1, according to one or more embodiments. In an embodiment, each of the techniques and/or functionality described herein is performed automatically and may be implemented using one or more computer programs, other software elements, and/or digital logic in any of a general-purpose computer or a special-purpose computer, while performing data retrieval, transformation, and storage operations that involve interacting with and transforming the physical state of memory of the computer”), and
wherein the stored program instructions are downloaded in response to a request over a network to a remote data processing system for use in a computer readable storage device associated with the remote data processing system (Varadarajan ¶¶0143–0144: “Computer system 1100 can send messages and receive data, including program code, through the network(s), network link 1120 and communication interface 1118. In the Internet example, a server 1130 might transmit a requested code for an application program through Internet 1128, ISP 1126, local network 1122 and communication interface 1118. The received code may be executed by processor 1104 as it is received, and/or stored in storage device 1110, or other non-volatile storage for later execution”),
Varadarajan does not appear to explicitly teach:
further comprising: program instructions to meter use of the program instructions associated with the request; and
program instructions to generate an invoice based on the metered use.
However, Gan teaches:
further comprising: program instructions to meter use of the program instructions associated with the request; and program instructions to generate an invoice based on the metered use (Gan Claim 13: “The computer program product of claim 11, wherein the stored program instructions are stored in a computer readable storage device in a server data processing system, and wherein the stored program instructions are downloaded in response to a request over a network to a remote data processing system for use in a computer readable storage device associated with the remote data processing system, further comprising: program instructions to meter use of the program instructions associated with the request; and program instructions to generate an invoice based on the metered use”).
The systems of Varadarajan, the teachings of Gan, and the instant application are analogous art because they pertain to training and testing machine learning models.
It would be obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the systems of Varadarajan with the teachings of Gan to provide for instructions that will track the usage of the downloaded software to generate a bill for the service. One would be motivated to do so to more efficiently service engaged client corporations and the like (Gan Col. 21, lines 5–24: “Embodiments of the present invention may also be delivered as part of a service engagement with a client corporation, nonprofit organization, government entity, internal organizational structure, or the like. Aspects of these embodiments may include configuring a computer system to perform, and deploying software, hardware, and web services that implement, some or all of the methods described herein. Aspects of these embodiments may also include analyzing the client's operations, creating recommendations responsive to the analysis, building systems that implement portions of the recommendations, integrating the systems into existing processes and infrastructure, metering use of the systems, allocating expenses to users of the systems, and billing for use of the systems”).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS SHINE whose telephone number is (571)272-2512. The examiner can normally be reached M-F, 9a-5p ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached on (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.B.S./Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126