Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination (RCE) under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on September 14, 2026, has been entered.
Remarks
This Office Action is in response to applicant’s amendment and RCE filed on September 14, 2026, under which claims 1-20 are pending and under consideration.
Response to Arguments
Applicant’s amendments have overcome the previous grounds of rejection under § 112 and § 103. However, upon further consideration, new grounds of rejection have been made under § 103.
Applicant’s arguments for the previous § 103 rejections have been fully considered but are moot under the new grounds of rejection because the new limitations relied upon are addressed by new references, including Yan et al.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
1. Claims 1, 6-8, 13-15, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Qi et al. (US 2021/0304055 A1) (“Qi”) in view of Rebuffi et al., “Learning multiple visual domains with residual adapters,” arXiv:1705.08045v5 [cs.CV] 27 Nov 2017 (“Rebuffi”) and Yan et al., Positive-Congruent Training: Towards Regression-Free Model Updates,” arXiv:2011.09161v3 [cs.CV] 17 May 2021 (“Yan”).
As to claim 1, Qi teaches a computer-implemented [[0007]: “The system/apparatus may comprise one or more processors and a memory coupled to the one or more processors. The memory may comprise instructions which, when executed by the one or more processors, cause the one or more processors to perform various ones of, and combinations of, the operations outlined above with regard to the method illustrative embodiment.”] method comprising:
initializing a machine learning algorithm with a set of hyperparameter values; [[0037]: “For example, an initial set of hyperparameter values for N parameters of a computer model may be set as beta=(param1, param2, . . . paramN), along with an initial upper bound and lower bound for each of these hyperparameter values.” The computer model is a “machine learning computer model” ([0018]).]
accessing a hyperparameter objective function that is defined at least in part on a plurality of domains of a search space that is associated with the machine learning algorithm, [[0035]: “As shown in FIG. 1, the AutoML process 110, comprises multiple stages of logic 112-116 which are repeatedly executed until the hyperparameter value settings for the ML model 140 are optimized, e.g., a particular time constraint is met, a particular amount of improvement in the performance of the model, e.g., a loss or error, is equal to or less than a particular threshold, a particular amount of improvement in performance is not able to be achieved, or any other suitable stopping criteria for the AutoML process 110.” See also [0040]-[0041] for examples. That is, the performance measure corresponds to a “hyperparameter objective function.” It is defined on a plurality of domains, since the data used in the training includes a plurality of domains. A “plurality of domains” is disclosed in [0005]: “executing an initial AutoML process on the machine learning model based on a plurality of datasets comprising a plurality of domains of data elements, utilizing the initially configured AutoML logic.” See also [0041]: “The one or more input datasets 130 may comprise data in a variety of different domains, e.g., domains D1-D4.”] wherein the search space comprises training datasets and evaluation datasets with each domain comprising a subdivision of the search space with at least one training dataset and at least one evaluation dataset, and wherein the hyperparameter objective function comprises […]; [[0042]: “In the case of training data, the labels of the data in the datasets 130 further comprise a ground truth classification or output that is a correct output of a properly trained ML model given the corresponding input data. This ground truth output may be compared to the actual output generated by the ML model 140 to evaluate a performance of the ML model 140 with regard to the correct output, e.g., a loss function may be used to calculate an error in the ML model 140 output” [0047]: “…trained on one or more input datasets, e.g., one or more datasets 132-138 having labeled domains, to recognize patterns of content indicative of particular domains and thereby classify data elements of the input datasets into a plurality of predefined domains, e.g., domains D1-D4, where an input dataset 132 may comprise data elements of various domains D1-D4 such that it represents a mixed domain dataset.” In regards to the limitation of an “evaluation dataset” for each domain, the above part of [0042] teaches that the dataset is both the “training dataset” and the “evaluation dataset.” See also [0041], which refers to “evaluate the performance of the ML model 140, the AutoML process 110 comprises a second stage 114” with the use of the input datasets 130.]
for each trial of a hyperparameter tuning process: [The disclosed process is for “determination of the best performance set of hyperparameters for configuring the machine learning model” ([0020]). The limitation of “for each trial” does not require a plurality of trials. Nonetheless, it is taught that the process repeats, in the form of multiple trials. See [0066]: “This process may be repeated for each subsequent new dataset 310 received such that the learning of the hyperparameter sampling configuration parameters is continuously or periodically updated.”]
training the machine learning algorithm for each domain using the at least one training dataset associated with each domain and the set of hyperparameter values, wherein the training outputs a plurality of machine learning models comprising a machine learning model for each domain; [[0060]: “As shown in FIG. 2, for each of these datasets, a learned value for the hyperparameter param1 is determined through the AutoML process, e.g., AutoML process 110 in FIG. 1.” As noted above, training is described in [0042]: “In the case of training data, the labels of the data in the datasets 130 further comprise a ground truth classification or output that is a correct output of a properly trained ML model given the corresponding input data. This ground truth output may be compared to the actual output generated by the ML model 140 to evaluate a performance of the ML model 140 with regard to the correct output.” Furthermore, as illustrated in FIG. 2, the same domains (e.g., D1 and D2) appear across multiple workspaces, each of which has a corresponding learned value. The limitation of a plurality of machine learning models is described because as shown in FIG. 2, different learned values correspond to different machine learning models.]
evaluating the machine learning model for each domain using the at least one evaluation dataset associated with each domain and the set of hyperparameter values, wherein the evaluating comprises generating the […]; [[0058]: “This initial set of hyperparameter sampling configuration parameters 210 are used to perform an initial AutoML process, such as described previously with regard to FIG. 1, and thereby generate for performance metrics for each sampled set of hyperparameter values which are then used to identify a particular setting of hyperparameter values that provide a best, or optimum, performance of the ML model when the ML model is configured with the selected set of hyperparameter values. As shown in FIGS. 2-4, the value of hyperparameter param1 that provides the best performance of the ML model based on the evaluation of the performance metrics generated by this initial AutoML operation is the “learned” value for the hyperparameter.” That is, for each workspace in FIG. 2, the performance is calculated in order to derive the optimal value that is learned for a particular hyperparameter.]
calculating, using the hyperparameter objective function, a current trial objective score based on the […] and a domain weight associated with each domain; [[0040]: “the performance of the ML model 140, configured with the hyperparameter values corresponding to the selected set of hyperparameter values (beta*i), is evaluated with regard to one or more performance metrics of interest, e.g., accuracy of the output of the ML model 140 relative to a ground truth as may be measured by a loss function, for example.”] and
determining whether the machine learning model has reached convergence based on the current trial objective score; [[0035]: “As shown in FIG. 1, the AutoML process 110, comprises multiple stages of logic 112-116 which are repeatedly executed until the hyperparameter value settings for the ML model 140 are optimized, e.g., a particular time constraint is met, a particular amount of improvement in the performance of the model, e.g., a loss or error, is equal to or less than a particular threshold, a particular amount of improvement in performance is not able to be achieved, or any other suitable stopping criteria for the AutoML process 110.”] and
in response to determining the machine learning model has reached convergence, providing at least one of the plurality of machine learning models. [[0035]: “Once the AutoML process 110 completes, a set of learned hyperparameter values are generated that provide an optimum performance of the ML model 140 during training of the ML model 140 and/or runtime deployment of the ML model 140. That is, this set of learned hyperparameters are learned from the AutoML process 110 and may be used to configure an instance of the ML model for training using a training dataset and/or in a runtime environment which processes new workloads using the configured model.”]
Qi does not teach:
(1) the limitation that the hyperparameter objective function comprises “a domain score for each domain that is calculated based on a number of instances within the at least one evaluation dataset that are correctly or incorrectly predicted by the machine learning algorithm during a given trial” and the related limitation of the use of “the domain score for each domain” for the evaluating and the limitation of “based on the domain score for each domain and a domain weight associated with each domain” for calculation of the current trial objective score; and
(2) “wherein the quantification of instance-level changes comprises calculating a first intersection of specific instances within the at least one evaluation dataset correctly predicted during the given trial and incorrectly predicted by the baseline model, and calculating a second intersection of specific instances within the at least one evaluation dataset incorrectly predicted during the given trial and correctly predicted by the baseline model.”
Rebuffi teaches “a domain score for each domain that is calculated based on a number of instances within the at least one evaluation dataset that are correctly or incorrectly predicted by the machine learning algorithm during a given trial” [§ 4, paragraph 4: “Performance is measured in terms of a single scalar score S determined as in the decathlon discipline. Performing well at this metric requires algorithms to perform well in all tasks, compared to a minimum level of baseline performance for each. In detail, S is computed as follows: [See equations in expression (1)] where Ed is the average test error for each domain… The coefficient αd is set to 1, 000 (Emaxd)−γd so that a perfect result receives a score of 1,000 (10,000 in total).” That is, referring to equation 1, αdmax{0, Edmax – Ed}γd constitutes a domain score for domain d. Note that as further shown in the formula for Ed, this measures the test error, which is measured over the number of instances in the evaluation set Ddtest.] and thus the related limitation of use of “the domain score for each domain” for evaluating and for calculation of a current trial objective score “based on the domain score for each domain and a domain weight associated with each domain” [As noted above, the individual scores Ed (or max{0, Edmax – Ed}) are used to compute the performance scalar score S and is based on the domain weight αd. This scalar score is analogous to the performance metric disclosed in the base reference.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of Qi with the teachings of Rebuffi by implementing domain-specific scoring technique of Rebuffi in the performance measurement process of Qi, specifically by implementing the hyperparameter objective function to comprise “a domain score for each domain that is calculated based on a number of instances within the at least one evaluation dataset that are correctly or incorrectly predicted by the machine learning algorithm during a given trial”, the use of “the domain score for each domain” for the evaluating, and for calculation of the current trial objective score so as to be “based on the domain score for each domain and a domain weight associated with each domain”, so as to arrive at the above features of the instant claim. The motivation would have been to implement a measure of performance that evaluates a model for multiple-domain learning, in a way that assesses whether a method can successfully learn to perform well in several different domains at the same time, as suggested by Rebuffi, § 4, paragraph 1 (“In this section we introduce a new benchmark, called visual decathlon, to evaluate the performance of algorithms in multiple-domain learning. The goal of the benchmark is to assess whether a method can successfully learn to perform well in several different domains at the same time.”).
The combination of references thus far does not teach limitations (2) listed above.
Raschka teaches “wherein the quantification of instance-level changes comprises calculating a first intersection of specific instances within the at least one evaluation dataset correctly predicted during the given trial and incorrectly predicted by the baseline model, and calculating a second intersection of specific instances within the at least one evaluation dataset incorrectly predicted during the given trial and correctly predicted by the baseline model.” [§ 3.1, paragraph 1: “In Fig. 1 we illustrate the different errors occuring after a model upgrade, comparing the predictions ŷold and ŷnew. The consistent areas are where ϕwold and ϕwnew are both either correct (ŷold(xi) = ŷnew(xi) = yi) or incorrect (ŷold(xi) ≠ yi, ŷnew(xi) ≠ yi). Positive flips occur when samples were incorrectly predicted by ϕwold, but correctly predicted by ϕwnew. The most relevant to this paper are negative flips, where samples flipped from correct ϕwold to incorrect predictions ϕwnew. The negative flip rate (NFR) measures the fraction of samples that are negatives…” That is, referring to the 2x2 diagram in FIG. 1, the positive flips correspond to a first intersection (predicted correctly by a new model and not by the old model), and the negative flips correspond to the second intersection. Note that the concept of new model and old model in this paper is analogous to a tested proposed model and a baseline model. See abstract: “A new model incorrectly predicts the output for a test sample that was correctly classified by the old (reference) model.” See also FIG. 1 caption: “Figure 1: Regression in model update: When updating an old classifier (red) to a new one (dashed blue line), we correct mistakes (top-right, white), but we also introduce errors that the old classifier did not make (negative flips, bottom-left, red). While on average the errors decrease (from 57% to 42% in this toy example), regression can wreak havoc with downstream processing, nullifying the benefit of the update.”]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far with the teachings of Yan by implementing the use of positive and negative test for comparison with a baseline model, so as to arrive at the claimed invention. Doing so would have enabled comparison between two models, as suggested by Yan (§ 3.1, paragraph 1: “In Fig. 1 we illustrate the different errors occuring after a model upgrade, comparing the predictions”), specifically in the metrics of “regression in model update” and accuracy gain (see Yan, FIG. 1, caption, and FIG. 2, which teaches accuracy gain between old and new models).
As to claim 6, the combination of Qi, Rebuffi, and Yan teaches the computer-implemented method of claim 1, as set forth above.
Rebuffi further teaches “wherein the hyperparameter objective function is formulated to include a parameter for an acceptable regression ratio, m, for one or more of the plurality of domains.” [§ 4, equation (1) and text below: “Edmax the baseline error (section 5), above which no points are scored.” That is, Edmax constitutes an acceptable regression ratio, since if the error (regression) is too high, no points are scored. Additionally, referring equation 1, (Edmax – Ed) corresponds to a parameter for that regression ratio for the domain d, and Edmax is a ratio because Ed is also a ratio under the definition of equation (1), in which Ed represents the percentage of errors.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of Qi and Rebuffi to have also arrived at the limitations of the instant dependent claim. Since the parts of Rebuffi discussed above are part of the techniques discussed in the rejection of the parent independent claim, the motivation for doing so is covered by the one given for the teachings of Rebuffi in the rejection of the parent independent claim.
As to claim 7, the combination of Qi, Rebuffi, and Yan teaches the computer-implemented method of claim 6, as set forth above.
Rebuffi further teaches “wherein the parameter is defined based on a regression score, and if the regression score is less that the acceptable regression ratio, m, then the regression score is set to zero otherwise the regression score is calculated based on: (i) a count of the second intersection and (ii) a total count” [As shown in Rebuffi, equation (1) and the following text, the parameter (Edmax – Ed) is defined based on a regression (error) score Ed, where the expression max{0, Edmax – Ed} indicates that the score Ed is less than the acceptable ratio Edmax, the score (note that both Edmax and the whole “max” expression corresponds to the score) is set to zero by the operation max{0, Edmax – Ed}. Otherwise, as shown in equation (1), the score Ed (and the max expression) is calculated based on the number of instances that are incorrectly predicted (summation in equation (1), as indicated by “y≠Φ(x,d)” which represents incorrect predictions) in a given trial, and the total number of instances, as represented by Ddtest. Furthermore, Edmax corresponds to a baseline error rate. Since an error rate is based on the number of incorrect predictions, an error rate is also incorrectly based on the number of correct predictions. Note that the baseline error rate is computed from the results of an actual model. See caption of Table 1: “The fully-finetuned model, written blue, is used as a baseline to compute the decathlon score.” In regards to, the limitation of “a count of the second intersection”, the negative flips already taught in Yan in the combination of are the count of the second intersection. Furthermore, it is readily understood that the total error count in the summation of Ed in Rebuffi’s equation (1) is simply a sum of the count of the second intersection (i.e., negative flips) and the errors incurred by both models (the “both wrong” section of FIG. 1 of Yan). Therefore, since Yan as applied thus far already teaches the existence of this variable, the total error count in the summation of Ed is also based on the count of the second intersection. (Examiner’s Note: Although Yan also teaches a negative flip rate, the instant claim does not define “regression” score more precisely in a manner that requires this part of Yan to be applied in order to meet the limitations of the instant claim.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far, including the above teachings of Rebuffi, to have also arrived at the claimed invention of the instant dependent claim. Since the parts of Rebuffi discussed above are part of the techniques discussed in the rejection of the parent independent claim, the motivation for doing so is covered by the one given for the teachings of Rebuffi in the rejection of the parent independent claim.
As to claims 8 and 13-14, these claims are directed to a system for performing operations that are the same or substantially the same as those of claims 1 and 6-7. Therefore, the rejections made to claims 1 and 6-7 are applied to claims 8 and 13-14, respectively.
Furthermore, Qi teaches “a system comprising: one or more processors; and one or more computer-readable media storing instructions which, when executed by the one or more processors, cause the system to perform operations” [[0007]: “The system/apparatus may comprise one or more processors and a memory coupled to the one or more processors. The memory may comprise instructions which, when executed by the one or more processors, cause the one or more processors to perform various ones of, and combinations of, the operations outlined above with regard to the method illustrative embodiment.”]
As to claims 15 and 20, these claims are directed to a computer-readable medium for performing operations that are the same or substantially the same as those of claims 1 and 7. Therefore, the rejections made to claims 1 and 7 are applied to claims 1 and 20, respectively.
Furthermore, Qi teaches “one or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause a system to perform operations” [[0006]: “a computer program product comprising a computer useable or readable medium having a computer readable program is provided. The computer readable program, when executed on a computing device, causes the computing device to perform various ones of, and combinations of, the operations outlined above with regard to the method illustrative embodiment.”]
2. Claims 2-3, 9-10, and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Qi in view of Rebuffi and Yan, and further in view of Raschka, “Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning,” arXiv:1811.12808v3 [cs.LG] 11 Nov 2020 (“Raschka”).
As to claim 2, the combination of Qi, Rebuffi, and Yan teaches the computer-implemented method of claim 1, as set forth above.
Yan further teaches “wherein the hyperparameter objective function is formulated to normalize […] a count of the second intersection over a total number of instances in the at least one evaluation dataset to obtain an improvement score and a regression score” [§ 3.1, paragraph 1: “The negative flip rate (NFR) measures the fraction of samples that are negatives…” As shown in equation (2) in this paragraph, the negative flip rate is the count of negative flips divided by N, which is defined in § 3, paragraph 1 as the total number of items in the dataset. The negative flip rate constitutes a regression score, while the accuracy gain shown in FIG. 2 constitutes an improvement score.] and suggests “wherein the domain score for each domain is calculated based on the improvement score and the regression score” [The feature of “wherein the domain score for each calculated” is already disclosed in Rebuffi, which teaches the use of a performance metric. Since Yan teaches an “improvement score” and a “regression score” as forms of performance metrics and the instant limitation of “based on” does not require any specific mathematical relationship other than the use of the scores, the instant limitation is taught by the combined teachings of the references in which the performance metric is a composite score of the improvement score and the regression score.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far, including the teachings of Yan discussed above, by using an improvement score and a regression score as taught in Yan as a performance metric, so as to arrive at the above features of the instant dependent claim. The motivation for doing so is covered by the motivation given for the teachings of Yan in the rejection of the parent independent claim.
The combination of references thus far does not explicitly teach “normalize a count of the first intersection.”
Raschka teaches “normalize a count of the first intersection” [In general, § 4.3 (page 36), paragraph 1: “The McNemar test…is a non-parametric statistical test for paired comparisons that can be applied to compare the performance of two machine learning classifiers.” § 4.3, paragraph 2: “Given such a 2x2 confusion matrix as shown in Figure 19, we can compute the accuracy of a Model 1 via (A + B)=(A + B + C + D), where A + B + C + D is the total number of test examples n. Similarly, we can compute the accuracy of Model 2 as (A + C)=n. The most interesting numbers in this table are in cells B and C, though, as A and D merely count the number of samples where both Model 1 and Model 2 made correct or wrong predictions, respectively.” That is, as shown in FIG. 19, A+B+C+D is a full set of test instances, and each of A, B, C, D is an intersection of various results of Models 1 and 2. Thus, with model 1 and model 2 being applicable and analogous to a proposed and a baseline model (for example), B would be the intersection of instances correctly predicted by the proposed model and incorrectly predicted by the baseline model (analogous to a “first intersection”), and C would be the intersection of instances incorrectly predicted by the proposed model and correctly predicted by the baseline model (analogous to a “second intersection”), or vice versa. In regards to the limitation of “normalize a count of the first intersection”, the bottom of page 36 teaches the use of the model 1 correct model 2 incorrect count (11 in subpanel A, which is quadrant B in FIG. 19) normalized by the total count 10000 together with the count of 9959 for quadrant A). This is used to measure the accuracy of model 1, which is analogous to accuracy as disclosed in Yan.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far with the teachings of Raschka by normalizing a count of the first intersection, so as to arrive at the claimed invention. The motivation would have been to implement a manner of measuring the accuracy of a model, such as a proposed candidate model, for comparison with the accuracy with the baseline mode, as suggested by Raschka (see parts quoted above).
As to claim 3, the combination of Qi, Rebuffi, Yan, and Raschka teaches the computer-implemented method of claim 2, as set forth above.
“wherein the improvement score is calculated […]” and
Yan further teaches “wherein the regression score is calculated based on (i) a count of the second intersection and (ii) the total count of instances within the at least one evaluation dataset” [§ 3.1, paragraph 1: “The negative flip rate (NFR) measures the fraction of samples that are negatives…” As shown in equation (2) in this paragraph, the negative flip rate is the count of negative flips divided by N, which is defined in § 3, paragraph 1 as the total number of items in the dataset. The negative flip rate constitutes a regression score, while the accuracy gain shown in FIG. 2 constitutes an improvement score.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far, including the above teachings of Yan, so as to have also arrived at the above features of the claimed invention. The motivation for doing so is covered by the motivation given for the teachings of Yan in the rejection of the parent independent claim.
Raschka further teaches the improvement score being calculated “based on (i) a count of the first intersection, and (ii) a total count of instances withing the at least one evaluation set” [the bottom of page 36 teaches the use of the model 1 correct model 2 incorrect count (11 in subpanel A, which is quadrant B in FIG. 19) normalized by the total count 10000 together with the count of 9959 for quadrant A). This is used to measure the accuracy of model 1, which is analogous to accuracy as disclosed in Yan.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far, including the above teachings of Raschka, so as to have also arrived at the above features of the claimed invention. The motivation for doing so is covered by the motivation given for the teachings of Raschka in the rejection of the parent dependent claim.
As to claims 9-10 and 16-17, the further limitations recited in these claims are the same or substantially the same as those of claims 2-3. Therefore, the rejections made to claims 2-3 are applied to claims 9-10 and 16-17.
3. Claims 4-5, 11-12, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Qi in view of Rebuffi and Yan, and further in view of Mindermann et al., “Prioritized Training on Points that are learnable, Worth Learning, and Not Yet Learnt,” Proceedings of the 39th International Conference on Machine Learning, Baltimore, Maryland, USA, PMLR 162, 17-23 July 2022 (“Mindermann”).
As to claim 4, the combination of Qi, Rebuffi, and Yan teaches the computer-implemented method of claim 1, but does not teach the further limitations of the instant dependent claim that the hyperparameter objective function is formulated to “exclude unstable instances, which are instances within the at least one evaluation dataset that are determined to yield prediction results that differ from one another using a same machine learning model.”
Mindermann teaches “exclude unstable instances, which are instances within the at least one evaluation dataset” [Page 4, first paragraph: “We now provide intuition on why reducible holdout loss selection (RHOLOSS) avoids redundant, noisy, and less relevant points… ii) Noisy points. While prior methods select based on high training loss (or gradient norm), not all points with high loss are informative—some may have an ambiguous or incorrect (i.e. noisy) label. The labels of such points cannot be predicted using the holdout set (Chen et al., 2019). Such points have high IL and, consequently, low reducible loss. These noisy points are less likely to be selected compared to equivalent points with less noise.” As shown in Algorithm 1 on page 3, the selection of the points in line 8 is based on the top nb, which includes “nosiy points” discussed above. Note that the noisy points as defined here are those that have an ambiguous label, and are thus unstable in the sense that they cannot be predicted in a manner that results in effective training in terms of accuracy. In regards to the context that the hyperparameter objective function is formulated for such exclusion, the Examiner notes that the context of the existing reference already teaches the formulation of the objective function in a manner that is dependent on training.] “that are determined to yield prediction results that differ from one another using a same machine learning model” [As shown in Algorithm 1 on page 3, liens 4-10 are part of a training process, with line 10 being the parameter update process. Since the model is updated over time, meaning that the result of the model varies depending on its parameters, the feature that the same point results in different predictions for the same model at different stages of the training process would merely be workable ranges of the predictions discoverable under routine experimentation. The Examiner notes that the instant claim does not define the reason that the prediction differs, nor does it define the circumstances in which the differing results are calculated. Therefore, when the teachings of this reference are applied to the existing reference, the instant limitation would have been obvious as a discovery of workable range of predictions over different training iterations in routine experimentation. See MPEP § 2144.05(II)(A), stating that where the general conditions of a claim are disclosed in the prior art, it is not inventive to discover the optimum or workable ranges by routine experimentation.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of the references combined thus far with the teachings of Mindermann by implementing the training process of Qi, as modified thus far, to use the reducible holdout loss selection technique taught in Mindermann, so as to arrive at the limitations of the instant claim. The motivation would have been to avoid points that are not learnable, so as to increase accuracy and training speed. See Mindermann, abstract: “In contrast, RHO-LOSS selects points that are learnable, worth learning, and not yet learnt. RHO-LOSS trains in far fewer steps than prior art, improves accuracy, and speeds up training on a wide range of datasets, hyperparameters, and architectures (MLPs, CNNs, and BERT).”
As to claim 5, the combination of Qi, Rebuffi, Yan, and Mindermann teaches the computer-implemented method of claim 4, as set forth above.
Mindermann further teaches “wherein excluding unstable instances comprises: (i) subtracting a count of the excluded unstable instances from the count of the first intersection, and (ii) subtracting the count of the excluded unstable instances from a count of the second intersection.” [In Algorithm 1, line 8 samples the top-nb samples, which therefore subtracts out the count of the excluded unstable samples from all measures of performance, whether they are correctly or incorrectly predicted. Therefore, this limitation flows from the combined teachings of the references.]
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of Qi, Rebuffi, Yan, and Mindermann to have also arrived at the limitations of the instant dependent claim. Since the parts of Mindermann discussed above are part of the techniques discussed in the rejection of the parent dependent claim, the motivation for doing so is covered by the one given for the teachings of Mindermann in the rejection of the parent dependent claim.
As to claims 11-12 and 18-19, the further limitations recited in these claims are the same or substantially the same as those of claims 4-5. Therefore, the rejections made to claims 4-5 are applied to claims 11-12 and 18-19.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. The following documents depict the state of the art.
Xie et al., “Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates,” Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6589–6602 August 1–6, 2021 teaches additional techniques based on the negative flip rate taught in Yan.
Kuncheva et al., “Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy,” Machine Learning, 51, 181–207, 2003 teaches a 2x2 confusion/disagreement matrix between two models.
Jordan et al., “Using Cumulative Distribution Based Performance Analysis to Benchmark Models,” 32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada teaches the use of task-specific benchmarks, similar to domain-specific scores of the instant application.
US 20220343212 A1 teaches the use of a negative flip rate.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YAO DAVID HUANG whose telephone number is (571)270-1764. The examiner can normally be reached Monday - Friday 9:00 am - 5:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached at (571) 270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Y.D.H./Examiner, Art Unit 2124
/MIRANDA M HUANG/Supervisory Patent Examiner, Art Unit 2124
