DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 7-8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Megha Srivastava et al. (“An Empirical Analysis of Backward Compatibility in Machine Learning Systems”, 2020-08-11) in view of Sung-Hyuk Cha et al. (hereinafter Cha) (“Enhancing Binary Feature Vector Similarity Measures”, 2005-1-1) further in view of Gagan Bansal et al. (hereinafter Bansal) (“Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff,” 2019-07-17).
Regarding claim 1, Megha teaches;
A compatibility evaluation ([pg. 3] We define two measures of backward compatibility, which we use in our experiments) device comprising: a memory configured to store instructions; and one or more processors configured to execute the instructions to: (NOTE: Megha teaches computer implemented processes) acquire, from a plurality of prediction models ([pg. 3] Each model h is a function h : x→y … h1 predicted … h2 … predicted) comprising a first predictor and a second predictor ([pg. 3] h1 and h2), evaluation data comprising an output of the first predictor and an output of the second predictor ([pg. 3] h1(xi) … h2(xi)); … a plurality of relational expressions representing a relationship between the output of the first predictor and the output of the second predictor,
([pg. 3]
PNG
media_image1.png
104
454
media_image1.png
Greyscale
PNG
media_image2.png
102
456
media_image2.png
Greyscale
)
… calculate, based on the output of the first predictor, the output of the second predictor, and the evaluation index, a score indicating a compatibility of the first predictor and the second predictor ([pg. 3] Backward Trust Compatibility (BTC) score, which is the ratio of points in a held out test set (e.g., Dtest) that h2 predicted correctly among all points h1 had already predicted correctly ... Backward Error Compatibility (BEC) score, which is the proportion of points in a held-out test set that h2 predicted incorrectly, out of which h1 also predicted incorrectly); … wherein the evaluation index is configured to indicate: a Backward Trust Compatibility (BTC) score ([pg. 3] Backward Trust Compatibility (BTC) score), a Backward Error Compatibility (BEC) score ([pg. 3] Backward Error Compatibility (BEC) score), …
Megha fails to explicitly teach but Cha teaches;
acquire a generalized ([pg. 8] Sweighted-00-11) comprising: … a plurality of relational expressions representing a relationship between [x] … and [y] … and a weight for each of the plurality of relational expressions ([pg. 8] Sweighted-00-11(x, y) = … w⨁ixiyi … w⦵ix̄iȳi); receive a designation of a compatibility index ([pg. 8] Sweighted-00-11 = Sweighted-hamming … Sweighted-00-11 = Sweighted-inner-product); set the weight for each of the plurality of relational expressions based on the designation ([pg. 8] if w⨁ and w⦵ are identical, Sweighted-00-11 = Sweighted-hamming and if w⦵ = 0, Sweighted-00-11 = Sweighted-inner-product); determine an evaluation index based on the [generalized] ([pg. 8] if w⨁ and w⦵ are identical, Sweighted-00-11 = Sweighted-hamming and if w⦵ = 0, Sweighted-00-11 = Sweighted-inner-product … weight adaptive model [Sweighted-00-11] … depends on a set of weights); … calculate, based on [x], [y], and the evaluation index ([pg. 8] Sweighted-00-11(x, y) = … w⨁ixiyi … w⦵ix̄iȳi), a score (NOTE: the similarity measure output from Sweighted-00-11(x, y)) … and a compatibility score other than the BTC score and the BEC score ([pg. 7] Sweighted-hamming … Sweighted-inner-product, etc.).
OBVIOUSNESS TO COMBINE CHA:
Cha is analogous art to the present disclosure as it pertains to generalized indexes comprising a plurality of weighted relational expressions.
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the compatibility evaluation technique of Megha according to the weighted similarity technique of Cha. Megha teaches evaluating backward compatibility between outputs of a first prediction model and a second prediction model, including BTC and BEC scores based on relationships between the predictions of the respective models. Cha teaches a generalized, weight adaptive similarity measure for evaluating relationships between two vectors, wherein respective relational expressions are assigned weights, and the weights may be selected or determined to configure the generalized measure to provide different similarity measures. For example, Cha teaches that particular selections of weights cause its generalized weighted similarity measure to correspond to weighted Hamming similarity or weighted inner product similarity. One of ordinary skill in the art would have been motivated to apply Cha’s generalized weighted similarity technique to Megha’s comparison of predictor outputs to provide a more flexible compatibility evaluation index capable of selectively emphasizing different types of agreements and disagreements between the outputs and capable of providing different compatibility / similarity indices depending on the appropriate evaluation. Such a modification would have amounted to applying Cha’s known technique for weighted comparison of corresponding outputs to Megha’s known comparison of outputs from different prediction models, with the predictable result of permitting Megha’s model compatibility to be evaluated according to selectable weighted relationships and different resulting evaluation indices.
Megha and Cha fail to explicitly teach but Bansal teaches;
and train the first predictor ([pg. 2431] To train classifiers, … optimize for the predictive performance of h2 by specifying, and minimizing, a classification loss L) based on a regularization term that includes the
[pg. 2431]
PNG
media_image3.png
84
682
media_image3.png
Greyscale
([pg. 2432] Dissonance D … Recall that C(h1, h2) is high when both h1 and h2 are correct (Equation 1). Dissonance expresses the opposite notion … Equation 3 defines the new loss. Lc = L+λc·D)
OBVIOUSNESS TO COMBINE BANSAL:
Bansal is analogous art to the present disclosure as it pertains to training machine learning models based on compatibility.
Megha teaches evaluating backward compatibility between different versions of a prediction model, while Cha teaches a generalized weighted similarity measure capable of representing different particular compatibility similarity indexes depending on the selected weights. Bansal similarly addresses compatibility between an existing predictor and an updated predictor and teaches improving such compatibility during training by augmenting the ordinary classification loss with a compatibility-based regularization term that penalizes incompatibility between the predictors ([Bansal, Abstract] We propose a re-training objective to improve the compatibility of an update by penalizing new errors. The objective offers full leverage of the performance/compatibility tradeoff across different datasets, enabling more compatible yet accurate updates).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to incorporate the generalized compatibility index of the Megha-Cha system into Bansal’s compatibility based training objective because doing so would permit the same compatibility criterion used to evaluate the relationship between predictors to also guide training of the updated predictor, thereby reducing incompatible prediction changes while maintaining predictive performance. Such a modification would have amounted to applying Bansal’s known technique of compatibility aware regularization to the generalized compatibility measure of Mecha as modified by Cha, with the predictable result of training the updated predictor to exhibit improved compatibility according to the selected compatibility index.
Regarding claim 2, Megha teaches;
(backward compatibility) BC index ([pg. 3] BTC … BEC)
Megha fails to teach but Cha teaches;
the [generalized] index further comprises four arithmetic operations
[pg. 8]
PNG
media_image4.png
169
931
media_image4.png
Greyscale
NOTE: Cha’s generalized index is represented by at least four arithmetic operations performed on the plurality of weighted relational expressions.
of the plurality of weighted relational expressions. ([pg. 8] w⨁ixiyi … w⦵ix̄iȳi)
OBVIOUSNESS:
Using the same reasoning from claim 1.
Regarding claim 7,
Claim 7 is a method claim that is substantially similar to claim 1 and is rejected using the same reasoning.
Regarding claim 8,
Claim 8 is a non-transitory computer-readable recording medium claim that is substantially similar to claim 1 and is rejected using the same reasoning.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Megha (“An Empirical Analysis of Backward Compatibility in Machine Learning Systems”, 2020-08-11) in view of Cha (“Enhancing Binary Feature Vector Similarity Measures”, 2005-1-1), further in view of Bansal (“Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff,” 2019-07-17) as applied to claim 1, further in view of Tom Dietterich (hereinafter Dietterich) (“Evaluation of Classifiers”, 2005).
Regarding claim 4, Megha, Cha, and Bansal fail to explicitly teach but Dietterich teaches;
the plurality of relational expressions includes: a first equation indicating a percentage that the output of the first predictor and the output of the second predictor are both correct ([pg. 22] n00: the number of examples correctly classified by both classifiers … [pg. 24] pij = nij/n); a second equation indicating a percentage that the output of the first predictor and the output of the second predictor are both incorrect ([pg. 22] n00: the number of examples misclassified by both h1 and h2 [NOTE: n00 is supposed to say n11 here] … [pg. 24] pij = nij/n); a third equation indicating a percentage that the output of the first predictor is incorrect and the output of the second predictor is correct ([pg. 22] n10: the number of examples misclassified by h1, but correctly classified by h2 … [pg. 24] pij = nij/n); and a fourth equation indicating a percentage that the output of the first predictor is correct and the output of the second predictor is incorrect ([pg. 22] n01: the number of examples correctly classified by h1, but misclassified by h2 … [pg. 24] pij = nij/n).
OBVIOUSNESS TO COMBINE DIETTERICH:
Dietterich is analogous art to the present disclosure as it pertains to comparing prediction model outputs using a plurality of relational expressions.
Megha teaches evaluating compatibility between two prediction models based on relationships between their respective prediction outcomes, while Cha teaches a generalized weighted similarity framework for combining such relationships into a configurable compatibility / evaluation index. Dietterich teaches comparing two prediction models by expressly categorizing their joint prediction outcomes into four cases (both correct, both incorrect, first correct and second incorrect, first incorrect and second correct) and further converting the counts for those cases into corresponding probabilities. One of ordinary skill in the art would have been motivated to use Dietterich’s four outcomes in the Megha-Cha compatibility framework because it provides a conventional and systematic way to quantify all possible agreement and disagreement relationships between two classifiers, thereby enabling the generalized weighted compatibility measure to evaluate the relative frequency of each type of joint prediction outcome. Such a modification would have predictably provided a more complete characterization of compatibility between the two predictors while using known classifier comparison techniques for their established purpose.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Megha (“An Empirical Analysis of Backward Compatibility in Machine Learning Systems”, 2020-08-11) in view of Cha (“Enhancing Binary Feature Vector Similarity Measures”, 2005-1-1), further in view of Bansal (“Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff,” 2019-07-17), further in view of Dietterich (“Evaluation of Classifiers”, 2005) as applied to claim 4 above, further in view of Jinbo Bi et al. (hereinafter Bi) (“Regression Error Characteristic Curves”, 2003).
Regarding claim 5, Megha teaches;
output of the first predictor and the second predictor ([pg. 3] h1(xi) … h2(xi))
Megha, Cha, Bansal, and Dietterich fail to teach but Bi teaches;
the first predictor and the second predictor configured to perform a regression analysis ([pg. 43] assess the relative merits of many regression functions), … regard the output of the first predictor and the second predictor as correct ([pg. 45] residuals must be greater than a tolerance e before they are considered as errors) based on a difference between an expected value and an actual value ([pg. 43] difference between the predicted value f(x) and actual value y … [pg. 45] In regression … the residual y – f(x)) corresponding to the expected value being equal to or smaller than a predetermined threshold value ([pg. 45] residuals must be greater than a tolerance e before they are considered as errors [also see equation acc(e) on pg. 45]), the expected value being the output of the first predictor and the second predictor ([pg. 43] assess the relative merits of many regression functions … predicted value f(x)), and regard the output of the first predictor and the second predictor as incorrect based on the difference being larger than the predetermined threshold value ([pg. 45] residuals must be greater than a tolerance e before they are considered as errors)
OBVIOUSNESS TO COMBINE BI:
Bi is analogous art to the present disclosure as it provides a regression model evaluation technique.
Megha teaches evaluating compatibility between prediction models based on whether their respective predictions are correct or incorrect, and Dietterich further teaches characterizing the joint prediction outcomes of two predictors according to the four possible correct / incorrect relationships. Bi teaches applying an analogous evaluation methodology to regression models by determining the difference between a predicted value and its corresponding actual value and using a predetermined error tolerance to distinguish correct / accurate predictions from errors. One of ordinary skill in the art would have been motivated to apply Bi’s regression-error tolerance technique to the compatibility framework of Megha as modified by Cha, Bansal, and Dietterich to extend the compatibility evaluation to regression predictors, for which prediction outputs are continuous values. Using Bi’s known regression error-tolerance technique would predictably allow the resulting correct / incorrect regression analysis outcomes to be evaluated using the same joint relational expressions and generalized compatibility measures of the combined system. Such a modification would have amounted to applying Bi’s known regression-evaluation technique according to its established function, with the predictable result of enabling backward-compatibility evaluation of regression predictors.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Megha (“An Empirical Analysis of Backward Compatibility in Machine Learning Systems”, 2020-08-11) in view of Cha (“Enhancing Binary Feature Vector Similarity Measures”, 2005-1-1), further in view of Bansal (“Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff,” 2019-07-17) as applied to claim 1, further in view of Julián Urbano et al. (hereinafter Urbano) (“The Treatment of Ties in AP Correlation”, 2017).
Regarding claim 6, Megha teaches;
outputs of the first predictor for two evaluation data … outputs of the second predictor for two evaluation data ([pg. 3] h1(xi) … h2(xi) [i = 1, i = 2, etc.]) … one or more processors (Using the same reasoning from claim 1)
Megha, Cha, and Bansal fail to teach but Urbano teaches teaches:
the relational expressions ([pg. 2, equation 2] cji = 1 [if] sign(xj − xi) = sign(yj − yi) [otherwise] 0) indicate a magnitude relationship of the [first] outputs … for two evaluation data ([pg. 2] X = <A,B,C,D,E,F> … sign(xj − xi)), and a magnitude relationship of the [second] outputs … for the two evaluation data ([pg. 2] Y = <C,A,B,D,F,E> … sign(yj − yi)) … calculate, as the score, the expected value with which the magnitude relationship of the [first] outputs … and the magnitude relationship of the [second] outputs … match ([pg. 2] a pair is concordant if their relative order is the same in both rankings … cij equals 1 if items i and j are concordant … The fraction of concordant pairs can be interpreted as the expected value)
OBVIOUSNESS TO COMBINE URBANO:
Urbano is analogous art to the present disclosure as it pertains to evaluating the relationship between corresponding outputs from two systems by quantifying how consistently the outputs agree.
Megha teaches evaluating backward compatibility between outputs of different prediction models, while Cha provides a generalized framework for evaluating relationships between such outputs. Urbano teaches a known technique for comparing two sets of values according to whether the relative order, or magnitude relationship, between pairs of values is preserved and further teaches quantifying such agreement based on the fraction of concordant pairs, which is interpretable as an expected value or probability of concordance. It would have been obvious to one of ordinary skill in the art, before the effective filing date, to apply Urbano’s pairwise concordance technique to the outputs of the predictors of the Megha-Cha-Bansal system in order to evaluate an additional aspect of compatibility between the predictors: whether the updated predictor preserves the relative ordering of outputs produced by the other predictor for corresponding evaluation data. Preserving such relative relationships would provide useful compatibility information beyond determining if predictions are correct or not, particularly where the relative magnitude or ranking of predictor outputs is relevant to downstream use. Such a modification would have amounted to applying Urbano’s known correlation technique to the corresponding outputs of Megha’s prediction models according to its established function, with the predictable benefit of providing a compatibility score representing the expected degree to which pairwise magnitude relationships of the two predictors agree.
Response to Arguments
The drawings have been reviewed and accepted in the present Office Action.
Applicant’s arguments, starting page 2, filed 06/25/2026, with respect to 35 U.S.C 112 have been fully considered and are persuasive. The treatment of claim(s) 1, 5, and 6 under 112(f) has been withdrawn in the present Office Action. Accordingly, the 112(b) rejections of claims 1, 5, and 6 have also been withdrawn.
Applicant’s arguments, starting page 3, filed 06/25/2026, with respect to 35 U.S.C. 101 have been fully considered and are persuasive. The abstract idea rejections of amended claim 1 and associated dependent claims have been withdrawn.
Applicant's arguments filed 06/25/2026, with respect to 35 U.S.C. 102 and 103, have been fully considered but they are not persuasive. The applicant states; “Applicant respectfully submits that none of the references cited in the Office Action, whether considered alone or in combination, disclose, teach, or suggest all elements of claim 1 including at least "train the first predictor based on a regularization term that includes the GBC index... wherein the evaluation index is configured to indicate: a Backward Trust Compatibility (BTC) score, a Backward Error Compatibility (BEC) score, and a score other than the BTC score and the BEC score" … As the Examiner tentatively agreed during the interview discussed above, in the instant case, none of the cited references teach or suggest all elements of claim 1, including at least "train the first predictor based on a regularization term that includes the GBC index... wherein the evaluation index is configured to indicate: a Backward Trust Compatibility (BTC) score, a Backward Error Compatibility (BEC) score, and a score other than the BTC score and the BEC score" … The Office applies the same cited reference(s) using the same analysis in rejecting claim(s) 7 and 8 as in rejecting claim 1. Therefore, claim(s) 7 and 8 is/are also patentable over the cited art for reasons similar to claim 1 discussed above… Applicant respectfully requests that the rejections be withdrawn because each of the remaining dependent claims depends from one of allowable independent claim(s) 1, 7, and 8, and because the cited art, either alone or in combination, does not cure the above-described deficiencies of Megha, Cha, Dietterich, Bi, or Urbano.” Examiner respectfully disagrees. Amended claims 1, 7, and 8 are now being rejected under 35 U.S.C. 103 by Megha in view of Cha, further in view of Bansal, as necessitated by the claim amendments. As reflected by the present Office Action, Megha teaches a BTC and BEC compatibility score, Cha teaches scores other than the BTC and BEC score, and the combination of Megha and Cha teaches a GBC index (see reasoning provided in claim 1). Bansal teaches training a predictor using a compatibility-based regularization term that penalizes incompatibility between predictors, and the combination of Megha, Cha, and Bansal teaches training a predictor based on a regularization term that includes the GBC index (see reasoning provided in claim 1).
Accordingly, the 35 U.S.C. 103 rejections of claims 1, 7, 8, and the remaining dependent claims stand.
CONCLUSION
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Matthew Alan Cady whose telephone number is (571) 272-7229. The examiner can normally be reached Monday - Friday, 7:30 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached on (571)272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC)
at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW ALAN CADY/ Examiner, Art Unit 2145
/CESAR B PAULA/ Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights