CTNF 18/578,406 CTNF 79889 Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Rejections - 35 USC § 112 07-30-01 AIA The following is a quotation of the first paragraph of 35 U.S.C. 112(a): (a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention. The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112: The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention. 07-31-01 Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, at the time the application was filed, had possession of the claimed invention. Claims 1, 9, and 16 each recite the confidence score is a score indicating how sure it is that a machine learning production model will provide a correct prediction of an input at inference . The specification does not describe a confidence score having this property in a manner that conveys possession of the claimed subject matter. The only disclosed computation of the confidence score is set forth at paragraph [0115] (and recited in dependent claims 3, 11, and 18) as Confidence Score = 1 − |y − p̂(x)|, where y is the ground truth value of a data instance and p̂(x) is the predicted score of the trained machine learning model for that data instance (see also the worked examples at paragraphs [0116]–[0118]). This quantity measures the prediction error of the trained machine learning model itself on an evaluation data instance for which the ground truth value is already known. It is computed solely from the trained machine learning model's output and the ground truth value, and no machine learning production model participates in its computation. The "machine learning production model" recited in the claim is, per the specification, a separate model — one of a plurality of segment-specific production models (e.g., first, second, and third machine learning production models 504, 506, 508 in FIG. 5; paragraphs [0111]–[0113], [0124], [0130]–[0134]) — that is distinct from the trained machine learning model and the final machine learning model (the confidence engine) that generate the confidence score. The specification does not describe how a confidence score derived exclusively from the trained machine learning model's own accuracy "indicat[es] how sure it is that a machine learning production model will provide a correct prediction of an input at inference." Although the specification repeats this functional phrase verbatim (e.g., paragraphs [0008], [0057], [0081], and [0107]), it nowhere describes the relationship, correlation, or algorithm by which the disclosed confidence-score computation measures or predicts the correctness of any machine learning production model. A mere restatement of a desired result or function, without a description of how that result is achieved, does not satisfy the written description requirement. See MPEP § 2163; Ariad Pharmaceuticals, Inc. v. Eli Lilly & Co., 598 F.3d 1336 (Fed. Cir. 2010) (en banc); see also Vasudevan Software, Inc. v. MicroStrategy, Inc., 782 F.3d 671 (Fed. Cir. 2015) (for a computer-implemented functional limitation, the specification must disclose the algorithm by which the claimed function is performed). Because the disclosed algorithm computes a quantity (the trained machine learning model's own prediction error) different from the property recited in the claim (an indication of a machine learning production model's correctness), the specification fails to reasonably convey that the inventor had possession of the claimed confidence score at the time the application was filed. As a separate written description concern, independent claims 1, 9, and 16 each recite augment[ing] the evaluation dataset based on the confidence score ... to generate an augmented evaluation dataset in functional, open-ended terms. The specification discloses only a single augmentation technique — replacing the ground truth value of each data instance with that instance's confidence score (paragraphs [0100] and [0119]; recited in dependent claims 4, 12, and 19). The specification does not describe any other manner of augmenting the evaluation dataset "based on the confidence score," nor any common principle that would convey possession of the broader, generically claimed genus of augmentation. To the extent the independent claims are construed to encompass augmentation techniques beyond the single disclosed species, they lack adequate written description support for that broader scope. See MPEP § 2163; Ariad, 598 F.3d at 1349-51. Claims 2-8 depend (directly or indirectly) from claim 1, claims 10-15 depend from claim 9, and claims 17-20 depend from claim 16. Each dependent claim incorporates the deficient "confidence score" limitation of its respective independent claim and does not cure the deficiency identified above. Dependent claims 3, 11, and 18 affirmatively recite the disclosed computation (Confidence Score = 1 − |y − p̂(x)|), which confirms — rather than cures — the disconnect between the disclosed computation and the claimed property, because that computation is a function of the trained machine learning model's output and the ground truth value only. Dependent claims 5-8, 13-15, and 20 add the downstream selection of a machine learning production model based on the confidence score but do not describe how the confidence score indicates the correctness of any production model. Accordingly, claims 2-8, 10-15, and 17-20 are rejected for the same reasons as their respective independent claims. To overcome this rejection, Applicant may consider amending the claims to recite the confidence score in terms commensurate with the disclosed computation (e.g., as a measure of the trained machine learning model's prediction accuracy for a data instance), and/or directing the Examiner to specific disclosure conveying how the confidence score indicates a machine learning production model's correctness, if such support exists in the application as filed. No new matter may be added. See 35 U.S.C. 132; MPEP § 2163.06. Claim Rejections - 35 USC § 112 07-30-02 AIA The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. 07-34-01 Claim 8 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 8 recites "a first machine learning production model associated with a high confidence score, a second machine learning production model associated with a medium confidence score, and a third machine learning production model associated with a low confidence score," and further requires determining "whether the confidence score of the input corresponds to the high confidence score, the medium confidence score, or the low confidence score" in order to select among the three production models. The terms "high confidence score," "medium confidence score," and "low confidence score" are relative terms that define the boundary between the tiers. The claim's operative selecting step requires assigning an input's confidence score to one of these three tiers; to perform this determination with reasonable certainty, a PHOSITA must know what values qualify as "high," "medium," or "low." The specification at paragraph [0126] provides non-limiting example threshold values for the three tiers (high: scores exceeding 0.5 up to 1.0; medium: scores exceeding 0.3 up to 0.5; low: scores exceeding 0.0 up to 0.3), but explicitly disclaims these values as non-limiting: "It should be appreciated that any conceivable grouping scheme based on a plurality of threshold values is contemplated by the disclosed subject matter and the grouping scheme is not limited to high, medium and, low groupings of confidence scores and may use any number and value of threshold values." Because the specification expressly does not establish any fixed standard for what constitutes a "high," "medium," or "low" confidence score, the metes and bounds of the claim cannot be determined with reasonable certainty. See Nautilus, Inc. v. Biosig Instruments, Inc., 572 U.S. 898, 901 (2014); Halliburton Energy Servs., Inc. v. M-I LLC, 514 F.3d 1244, 1255 (Fed. Cir. 2008); see also MPEP § 2173.05(b). For purposes of examination, "high confidence score" is interpreted under BRI to encompass confidence scores in the upper portion of the valid confidence score range (toward 1.0 on the [0,1] scale), "medium confidence score" to encompass confidence scores in an intermediate portion, and "low confidence score" to encompass confidence scores in the lower portion (toward 0.0). Appropriate correction is required. Claim Rejections - 35 USC § 102 07-07-aia AIA 07-07 The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action: A person shall be entitled to a patent unless – 07-08-aia AIA (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention. 07-15 AIA Claims 1, 2, 4, 9, 10, 12, 16, 17 and 19 are rej ected under 35 U.S.C. 102(a)( 1) as being ant icipated by Addressing Failure Prediction by L ea rning Model Confidence to Corbière et al., (hereinafter Corbière). Per claim 1, Corbière discloses A system (Corbière: Abstract and Section 2.2… Corbière describes a complete machine learning confidence system comprising a classification network M and a confidence network ConfidNet that share a common encoder, which constitutes the system under BRI, “We introduce a confidence neural network, termed ConfidNet, with parameters θ, which outputs a confidence prediction ĉ(x, θ)”) comprising: While not explicitly stated, Corbière intrinsically discloses at least one processor (Corbière: Section 3.1…Corbière trains and evaluates deep neural networks (VGG-16, SegNet) and releases executable code implementing the approach; executable deep-learning code is required to be executed on at least one processor programmed or configured to perform the disclosed training, evaluating, augmenting and retraining operations, “Our code is available at https://github.com/valeoai/ConfidNet”) programmed or configured to: . train a base machine learning model using a training dataset to generate a trained machine learning model (Corbière: Section 2.2 and Equations (1),(4)…the classification network M, e.g., the base model of Corbière’s confidence system, is first trained on the training dataset D = {(xᵢ, y*ᵢ)} using cross-entropy loss L CE , yielding a trained model with learned parameters w, “ConfidNet builds upon a classification neural network M, whose parameters w are preliminary learned using cross-entropy loss L CE in (1)”), wherein the base machine learning model is configured to provide a confidence score (Corbière: Section 2…the trained network computes a softmax predictive distribution from which the model’s confidence score is directly provided, “given an input x, the network assigns a probabilistic predictive distribution P(Y|w, x) by computing the softmax output for each class k and where w are the parameters of the network”), wherein the confidence score is a score indicating how sure it is that a machine learning production model will provide a correct prediction of an input at inference (Corbière: Section 2.1…Corbière’s confidence criterion is a quantitative per-input measure of how certain the deployed classification model — the production model whose failures are being predicted at inference — is to have predicted correctly, which are confidence-score semantics under BRI, “A confidence criterion is a quantitative measure to estimate the confidence of the model prediction. The higher the value, the more certain the model about its prediction”; Section 1…“This paper addresses the challenge of failure prediction with deep neural networks…The objective is to provide confidence measures for model’s predictions that are reliable”); evaluate the trained machine learning model using an evaluation dataset (Corbière: Section 3.3 and Table 3…Corbière applies the trained model to a hold-out validation dataset, e.g., an evaluation dataset under BRI, when learning TCP confidence, “we also experimented with training ConfidNet on a hold-out dataset. We report results on all datasets in Table 3 for validation sets with 10% of samples”), wherein when evaluating the trained machine learning model using the evaluation dataset, the at least one processor is programmed or configured to: generate a confidence score for each data instance of the evaluation dataset with the trained machine learning model (Corbière: Section 2.2…for every data instance (xᵢ, y*ᵢ) of the dataset, the trained model’s softmax output is used to compute the per-instance target confidence value c* = TCP, i.e., a confidence score is generated for each data instance with the trained model, “we propose to learn TCP confidence c*(x, y*) = P(Y = y*|w, x), our target confidence value”); augment the evaluation dataset based on the confidence score for each data instance of the evaluation dataset to generate an augmented evaluation dataset (Corbière: Section 2.2 and Equation (4) as instantiated on the hold-out evaluation dataset in Section 3.3 and Table 3…the per-instance TCP confidence values are attached to the dataset as new per-instance regression targets, producing a modified dataset in which every data instance is paired with its confidence score, e.g.,. an augmented evaluation dataset under BRI; in the hold-out instantiation of Section 3.3, the ConfidNet “training samples” are the instances of the hold-out validation (evaluation) dataset, so the dataset augmented with confidence-score targets is the evaluation dataset itself, “During training, we seek θ such that ĉ(x, θ) is close to c*(x, y*) on training samples”; Section 2.2…“Since we want to regress a score between 0 and 1, we use the ℓ 2 loss to train ConfidNet”; Section 3.3…“we also experimented with training ConfidNet on a hold-out dataset. We report results on all datasets in Table 3 for validation sets with 10% of samples”); and retrain the trained machine learning model using the augmented evaluation dataset to generate a final machine learning model (Corbière: Section 2.2…the complete confidence model shares its ConvNet encoder with the trained classification model M, and in the fine-tuning step the trained model’s learned encoder parameters are further trained on the confidence-target (augmented) dataset; although Corbière duplicates the encoder so that M’s classification outputs remain fixed, the duplicated encoder is the trained model’s own learned encoder, and its further training on the augmented dataset generates the final confidence model, thus retrain the trained machine learning model … to generate a final machine learning model reads on further training the trained model’s learned parameters, whether in place or in a decoupled copy that becomes the final model, “Our complete confidence model, from input image to confidence score, shares its first encoding part (‘ConvNet’ in Fig.2) with the classification model M”; Section 2.2….“The training of ConfidNet starts by fixing entirely M (freezing w) and learning θ using loss (4). In a next step, we can then fine-tune the ConvNet encoder”; Section 3.3 and Table 2…the retraining yields the improved final model, “By allowing subsequent fine-tuning as described in section 2.2, ConfidNet performance is further boosted in every setting”). Per claim 2, Corbière discloses claim 1, further disclosing wherein each data instance of the evaluation dataset comprises at least one feature value and a ground truth value (Corbière: Section 2…each sample of the dataset pairs a d-dimensional feature vector with its true class, e.g., a ground truth value, “xᵢ ∈ Rᵈ is a d-dimensional feature and y*ᵢ ∈ Y = {1, ..., K} is its true class”) , when generating the confidence score for each data instance of the evaluation dataset with the trained machine learning model, the at least one processor is programmed or configured to: determine a score of the trained machine learning model for each data instance of the evaluation dataset (Corbière: Section 2…the trained model outputs a softmax probability score for each data instance, “given an input x, the network assigns a probabilistic predictive distribution P(Y|w, x) by computing the softmax output for each class k”) and calculate the confidence score for each data instance of the evaluation dataset based on the score of the trained machine learning model for each data instance of the evaluation dataset and the ground truth value for each data instance of the evaluation dataset (Corbière: Section 2.1 and Equation (2)…the TCP confidence score of each instance is calculated from the model’s score evaluated at the instance’s ground-truth class, i.e., from both the model score and the ground truth value, “we propose to consider the True Class Probability as a suitable confidence criterion for failure prediction”). Per claim 4, Corbière discloses claim 1, further disclosing wherein each data instance of the evaluation dataset comprises at least one feature value and a ground truth value (Corbière: Section 2…each data instance is a feature/ground-truth pair, “xᵢ ∈ Rᵈ is a d-dimensional feature and y*ᵢ ∈ Y = {1, ..., K} is its true class”) , and wherein, when augmenting the evaluation dataset, the at least one processor is programed or configured to: replace the ground truth value for each data instance of the evaluation dataset with the confidence score for each data instance in the evaluation dataset to provide the augmented evaluation dataset (Corbière: Section 2.2 and Equation (4)…in the confidence-learning dataset the supervision target of every instance xᵢ is no longer the ground-truth class y*ᵢ but the TCP confidence value c*(xᵢ, y*ᵢ); substituting the per-instance target value in this manner replaces the ground truth value with the confidence score under BRI, “During training, we seek θ such that ĉ(x, θ) is close to c*(x, y*) on training samples”; Section 3.3…the confidence value serves as the per-sample supervisory signal in place of the class label, “TCP regularizes training by providing more fine-grained information about the quality of the classifier regarding a sample’s prediction”). Claims 9, 10 and 12 are substantially similar in scope and spirit as claims 1, 2 and 4, reciting the same operations as a computer-implemented method performed with at least one processor. Therefore the rejections of claims 1, 2 and 4 are applied accordingly. Per claims 16, 17 and 19, Corbière discloses the operations of claim 1 in the form of A computer program product . While not explicitly stated, Corbière intrinsically discloses at least one non-transitory computer readable medium including one or more instructions that, when executed by at least one processor, cause the at least one processor to perform the recited operations (Corbière: Section 3.1…Corbière’s publicly released executable code implementing the disclosed training, evaluating, augmenting and retraining operations is necessarily stored as instructions on a non-transitory computer readable medium and executed by at least one processor — distributed software cannot exist or run otherwise, “Our code is available at https://github.com/valeoai/ConfidNet”). Claims 16, 17 and 19 are otherwise substantially similar in scope and spirit as claims 1, 2 and 4. Therefore the rejections of claims 1, 2 and 4 are applied accordingly . Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-23-aia AIA The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. 07-20-02-aia AIA This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention. 07-21-aia AIA Claims 3, 11 and 18 are re jected under 35 USC 103 as being unpatentable over Co rbière. Pe r claim 3, Corbière discloses claim 2, including calculating the TCP confidence score for each data instance from the trained model’s score and the instance’s ground truth value (Corbière: Section 2.1 and Equation (2)…“TCP : (x , y*) → P(Y = y*|w, x)”). Corbière does not expressly recite, in haec verba: calculate an absolute value of a difference of the ground truth value and the score of the trained machine learning model; and subtract the absolute value of a difference of the ground truth value and the score of the trained machine learning model from 1 to provide the confidence score (Corbière: Section 2.1, Equation (2) and Section 2…Corbière’s True Class Probability criterion, evaluated for a two-class problem (K = 2) with ground truth value y ∈ {0, 1} and predicted score p̂(x) = P(Y = 1|w, x), equals p̂(x) when y = 1 and 1 − p̂(x) when y = 0, which is identically 1 − |y − p̂(x)| — i.e., the absolute value of the difference of the ground truth value and the model score, subtracted from 1; Corbière’s formulation expressly encompasses the two-class case, “we propose to consider the True Class Probability as a suitable confidence criterion for failure prediction”; “y*ᵢ ∈ Y = {1, ..., K} is its true class”). Corbière is analogous art to the claimed invention because it is from the same field of endeavor, namely estimating and providing confidence scores indicating whether a machine learning model’s prediction of an input will be correct (Corbière: Abstract… “Assessing reliably the confidence of a deep neural network and predicting its failures is of primary importance for the practical deployment of these models”; Section 2.1…“A confidence criterion is a quantitative measure to estimate the confidence of the model prediction”). Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to compute Corbière’s TCP confidence criterion for a two-class classification task directly as one minus the absolute value of the difference between the ground truth value and the model’s predicted score, because the two expressions are mathematically identical for K = 2 and the use of an algebraically equivalent formulation of a disclosed criterion is the application of a known technique to a known method ready for improvement, yielding entirely predictable results (KSR rationale (D); MPEP 2143(I)). The suggestion/motivation for doing so would have been Corbière’s express teaching that TCP is “a suitable confidence criterion for failure prediction” (Corbière: Section 2.1) together with its generic K-class formulation (Corbière: Section 2, “y*ᵢ ∈ Y = {1, ..., K}”), which directs the skilled artisan to evaluate the same criterion when the production task is binary — where it necessarily takes the claimed arithmetic form. Claims 11 and 18 are substantially similar in scope and spirit as claim 3. Therefore the rejection of claim 3 is applied accordingly . 07-21-aia AIA Claims 5-8, 13-15 and 20 are rejecte d under 35 USC 103 as being unpatentable over Corbièr e in view of Adaptive Neural Networks for Efficient Inference to Bolukbasi et al. (hereinafter Bolukbasi). Per clai m 5, Corbière discloses claim 1, further disclosing receive an input (Corbière: Section 2…the trained network receives each input to be scored at inference, “given an input x, the network assigns a probabilistic predictive distribution P(Y|w, x) by computing the softmax output for each class k”) and determine a confidence score for the input (Corbière: Section 2.2…ConfidNet outputs a confidence score for each received input, “We introduce a confidence neural network, termed ConfidNet, with parameters θ, which outputs a confidence prediction ĉ(x, θ)”). Corbière does not expressly disclose, but with Bolukbasi does teach: select a machine learning production model of a plurality of machine learning production models based on the confidence score for the input (Bolukbasi: Fig. 2 and Section 4…Bolukbasi’s network selection policy uses the confidence feedback obtained for the input to select which of a plurality of deployed pre-trained production networks (AlexNet, GoogLeNet, ResNet) processes the input, “The policy evaluates Alexnet, receives confidence feedback and decides to jump directly to Resnet or send the sample to GoogLeNet->Resnet cascade”; Section 1…“We extend this to learn a network selection system that adaptively selects the network to be evaluated for each example”) Corbière and Bolukbasi are analogous art because they are from the same field of endeavor, specifically deep learning and confidence-based control of neural network inference. They address the same problem solving area of using a model-confidence signal to decide how an input should be handled at inference. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to use the per-input confidence score produced by Corbière’s final confidence model to drive Bolukbasi’s network selection policy, so that each input is routed to the machine learning production model — of a plurality of deployed production models — best matched to the confidence that the input will be predicted correctly (KSR rationale (A): combining prior art elements according to known methods to yield predictable results). The suggestion/motivation for doing so would have been expressly stated in Corbière itself: equipped with its confidence measure, “a system could decide to stick to the prediction or, on the contrary, to hand over to a human or a back-up system” (Corbière: Section 1). Bolukbasi supplies exactly such a confidence-driven hand-over mechanism over a plurality of production networks and demonstrates its predictable benefit, “up to a 2.8x speedup on state-of-the-art networks from the ImageNet image recognition challenge with minimal (< 1%) loss of top5 accuracy” (Bolukbasi: Abstract). Per claim 6, Corbière combined with Bolukbasi discloses claim 5. Bolukbasi further teaches: compare the confidence score of the input to a plurality of threshold values; determine whether the confidence score of the input satisfies one or more of the plurality of threshold values; and select the machine learning production model of the plurality of machine learning production models based on determining that the confidence score satisfies one or more of the plurality of threshold values (Bolukbasi: Section 5.1…Bolukbasi expressly teaches thresholding on model confidence, “a myopic policy which learns a single threshold based on model confidence”; Section 4… deploys a confidence-based decision function at each of multiple stages of its staged topology — κ₁ after evaluation of N1 and κ₂ after evaluation of N2 — so that an input’s confidence is tested at a plurality of decision points, each applying its own learned threshold and whether the input’s confidence satisfies the threshold at a given stage determines whether the prediction is returned or the input is routed onward to another production network, “κ₁ : |X| → {N 1 , N 2 , N 3 } is applied after evaluation of N 1 to determine if the classification decision from N 1 should be returned or if network N 2 or network N 3 should be evaluated for the example”; “For examples that are evaluated on N 2 , κ₂ : |X| → {N 2 , N 3 } determines if the classification decision from N 2 should be returned or if network N 3 should be evaluated”). The rationale to combine Bolukbasi with Corbière is the same as for claim 5. Per claim 7, Corbière combined with Bolukbasi discloses claim 5. Bolukbasi further teaches: perform a distribution analysis based on the confidence score of the input; and select the machine learning production model of the plurality of machine learning production models based on the distribution analysis (Bolukbasi: Section 5…Bolukbasi’s selection policy computes the entropy of the input’s prediction probabilities — an analysis of the distribution of the per-class confidence values that Bolukbasi expressly characterizes as the policy’s “confidence feedback” for the input — and the routing decision is made on those distribution-analysis features; under BRI, computing the entropy of the input’s per-class confidence (probability) distribution is a distribution analysis based on the confidence score of the input, “we augment the feature space with the entropy of prediction probabilities”; Fig. 2…“The policy evaluates Alexnet, receives confidence feedback and decides to jump directly to Resnet or send the sample to GoogLeNet->Resnet cascade”; Section 5.3…“entropy of classification decisions is an important feature in making policy decisions, as examples likely to be incorrectly classified by Alexnet are likely to be classified correctly by a later network”). The rationale to combine Bolukbasi with Corbière is the same as for claim 5. Per claim 8, Corbière combined with Bolukbasi discloses claim 5. Bolukbasi further teaches: the plurality of machine learning production models comprises a first machine learning production model associated with a high confidence score, a second machine learning production model associated with a medium confidence score, and a third machine learning production model associated with a low confidence score (Bolukbasi: Section 4…Bolukbasi’s deployed system comprises exactly three pre-trained production networks ordered by the confidence that the input is easily (correctly) classified: high-confidence inputs are answered by the first network (AlexNet), intermediate-confidence inputs are routed to the second (GoogLeNet), and low-confidence (difficult) inputs to the third (ResNet50), “assume we have three pre-trained networks, N 1 , N 2 , and N 3 ”; Section 5.1…“Following the evaluation of Alexnet, the system determines for each example either to return the prediction, route the example to GoogLeNet, or route the example to Resnet50”); determine whether the confidence score of the input corresponds to the high confidence score, the medium confidence score, or the low confidence score; and select the first machine learning production model associated with the high confidence score, the second machine learning production model associated with the medium confidence score, or the third machine learning production model associated with the low confidence score based on determining whether the confidence score of the input corresponds to the high confidence score, the medium confidence score, or the low confidence score (Bolukbasi: Fig. 2…the policy’s confidence-feedback decision assigns each input to the production network matching its confidence level — most inputs are easy (high confidence) and remain with the first network while progressively less-confident inputs are sent to the deeper networks, “The policy evaluates Alexnet, receives confidence feedback and decides to jump directly to Resnet or send the sample to GoogLeNet->Resnet cascade”; Fig. 6 “Majority of the samples are easily classified by Alexnet, and only a minority of them require deeper networks”). The rationale to combine Bolukbasi with Corbière is the same as for claim 5. Claims 13, 14 and 15 are substantially similar in scope and spirit as claims 5, 6 and 7. Therefore the rejections of claims 5, 6 and 7 are applied accordingly. Claim 20 is substantially similar in scope and spirit as claim 5. Therefore the rejections of claim 5 is applied accordingly. Contact Information Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALAN CHEN whose telephone number is (571) 272-4143. The examiner can normally be reached M-F 10-7. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /ALAN CHEN/ Primary Examiner, Art Unit 2125 Application/Control Number: 18/578,406 Page 2 Art Unit: 2125 Application/Control Number: 18/578,406 Page 3 Art Unit: 2125 Application/Control Number: 18/578,406 Page 4 Art Unit: 2125 Application/Control Number: 18/578,406 Page 5 Art Unit: 2125 Application/Control Number: 18/578,406 Page 6 Art Unit: 2125 Application/Control Number: 18/578,406 Page 7 Art Unit: 2125 Application/Control Number: 18/578,406 Page 8 Art Unit: 2125 Application/Control Number: 18/578,406 Page 9 Art Unit: 2125 Application/Control Number: 18/578,406 Page 10 Art Unit: 2125 Application/Control Number: 18/578,406 Page 11 Art Unit: 2125 Application/Control Number: 18/578,406 Page 12 Art Unit: 2125 Application/Control Number: 18/578,406 Page 13 Art Unit: 2125 Application/Control Number: 18/578,406 Page 14 Art Unit: 2125 Application/Control Number: 18/578,406 Page 15 Art Unit: 2125 Application/Control Number: 18/578,406 Page 16 Art Unit: 2125 Application/Control Number: 18/578,406 Page 17 Art Unit: 2125 Application/Control Number: 18/578,406 Page 18 Art Unit: 2125 Application/Control Number: 18/578,406 Page 19 Art Unit: 2125 Application/Control Number: 18/578,406 Page 20 Art Unit: 2125 Application/Control Number: 18/578,406 Page 21 Art Unit: 2125
Prosecution Insights