CTNF 18/488,951 CTNF 101910 Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Information Disclosure Statement The information disclosure statement (IDS) submitted on 11/09/2023 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner. 12-151 AIA 26-51 12-51 Status of Claims The present application is being examined under the claims filed on 10/17/2023. Claims 1-23 are rejected. Claims 1-23 are pending. Specification The specification filed on 10/17/2023 is acceptable for examination purposes. Drawings The drawings filed on 10/17/2023 are acceptable for examination purposes. Claim Rejections - 35 USC § 101 07-04-01 AIA 07-04 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1-23 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Regarding Claim 1 , Step 1 : Claim 1 is a method claim. Therefore, Claims 1-10 are directed to either a process, machine, manufacture, or composition of matter. Step 2A Prong 1 : processing the new model input [using a classifier machine learning model] to generate a model output that comprises: (i) a respective classification probability for each of a plurality of classification categories, and (ii) a respective misclassification probability that represents a likelihood that the new model input will be misclassified by an expert system that is different from the classifier machine learning model (mental process - processing the new model input using a classifier machine learning model to generate a model output may be performed manually by a user with the aid of pen and paper by observing/analyzing the model input and a classifier machine learning model function/algorithm. See MPEP 2106.04(a)(2)(III)(C).) determining, from the model output, whether to defer the new model input for processing by the expert system, comprising: (mental process – determining, from the model output, whether to defer the new model input for processing by the expert system and obtaining a cost value that represents a cost associated with deferring model inputs to the expert system may be performed manually by a user with the aid of pen and paper by observing/analyzing the model input and output. See MPEP 2106.04(a)(2)(III)(C).) obtaining a cost value that represents a cost associated with deferring model inputs to the expert system (mental process – obtaining a cost value that represents a cost associated with deferring model inputs to the expert system may be performed manually by a user with the aid of pen and paper by observing/analyzing the model input and a cost value. See MPEP 2106.04(a)(2)(III)(C).) determining whether to defer the new model input based on the cost value and the model output (mental process – determining whether to defer the new model input based on the cost value and the model output may be performed manually by a user with the aid of pen and paper by observing/analyzing the cost value and the model output. See MPEP 2106.04(a)(2)(III)(C).) Step 2A Prong 2 : The judicial exceptions are not integrated into a practical application. Additional Elements: a method performed by one or more computers (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).) using a classifier machine learning model (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).) obtaining a new model input (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).) in response to determining to defer the new model input: providing the new model input for processing by the expert system (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).) in response to determining to defer the new model input: obtaining, from the expert system, an expert classification output for the new model input (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).) in response to determining to defer the new model input: providing, as output, the expert classification output (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).) Step 2B : The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Additional Elements: a method performed by one or more computers (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).) using a classifier machine learning model (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).) obtaining a new model input (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer) in response to determining to defer the new model input: providing the new model input for processing by the expert system (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer) in response to determining to defer the new model input: obtaining, from the expert system, an expert classification output for the new model input; (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer) in response to determining to defer the new model input: providing, as output, the expert classification output (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer) For the reasons above, Claim 1 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 2-10. The additional limitations of the dependent claims are addressed below. Regarding Claim 2 , Step 2A Prong 1 : in response to determining not to defer the new model input: determining a classification output from the respective classification probabilities for each of the plurality of classification categories (mental process – determining a classification output from the respective classification probabilities for each of the plurality of classification categories may be performed manually by a user with the aid of pen and paper by observing/analyzing the respective classification probabilities for each of the plurality of classification categories. See MPEP 2106.04(a)(2)(III)(C).) Step 2A Prong 2 : The judicial exceptions are not integrated into a practical application. Additional Elements: in response to determining not to defer the new model input: providing, as output, the classification output (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).) Step 2B : The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Additional Elements: in response to determining not to defer the new model input: providing, as output, the classification output (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer) Regarding Claim 3 , Step 2A Prong 1 : wherein determining a classification output from the respective classification probabilities for each of the plurality of classification categories comprising: selecting a classification category that has a highest probability among the classification probabilities (mental process - selecting a classification category that has a highest probability among the classification probabilities may be performed manually by a user with the aid of pen and paper by observing/analyzing the classification probabilities. See MPEP 2106.04(a)(2)(III)(C).) wherein determining a classification output from the respective classification probabilities for each of the plurality of classification categories comprising: generating a classification output that identifies the selected classification category (mental process – generating a classification output that identifies the selected classification category may be performed manually by a user with the aid of pen and paper by observing/analyzing the selected classification category. See MPEP 2106.04(a)(2)(III)(C).) Step 2A Prong 2 & Step 2B : There are no additional elements. Regarding Claim 4 , Step 2A Prong 1 : wherein determining whether to defer the new model input based on the cost value and the model output comprises: determining to defer the new model only when one minus the highest probability among the classification probabilities is greater than or equal to the sum of (i) the cost value and (ii) the misclassification probability (mathematical process – determining to defer the new model only when one minus the highest probability among the classification probabilities is greater than or equal to the sum of the cost value and the misclassification probability may be performed by mathematical process, utilizing the highest probability among the classification probabilities, the cost value and the misclassification probability. See MPEP 2106.04(a)(2)(I)(C).) Step 2A Prong 2 & Step 2B : There are no additional elements. Regarding Claim 5 , Step 2A Prong 1 : wherein the classifier model is configured to generate: (i) a respective classification logit score for each of the plurality of classification categories, and (ii) a deferral logit score (mental process – generating a respective classification logit score for each of the plurality of classification categories may be performed manually by a user with the aid of pen and paper by observing/analyzing the classifier model and the classification categories. See MPEP 2106.04(a)(2)(III)(C).) wherein processing the new model input using a classifier machine learning model to generate a model output comprises: applying a softmax function to the classification logit scores to generate the respective classification probabilities (mathematical process – applying a softmax function to the classification logit scores to generate the respective classification probabilities may be performed by mathematical process, utilizing a softmax function and the classification logit scores. See MPEP 2106.04(a)(2)(I)(C).) wherein processing the new model input using a classifier machine learning model to generate a model output comprises: applying an inverse link function to the deferral logit score to generate the misclassification probability (mental process – applying an inverse link function to the deferral logit score to generate the misclassification probability may be performed manually by a user with the aid of pen and paper by observing/analyzing an inverse link function and the deferral logit score. See MPEP 2106.04(a)(2)(III)(C).) Step 2A Prong 2 & Step 2B : There are no additional elements. Regarding Claim 6 , Step 2A Prong 1 : wherein the inverse link function is a sigmoid function (mathematical process – the inverse link function is a sigmoid function may be performed by mathematical process, utilizing a sigmoid function. See MPEP 2106.04(a)(2)(I)(C).) Step 2A Prong 2 & Step 2B : There are no additional elements. Regarding Claim 7 , Step 2A Prong 1 : See the rejection of Claim 1 above, which Claim 7 depends on. Step 2A Prong 2 : The judicial exceptions are not integrated into a practical application. Additional Elements: training the classifier machine learning model on a training data set to minimize a loss function that measures a performance of classification outputs generated based on model outputs generated by the classifier machine learning model (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).) Step 2B : The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Additional Elements: training the classifier machine learning model on a training data set to minimize a loss function that measures a performance of classification outputs generated based on model outputs generated by the classifier machine learning model (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).) Regarding Claim 8 , Step 2A Prong 1 : wherein the loss function assumes that the cost value is zero (mathematical process – the loss function assumes that the cost value is zero may be performed by mathematical process, utilizing the loss function and the cost value. See MPEP 2106.04(a)(2)(I)(C).) Step 2A Prong 2 & Step 2B : There are no additional elements. Regarding Claim 9 , Step 2A Prong 1 : wherein the loss function is a hybrid loss function that estimates ground truth classification outputs using a cross-entropy loss and ground truth expert misclassification probabilities using a learning to defer loss (mathematical process – the loss function is a hybrid loss function that estimates ground truth classification outputs using a cross-entropy loss and ground truth expert misclassification probabilities using a learning to defer loss may be performed by mathematical process, utilizing a cross-entropy loss and misclassification probabilities. See MPEP 2106.04(a)(2)(I)(C).) Step 2A Prong 2 & Step 2B : There are no additional elements. Regarding Claim 10 , Step 2A Prong 1 : wherein the learning to defer loss is a one-versus-all (OvA) loss (mental process – the learning to defer loss is a one-versus-all (OvA) loss may be performed manually by a user with the aid of pen and paper by observing/analyzing the loss function and one-versus-all (OvA) loss. See MPEP 2106.04(a)(2)(III)(C).) Step 2A Prong 2 & Step 2B : There are no additional elements. Regarding Claim 11 , Step 1 : Claim 11 is a method claim. Therefore, Claims 11-22 are directed to either a process, machine, manufacture, or composition of matter. Step 2A Prong 1 : to process the model input to generate a model output that comprises:(i) a respective classification logit score for each of a plurality of classification categories, and (ii) a deferral logit score (mental process - processing the model input to generate a model output that comprises: a respective classification logit score for each of a plurality of classification categories, and a deferral logit score may be performed manually by a user with the aid of pen and paper by observing/analyzing the model input and the classification categories. See MPEP 2106.04(a)(2)(III)(C).) Step 2A Prong 2 : The judicial exceptions are not integrated into a practical application. Additional Elements: a method performed by one or more computers and for training a classifier machine learning model, wherein the classifier machine learning model is configured (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).) to receive as input a model input (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).) training the classifier machine learning model on a first training data set to minimize a loss function that measures a performance of a system that determines, using at least the deferral logit scores, whether to generate classification outputs based on model outputs generated by the classifier machine learning model or based on expert classification outputs generated by an expert system (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).) after training the classifier machine learning model on the first training data set, training the classifier machine learning model on a second training data set to minimize a surrogate rejector loss that, for any given model input, measures an expected loss incurred by deferring the given model input for processing by the expert system given a) whether the given model input is misclassified by the classifier machine learning model and the expert system and b) a cost value that represents a cost associated with deferring model inputs to the expert system (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).) Step 2B : The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Additional Elements: a method performed by one or more computers and for training a classifier machine learning model, wherein the classifier machine learning model is configured (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).) to receive as input a model input (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer) training the classifier machine learning model on a first training data set to minimize a loss function that measures a performance of a system that determines, using at least the deferral logit scores, whether to generate classification outputs based on model outputs generated by the classifier machine learning model or based on expert classification outputs generated by an expert system (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).) after training the classifier machine learning model on the first training data set, training the classifier machine learning model on a second training data set to minimize a surrogate rejector loss that, for any given model input, measures an expected loss incurred by deferring the given model input for processing by the expert system given a) whether the given model input is misclassified by the classifier machine learning model and the expert system and b) a cost value that represents a cost associated with deferring model inputs to the expert system (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).) For the reasons above, Claim 11 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 12-22. The additional limitations of the dependent claims are addressed below. Regarding Claim 12 , Step 2A Prong 1 : processing the new model input [using the classifier machine learning model] to generate a new model output (mental process - processing the new model input using the classifier machine learning model to generate a new model output may be performed manually by a user with the aid of pen and paper by observing/analyzing the model input and a classifier machine learning model function/algorithm. See MPEP 2106.04(a)(2)(III)(C).) determining, from the new model output, whether to defer the new model input for processing by the expert system (mental process – determining, from the new model output, whether to defer the new model input for processing by the expert system may be performed manually by a user with the aid of pen and paper by observing/analyzing the model input/output. See MPEP 2106.04(a)(2)(III)(C).) Step 2A Prong 2 : The judicial exceptions are not integrated into a practical application. Additional Elements: after training the classifier machine learning model on the second training data set, obtaining a new model input (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).) using the classifier machine learning model (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).) in response to determining to defer the new model input: providing the new model input for processing by the expert system (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).) in response to determining to defer the new model input: obtaining, from the expert system, an expert classification output for the new model input (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).) in response to determining to defer the new model input: providing, as output, the expert classification output (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).) Step 2B : The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Additional Elements: after training the classifier machine learning model on the second training data set, obtaining a new model input (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer) using the classifier machine learning model (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).) in response to determining to defer the new model input: providing the new model input for processing by the expert system (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer) in response to determining to defer the new model input: obtaining, from the expert system, an expert classification output for the new model input (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer) in response to determining to defer the new model input: providing, as output, the expert classification output (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer) Regarding Claim 13 , Step 2A Prong 1 : in response to determining not to defer the new model input: determining a classification output from the respective classification logits for each of the plurality of classification categories in the new model output (mental process - determining a classification output from the respective classification logits for each of the plurality of classification categories in the new model output may be performed manually by a user with the aid of pen and paper by observing/analyzing the respective classification logits for each of the plurality of classification categories in the new model output. See MPEP 2106.04(a)(2)(III)(C).) Step 2A Prong 2 : The judicial exceptions are not integrated into a practical application. Additional Elements: in response to determining not to defer the new model input: providing, as output, the classification output (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).) Step 2B : The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Additional Elements: in response to determining not to defer the new model input: providing, as output, the classification output (MPEP 2106.05(d)(II) indicates that merely “Presenting offers and gathering statistics” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer) Regarding Claim 14 , Step 2A Prong 1 : wherein determining a classification output from the respective classification logits for each of the plurality of classification categories in the new model output comprises: selecting a classification category having a largest classification logit from among the respective classification logits in the new model output (mental process – selecting a classification category having a largest classification logit from among the respective classification logits in the new model output may be performed manually by a user with the aid of pen and paper by observing/analyzing the respective classification logits in the new model output. See MPEP 2106.04(a)(2)(III)(C).) Step 2A Prong 2 & Step 2B : There are no additional elements. Regarding Claim 15 , Step 2A Prong 1 : wherein determining, from the new model output, whether to defer the new model input for processing by the expert system comprises: determining not to defer the new model input only when a largest classification logit in the new model output is larger than the deferral logit in the new model output (mental process – determining not to defer the new model input only when a largest classification logit in the new model output is larger than the deferral logit in the new model output may be performed manually by a user with the aid of pen and paper by observing/analyzing a largest classification logit in the new model output and the deferral logit in the new model output. See MPEP 2106.04(a)(2)(III)(C).) Step 2A Prong 2 & Step 2B : There are no additional elements. Regarding Claim 16 , Step 2A Prong 1 : wherein the classifier machine learning model is configured to: process the model input to generate a shared embedding of the model input (mental process – processing the model input to generate a shared embedding of the model input may be performed manually by a user with the aid of pen and paper by observing/analyzing the model input. See MPEP 2106.04(a)(2)(III)(C).) wherein the classifier machine learning model is configured to: for each classification category, apply a respective set of logit weights for the classification category to the shared embedding to generate the classification logit for the classification category, and apply a set of deferral logit weights to the shared embedding to generate the deferral logit (mental process – applying a respective set of logit weights for the classification category to the shared embedding to generate the classification logit for the classification category, and applying a set of deferral logit weights to the shared embedding to generate the deferral logit may be performed manually by a user with the aid of pen and paper by observing/analyzing each classification category, a set of logit weights, a set of deferral logit weights and the shared embedding. See MPEP 2106.04(a)(2)(III)(C).) Step 2A Prong 2 & Step 2B : There are no additional elements. Regarding Claim 17 , Step 2A Prong 1 : wherein training the classifier on the second training data set comprises training the deferral logit weights while holding the classification logit weights for the classification categories fixed (mental process – training the deferral logit weights while holding the classification logit weights for the classification categories fixed may be performed manually by a user with the aid of pen and paper by observing/analyzing the deferral logit weights and the classification logit weights. See MPEP 2106.04(a)(2)(III)(C).) Step 2A Prong 2 & Step 2B : There are no additional elements. Regarding Claim 18 , Step 2A Prong 1 : wherein the surrogate rejector loss, for any given model input, (i) is based on an output of a rejector function that is applied to a given model output for the given model input and that defines whether the given model input will be deferred to the expert system for processing and (ii) includes a) a first term that is based on a non-deferral confidence score for the given model input derived from the output of the rejector function and on whether the given model input is misclassified by the given model output and b) a second term that is based on a deferral confidence score for the given model input derived from the output of the rejector function and a cost value that represents a cost associated with deferring model inputs to the expert system (mental process – the surrogate rejector loss, for any given model input, (i) is based on an output of a rejector function that is applied to a given model output for the given model input and that defines whether the given model input will be deferred to the expert system for processing and (ii) includes a) a first term that is based on a non-deferral confidence score for the given model input derived from the output of the rejector function and on whether the given model input is misclassified by the given model output and b) a second term that is based on a deferral confidence score for the given model input derived from the output of the rejector function and a cost value that represents a cost associated with deferring model inputs to the expert system may be performed manually by a user with the aid of pen and paper by observing/analyzing an output of a rejector function, a model input and a cost value. See MPEP 2106.04(a)(2)(III)(C).) Step 2A Prong 2 & Step 2B : There are no additional elements. Regarding Claim 19 , Step 2A Prong 1 : wherein the rejector function measures a difference between a largest classification logit in the given model output and the deferral logit in the given model output (mathematical process – the rejector function measures a difference between a largest classification logit in the given model output and the deferral logit in the given model output may be performed by mathematical process, measuring a difference between a largest classification logit in the given model output and the deferral logit in the given model output. See MPEP 2106.04(a)(2)(I)(C).) Step 2A Prong 2 & Step 2B : There are no additional elements. Regarding Claim 20 , Step 2A Prong 1 : wherein the rejector function is based on an input-dependent bias term and probabilities determined from the classification logits in the given model output, the deferral logit in the given model output, or both (mental process - the rejector function is based on an input-dependent bias term and probabilities determined from the classification logits in the given model output, the deferral logit in the given model output, or both may be performed manually by a user with the aid of pen and paper by observing/analyzing an input-dependent bias term and probabilities, the classification logits in the given model output and the deferral logit in the given model output. See MPEP 2106.04(a)(2)(III)(C).) Step 2A Prong 2 & Step 2B : There are no additional elements. Regarding Claim 21 , Step 2A Prong 1 : wherein the deferral confidence score is generated by applying a proper composite loss to a negative of the output of the rejector function (mental process - the deferral confidence score is generated by applying a proper composite loss to a negative of the output of the rejector function may be performed manually by a user with the aid of pen and paper by observing/analyzing a composite loss and a negative of the output of the rejector function. See MPEP 2106.04(a)(2)(III)(C).) Step 2A Prong 2 & Step 2B : There are no additional elements. Regarding Claim 22 , Step 2A Prong 1 : wherein the non-deferral confidence score is generated by applying a proper composite loss to the output of the rejector function (mental process - the non-deferral confidence score is generated by applying a proper composite loss to the output of the rejector function may be performed manually by a user with the aid of pen and paper by observing/analyzing a composite loss and the output of the rejector function. See MPEP 2106.04(a)(2)(III)(C).) Step 2A Prong 2 & Step 2B : There are no additional elements. Regarding Claim 23 , Claim 23 recites substantially the same limitations as Claim 1, in the form of a system which performs the method of Claim 1. Therefore, it is rejected under the same rationale. Step 1 : Claim 23 is a system claim. Therefore, Claim 23 is directed to either a process, machine, manufacture, or composition of matter. Step 2A Prong 1 : processing the new model input [using a classifier machine learning model] to generate a model output that comprises: (i) a respective classification probability for each of a plurality of classification categories, and (ii) a respective misclassification probability that represents a likelihood that the new model input will be misclassified by an expert system that is different from the classifier machine learning model (mental process - processing the new model input using a classifier machine learning model to generate a model output may be performed manually by a user with the aid of pen and paper by observing/analyzing the model input and a classifier machine learning model function/algorithm. See MPEP 2106.04(a)(2)(III)(C).) determining, from the model output, whether to defer the new model input for processing by the expert system, comprising: obtaining a cost value that represents a cost associated with deferring model inputs to the expert system (mental process – determining, from the model output, whether to defer the new model input for processing by the expert system and obtaining a cost value that represents a cost associated with deferring model inputs to the expert system may be performed manually by a user with the aid of pen and paper by observing/analyzing the model input/output and a cost value. See MPEP 2106.04(a)(2)(III)(C).) determining, from the model output, whether to defer the new model input for processing by the expert system, comprising: determining whether to defer the new model input based on the cost value and the model output (mental process – determining whether to defer the new model input based on the cost value and the model output may be performed manually by a user with the aid of pen and paper by observing/analyzing the cost value and the model output. See MPEP 2106.04(a)(2)(III)(C).) Step 2A Prong 2 : The judicial exceptions are not integrated into a practical application. Additional Elements: a method performed by one or more computers (recited at a high-level of generality (i.e., as generic one or more computers) such that it amounts to no more than mere instructions to apply the exception using generic computer components. See MPEP 2106.05(f).) obtaining a new model input (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).) using a classifier machine learning model (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).) in response to determining to defer the new model input: providing the new model input for processing by the expert system (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).) in response to determining to defer the new model input: obtaining, from the expert system, an expert classification output for the new model input (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).) in response to determining to defer the new model input: providing, as output, the expert classification output (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).) Step 2B : The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Additional Elements: a method performed by one or more computers (mere instructions to apply the exception using generic computer components cannot provide an inventive concept) obtaining a new model input (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer) using a classifier machine learning model (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).) in response to determining to defer the new model input: providing the new model input for processing by the expert system (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer) in response to determining to defer the new model input: obtaining, from the expert system, an expert classification output for the new model input (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer) in response to determining to defer the new model input: providing, as output, the expert classification output (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer) For the reasons above, Claim 23 is rejected as being directed to an abstract idea without significantly more. Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim s 1-3 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Madras et al. ("Predict responsibly: improving fairness and accuracy by learning to defer.") (hereinafter Madras), in view of Mozannar et al. ("Consistent estimators for learning to defer to an expert.") (hereinafter Mozannar) . Regarding Claim 1 , Madras teaches: “A method performed by one or more computers, the method comprising:” (preamble) “obtaining a new model input” (Madras, Data in Fig. 1; Examiner’s note: Data X1, X2, X3 and X4 teach model inputs. PNG media_image1.png 165 622 media_image1.png Greyscale ) “processing the new model input using a classifier machine learning model generate a model output that comprises:” (Madras, Data, Model and Output in Fig. 1, Section 2.1, “a machine learning model be combined with the DM to leverage the DM’s extra insight […]”; Examiner’s note: Data X1, X2, X3 and X4 teach model inputs, Model is a classifier machine learning model and Output further teaches a model output.) “(i) a respective classification probability for each of a plurality of classification categories” (Madras, Section 2.1, “where P M is the probability assigned by the automated model, P D is the probability assigned by the DM (decision maker), and i indexes examples.”; Examiner’s note: i indexes examples teach a plurality of classification categories .) “determining, from the model output, whether to defer the new model input for processing by the expert system” (Madras, Fig. 1 & Section 2.1, “the first-step model can either predict (positive/negative) or say PASS […] if it says PASS, the DM (decision maker) makes its own decision”; Examiner’s note: PASS teaches deferring to DM which is an expert .) “in response to determining to defer the new model input: providing the new model input for processing by the expert system” (Madras, Fig. 1, Examiner’s note: passing the model input to DM teaches providing the new model input ) “obtaining, from the expert system, an expert classification output for the new model input; and providing, as output, the expert classification output” (Madras, Fig. 1 & Section 2.1, “if it says PASS, the DM (decision maker) makes its own decision”; Examiner’s note: receiving DM’s own decision teaches obtaining and providing the expert classification output ) Madras does not explicitly teach a respective misclassification probability that represents a likelihood that the new model input will be misclassified by an expert system that is different from the classifier machine, obtaining a cost value that represents a cost associated with deferring model inputs to the expert system and determining whether to defer the new model input based on the cost value and the model output. Mozannar teaches “Consistent estimators for learning to defer to an expert (title)” comprising: “(ii) a respective misclassification probability that represents a likelihood that the new model input will be misclassified by an expert system that is different from the classifier machine learning model” (Mozannar, Section 3, “Our goal is to build a predictor Ŷ : X -> Y Ս {ꓕ} that can either predict or defer the decision to the expert denoted by ꓕ. Our strategy for learning the predictor Ŷ will be to learn two separate functions h : X -> Y (classifier) and r : X -> {0, 1} (rejector).”; Examiner’s note: learning the predictor Ŷ, classifier and rejector teaches a respective misclassification probability representing that the new model input will be misclassified by an expert (rejector) that is different from the classifier machine learning model (classifier). ) “obtaining a cost value that represents a cost associated with deferring model inputs to the expert system” (Mozannar, Section 3, “c[K + 1] represents the cost of deferring to the expert.”) “determining whether to defer the new model input based on the cost value and the model output” (Mozannar, Section 3, “Querying the expert implies deferring the decision which incurs a cost l exp (x, y, m) that depends on the target y, covariate x and the expert’s prediction m”; Examiner’s note: the expert’s prediction teaches the model output and deferring the decision incurs a cost depending on the expert’s prediction. ) It would have been obvious to one having ordinary skill in the art before the effective filing date of the invention was made to modify the invention in Madras by applying the cost value as taught in Mozannar to the expert system in Madras in order to “minimize [] the cost sensitive loss” (Mozannar, Section 3, Col. 1, Lines 23-24). Regarding Claim 2 , The combination of Madras and Mozannar teaches: “The method of claim 1, further comprising:” (preamble) “in response to determining not to defer the new model input: determining a classification output from the respective classification probabilities for each of the plurality of classification categories” (Madras, Model and Output in Fig. 1 and Section 2.1, “For convenience, we compress the probabilistic notation: PNG media_image2.png 58 602 media_image2.png Greyscale “; Examiner’s note: Model and Output in Fig. 1 teaches determining not to defer the new model input (Not PASS) and in equation (2) Ŷ is a system prediction which further teaches a classification output from the respective classification probabilities.) “providing, as output, the classification output” (Madras, Output in Fig. 1, Examiner’s note: Output teaches the classification output. ) The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein. Regarding Claim 3 , The combination of Madras and Mozannar teaches: “The method of claim 2, wherein determining a classification output from the respective classification probabilities for each of the plurality of classification categories comprising:” (preamble) “selecting a classification category that has a highest probability among the classification probabilities” (Mozannar, Section 4, “h(x) = arg max y ∈ y g y ”; Examiner’s note: arg max teaches selecting a classification category with a highest probability. ) “generating a classification output that identifies the selected classification category” (Madras, Section 2.1, “ PNG media_image3.png 29 254 media_image3.png Greyscale “; Examiner’s note: Ŷ M is a model prediction which teaches a classification output.) The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein. Regarding Claim 7 , The combination of Madras and Mozannar teaches: “The method of claim 1, further comprising:” (preamble) “training the classifier machine learning model on a training data set to minimize a loss function that measures a performance of classification outputs generated based on model outputs generated by the classifier machine learning model” (Mozannar, Section 3, “The goal of this setup is to learn a predictor h : X -> [K + 1] minimizing the cost sensitive loss […]”; Mozannar, Section 6.2, “We can see that the model trained with L 1 CE dominates all other baselines giving better coverage and accuracy for the classifier’s predictions. This gives evidence that our loss allows the model to only predict when it is highly confident.”; Examiner’s note: the classifier’s predictions teach model outputs by the classifier machine learning model and a predictor h : X -> [K + 1] minimizing the cost sensitive loss teaches minimizing a loss function that measures a performance of classification outputs. ) The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein . 07-21-aia AIA Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Madras in view of Mozannar as applied in claim 1, and further in view of Chow ("On optimum recognition error and reject tradeoff.") . Regarding Claim 4 , The combination of Madras and Mozannar teaches: “The method of claim 1, wherein determining whether to defer the new model input based on the cost value and the model output comprises:” (preamble) the cost value and the misclassification probability (Mozannar, Section 3, “the costs are the misclassification error with the target. Formally, we define a 0-1 loss version of our system loss: PNG media_image4.png 86 568 media_image4.png Greyscale ”; Examiner’s note: the cost value is taught by Mozannar – see supra claim 1; the misclassification error with the target teaches misclassification probability.) Madras and Mozannar fail to teach determining to defer the new model only when one minus the highest probability among the classification probabilities. Chow teaches “On optimum recognition error and reject tradeoff (title)” comprising: determining to defer the new model only when one minus the highest probability among the classification probabilities (Chow, Section II, “ PNG media_image5.png 108 262 media_image5.png Greyscale “; Examiner’s note: 1 – max(probability) is taught by Chow.) It would have been obvious to one having ordinary skill in the art before the effective filing date of the invention was made to modify the combination of Madras and Mozannar by applying one minus the highest probability as taught in Chow to the expert in Mozannar in order to minimize the error rate (Chow, Fig. 2 teaches error-reject tradeoff curve) . 07-21-aia AIA Claim s 5, 6, 9 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Madras in view of Mozannar as applied in claim 1, and further in view of Verma et al. ("Calibrated learning to defer with one-vs-all classifiers.") (hereinafter Verma) . Regarding Claim 5 , The combination of Madras and Mozannar teaches: “The method of claim 1,” (preamble) “wherein the classifier model is configured to generate:” (Madras – see supra claim 1) “(i) a respective classification logit score for each of the plurality of classification categories and” (Madras, Section 3.1, “We use a neural network as our binary classifier […]”; Examiner’ note: using neural network classifiers teaches generating a respective classification logit score .) “(ii) a deferral logit score” (Madras, Equation (9), “ PNG media_image6.png 63 567 media_image6.png Greyscale “; Examiner’s note: equation (9) teaches a coefficient 𝜋 , where s ∼ Ber( 𝜋 ). 𝜋 is the probability of deferral and the probability of deferral is calculated using a deferral logit score. ) “applying a softmax function to the classification logit scores to generate the respective classification probabilities” (Madras, Equation (1), “ PNG media_image7.png 65 593 media_image7.png Greyscale “; Examiner’s note: P M which is the probability by the model which teaches the respective classification probabilities. Applycing a softmax function to classification logit scores was old and well known in the art before the effective filing date, as it provides means to provide probabilistic interpretability and multi-class support.) Madras and Mozannar fail to teach applying an inverse link function to the deferral logit score to generate the misclassification probability. Verma teaches “Calibrated learning to defer with one-vs-all classifiers (title)” comprising: “applying an inverse link function to the deferral logit score to generate the misclassification probability” (Verma, Section 4.1, “ PNG media_image8.png 28 318 media_image8.png Greyscale “; Examiner’s note: equation (9) teaches an inverse link function is applied to the rejector to generate the misclassification probability and the rejector teaches the deferral logit score.) It would have been obvious to one having ordinary skill in the art before the effective filing date of the invention was made to modify the combination of Madras and Mozannar by applying an inverse function as taught in Verma to the deferral logit score in Madras so that “our one-vs-all loss results in models better calibrated than those trained with the softmax-based surrogate” (Verma, Section 7). Regarding Claim 6 , The combination of Madras, Mozannar and Verma teaches: “The method of claim 5, wherein the inverse link function is a sigmoid function” (Verma, Theorem 4.1, “For a strictly proper binary composite loss ϕ with a well-defined continuous inverse link function γ − 1, ψOvA (Equation 8) is a calibrated surrogate for the 0 – 1 learning to defer loss (Equation 2).”; Examiner’s note: a calibrated surrogate for the 0-1 range teaches using a sigmoid function .) The reasons of obviousness have been noted in the rejection of Claim 5 above and applicable herein. Regarding Claim 9 , The combination of Madras, Mozannar and Verma teaches: “The method of claim 7, wherein the loss function is a hybrid loss function that estimates ground truth classification outputs using a cross-entropy loss and ground truth expert misclassification probabilities using a learning to defer loss” (Verma, Section 2, “reduction to cost sensitive learning that ultimately resembles the cross-entropy loss for a softmax parameterization”; Verma, Section 4.1, “the OvA formulation directly estimates the probability that the expert is correct”; Verma, Corollary 4.2, “ψOvA is a consistent surrogate for the 0-1 learning to defer loss”; Examiner’s note: cost sensitive learning teaches ground truth classification outputs using a cross-entropy loss, the probability that the expert is correct and a consistent surrogate for the 0-1 learning to defer loss teach ground truth expert misclassification probabilities using a learning to defer loss .) It would have been obvious to one having ordinary skill in the art before the effective filing date of the invention was made to modify the combination of Madras and Mozannar by using hybrid loss function as taught in Verma to the DM (decision maker) in Madras in order to “have[] an appropriate estimator” (Verma, Section 4.1). Regarding Claim 10 , The combination of Madras, Mozannar and Verma teaches: “The method of claim 9, wherein the learning to defer loss is a one-versus-all (OvA) loss” (Verma, Section 4.1, Col. 2, Lines 5-6 “Our one-vs-all (OvA) surrogate loss takes the following point-wise form […]”) The reasons of obviousness have been noted in the rejection of Claim 9 above and applicable herein . 07-21-aia AIA Claim s 8, 11-15 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Madras in view of Mozannar as applied in claim 1, and further in view of Charusaie et al. ("Sample efficient learning of predictors that complement humans.") (hereinafter Charusaie) . Regarding Claim 8 , The combination of Madras and Mozannar teaches: “The method of claim 7,” (Preamble) Madras and Mozannar do not explicitly teach wherein the loss function assumes that the cost value is zero. Charusaie teaches “Sample efficient learning of predictors that complement humans (title)” comprising: “wherein the loss function assumes that the cost value is zero” (Charusaie, Section 6.1, “ PNG media_image9.png 53 333 media_image9.png Greyscale ”; Examiner’s note: zero error L(h*,r*) = 0 teaches the loss function assuming that the cost value is zero .) It would have been obvious to one having ordinary skill in the art before the effective filing date of the invention was made to modify the combination of Madras and Mozannar by using a loss function assuming that the cost value is zero as taught in Charusaie to the classifier machine learning model in Madras in order to “care about misclassification costs with no additional penalties” (Charusaie, Section 3). Regarding Claim 11 , The combination of Madras, Mozannar and Charusaie teaches: “A method performed by one or more computers and for training a classifier machine learning model, wherein the classifier machine learning model is configured to receive as input a model input and to process the model input to generate a model output that comprises:” (Madras, Data and Model in Fig. 1; Examiner’s note: Data X1, X2, X3 and X4 teach model inputs and Model can be a classifier machine learning model. Learning-to-defer teaches training a classifier machine learning model. PNG media_image1.png 165 622 media_image1.png Greyscale ) “(i) a respective classification logit score for each of a plurality of classification categories” (Madras – see supra claim 5) “(ii) a deferral logit score” (Madras – see supra claim 5) “training the classifier machine learning model on a first training data set to minimize a loss function that measures a performance of a system that determines, using at least the deferral logit scores, whether to generate classification outputs based on model outputs generated by the classifier machine learning model or based on expert classification outputs generated by an expert system” (Charusaie, Section 7, Col. 2, Lines 11-5, “For staged learning, we train the classifier on the training data optimizing for performance on a validation set, and for the rejector we train a network to predict the expert error and defer at test time by comparing the confidence of the classifier and the expert”; Examiner’s note: Staged learning teaches different stage training. Training the classifier on the training data optimizing for performance further teaches training the classifier machine learning model on a first training data set to minimize loss function that measures a performance of a system . The confidence of the classifier and the expert further teaches the classification outputs based on model outputs generated by the classifier machine learning model or based on expert classification outputs generated by an expert system .) “after training the classifier machine learning model on the first training data set, training the classifier machine learning model on a second training data set” (Charusaie, Section 4.1, Col. 1, Lines 3-7, “first learning a classifier […] and then, learning a rejector that defers each point to either classifier or the expert, depending on who has a lower estimated error […]”; Examiner’s note: after first learning a classifier and then learning a rejector deferring to either classifier or the expert teaches training the classifier machine learning model on a second training data set after first training .) “to minimize a surrogate rejector loss that, for any given model input, measures an expected loss incurred by deferring the given model input for processing by the expert system given a) whether the given model input is misclassified by the classifier machine learning model and the expert system and b) a cost value that represents a cost associated with deferring model inputs to the expert system” (Charusaie, Section 5.1, “ PNG media_image10.png 65 329 media_image10.png Greyscale PNG media_image11.png 58 316 media_image11.png Greyscale PNG media_image12.png 121 327 media_image12.png Greyscale PNG media_image13.png 114 329 media_image13.png Greyscale “; Examiner’s note: In, Proposition 2, a consistent surrogate for multi-class classification teaches a surrogate rejector loss that measures a loss incurred by deferring to the expert system given whether the given model input is misclassified by the classifier and the expert. A surrogate for cost-sensitive learning further teaches a cost value representing a cost associated with deferring model inputs to the expert system .) It would have been obvious to one having ordinary skill in the art before the effective filing date of the invention was made to modify the combination of Madras and Mozannar by using a staged learning approach as taught in Charusaie to the classifier machine learning model in Madras in order to “improve sample complexity over joint learning” (Charusaie, Section 4.2). Regarding Claim 12 , The combination of Madras, Mozannar and Charusaie teaches: “The method of claim 11, further comprising:” (preamble) “after training the classifier machine learning model on the second training data set” (Charusaie, Section 4.1, Col. 1, Lines 5-7, “[…] and then, learning a rejector that defers each point to either classifier or the expert, depending on who has a lower estimated error […]”; Examiner’s note: learning a rejector deferring to either classifier or the expert teaches training the classifier machine learning model on the second training data set .) “obtaining a new model input” (Madras – see supra claim 1) “processing the new model input using the classifier machine learning model to generate a new model output” (Madras – see supra claim 1) “determining, from the new model output, whether to defer the new model input for processing by the expert system” (Madras – see supra claim 1) “in response to determining to defer the new model input: providing the new model input for processing by the expert system” (Madras – see supra claim 1) “obtaining, from the expert system, an expert classification output for the new model input; and providing, as output, the expert classification output” (Madras – see supra claim 1) The reasons of obviousness have been noted in the rejection of Claim 11 above and applicable herein. Regarding Claim 13 , The combination of Madras, Mozannar and Charusaie teaches: “The method of claim 12, further comprising:” (preamble) “in response to determining not to defer the new model input: determining a classification output from the respective classification logits for each of the plurality of classification categories in the new model output” (Madras – see supra claim 2) “providing, as output, the classification output” (Madras – see supra claim 2) The reasons of obviousness have been noted in the rejection of Claim 11 above and applicable herein. Regarding Claim 14 , The combination of Madras, Mozannar and Charusaie teaches: “The method of claim 13, wherein determining a classification output from the respective classification logits for each of the plurality of classification categories in the new model output comprises:” (preamble) “selecting a classification category having a largest classification logit from among the respective classification logits in the new model output” (Mozannar, Section 4, “h(x) = arg max y ∈ y g y ”; Examiner’s note: arg max teaches selecting a highest classification probability which is mathematically equivalent to selecting a largest classification logit. ) The reasons of obviousness have been noted in the rejection of Claim 11 above and applicable herein. Regarding Claim 15 , The combination of Madras, Mozannar and Charusaie teaches: “The method of claim 12, wherein determining, from the new model output, whether to defer the new model input for processing by the expert system comprises:” (preamble) “determining not to defer the new model input only when a largest classification logit in the new model output is larger than the deferral logit in the new model output” (Mozannar, Equation (5), “ PNG media_image14.png 57 293 media_image14.png Greyscale ”; Examiner’s note: equation (5) teaches rejecting (deferring) when a largest classification logit is less than or equal to the deferral probability/logit in the new model output. ) The reasons of obviousness have been noted in the rejection of Claim 11 above and applicable herein. Regarding Claim 18 , The combination of Madras, Mozannar and Charusaie teaches: “The method of claim 11, wherein the surrogate rejector loss, for any given model input, (i) is based on an output of a rejector function that is applied to a given model output for the given model input and that defines whether the given model input will be deferred to the expert system for processing” (Charusaie, Equations (9) and (10), Examiner’s note: Equation (10) teaches the surrogate rejector loss, for any given model input, is based on an output of a rejector function applied to a given model output for the given model input. Equation (9) teaches whether the given model input will be deferred to the expert system for processing (i.e. when r(x) = 1, the given model input will be deferred to the expert system for processing). PNG media_image10.png 65 329 media_image10.png Greyscale PNG media_image11.png 58 316 media_image11.png Greyscale ) “(ii) includes a) a first term that is based on a non-deferral confidence score for the given model input derived from the output of the rejector function and on whether the given model input is misclassified by the given model output” (Charusaie, Equation (9), Examiner’s note: Equation (9) teaches a first term that is based on a non-deferral confidence score for the given model input derived from the output of the rejector function and on whether the given model input is misclassified by the given model output. r(x) = 0 is the first term based on a non-deferral confidence score for the given model input. PNG media_image10.png 65 329 media_image10.png Greyscale ) “b) a second term that is based on a deferral confidence score for the given model input derived from the output of the rejector function and a cost value that represents a cost associated with deferring model inputs to the expert system” (Charusaie, Equation (9), Examiner’s note: equation (9) teaches a second term that is based on a deferral confidence score for the given model input derived from the output of the rejector function and a cost value that represents a cost associated with deferring model inputs to the expert system. r(x) = 1 is the second term based on a deferral confidence score for the given model input and equation (10) teaches a cost value representing a cost associated with deferring model inputs to the expert system. PNG media_image10.png 65 329 media_image10.png Greyscale PNG media_image11.png 58 316 media_image11.png Greyscale ) The reasons of obviousness have been noted in the rejection of Claim 11 above and applicable herein. Regarding Claim 19 , The combination of Madras, Mozannar and Charusaie teaches: “The method of claim 18, wherein the rejector function measures a difference between a largest classification logit in the given model output and the deferral logit in the given model output” (Mozannar, Equation (5), “ PNG media_image14.png 57 293 media_image14.png Greyscale ”; Examiner’s note: equation (5) teaches rejecting (deferring) when a largest classification logit is less than or equal to the deferral probability/logit in the given model output. ) ) The reasons of obviousness have been noted in the rejection of Claim 11 above and applicable herein. Regarding Claim 20 , The combination of Madras, Mozannar and Charusaie teaches: The method of claim 18, wherein the rejector function is based on an input-dependent bias term and probabilities determined from the classification logits in the given model output, the deferral logit in the given model output, or both (Madras, Equation (9), “ PNG media_image6.png 63 567 media_image6.png Greyscale “; Examiner’s note: in equation (9), Y is an input label which teaches an input-dependent bias term; Ŷ M is a model prediction which is a probability determined from the classification logits in the given model output; Ŷ D is a DM prediction which is a probability determined from the deferral logit in the given model output ) The reasons of obviousness have been noted in the rejection of Claim 11 above and applicable herein . 07-21-aia AIA Claim s 16 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Madras in view of Mozannar and further in view of Charusaie as applied in claim 11, in view of He et al. ("Deep residual learning for image recognition.") (hereinafter He) . Regarding Claim 16 , The combination of Madras, Mozannar and Charusaie teaches: “The method of claim 11, wherein the classifier machine learning model is configured to:” (preamble) apply a set of deferral logit weights to generate the deferral logit (Mozannar, Equation (3), “ PNG media_image15.png 127 562 media_image15.png Greyscale “; Examiner’s note: in equation (3), g 1 , …, g k+1 teaches a set of deferral logit weights applied to generate the deferral logit. ) Madras, Mozannar and Charusaie do not explicitly teach processing the model input to generate a shared embedding of the model input and for each classification category, applying a respective set of logit weights for the classification category to the shared embedding to generate the classification logit for the classification category. He teaches “Deep residual learning for image recognition (title)” comprising: “process the model input to generate a shared embedding of the model input” (He, Section 3.3, “We perform downsampling directly by convolutional layers […] The network ends with a global average pooling layer and a 1000-way fully-connected layer with softmax.”; Examiner’s note: Performing downsampling teaches processing the model input and fully-connected layer implicitly teaches a shared embedding .) “for each classification category, apply a respective set of logit weights for the classification category to the shared embedding to generate the classification logit for the classification category (He, FIG. 2 & Section 3.2, “ PNG media_image16.png 28 269 media_image16.png Greyscale ”; Examiner’s note: x is an input vector, y is an output vector, i indexes each classification category and W i teaches a respective set of logit weights for the classification category . A respective set of logit weights for the classification category (W i ) is applied to the shared embedding (F(x, {W i })) to generate the classification logit (y) for the classification category .) It would have been obvious to one having ordinary skill in the art before the effective filing date of the invention was made to modify the combination of Madras, Charusaie and Mozannar by generating a shared embedding of the model input and applying a set of logit weights to the shared embedding as taught in He to the classifier machine learning model in Madras in order to “overcome the optimization difficulty and demonstrate accuracy gains when the depth increases” (He, Section 4.2). Regarding Claim 17 , The combination of Madras, Mozannar, Charusaie and He teaches: “The method of claim 16, wherein training the classifier on the second training data set comprises” (preamble) training the deferral logit weights ( Examiner’s note: training the deferral logit weights is taught by Mozannar – see supra claim 16.) while holding the classification logit weights for the classification categories fixed (Charusaie, Section 4.1, “in the second step we learn the rejector r to minimize the joint loss (2) with the now fixed classifier”; Examiner’s note: fixed classifier in the second step when training the rejector teaches holding the classification logit weights for the classification categories fixed .) The reasons of obviousness have been noted in the rejection of Claim 16 above and applicable herein . 07-21-aia AIA Claim s 21 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Madras in view of Mozannar and further in view of Charusaie as applied in claim 11, in view of Verma . Regarding Claim 21 , The combination of Madras, Mozannar and Charusaie teaches: The method of claim 18, wherein the deferral confidence score is generated by the output of the rejector function (Charusaie, Equation (9), Examiner’s note: equation (9) teaches the deferral confidence score generated by the output of the rejector function. r(x) = 1 is the deferral confidence score generated by the output of the rejector function. PNG media_image10.png 65 329 media_image10.png Greyscale ) Madras, Mozannar and Charusaie do not explicitly teach applying a proper composite loss to a negative of the output of the rejector function. Verma teaches: applying a proper composite loss to a negative of the output (Verma, Section A.3., “if f : X → R is learnt by minimizing the logistic loss, then PNG media_image17.png 16 19 media_image17.png Greyscale = 1 / 1+exp( − f(x)) acts as a class probability estimate. Furthermore, strictly proper composite losses are classification calibrated (Reid & Williamson, 2010). Thus, based on the definition of calibration of the binary surrogate losses (Section A.2), we can write the final predictor learnt by minimizing the logistic loss as: PNG media_image18.png 32 226 media_image18.png Greyscale ”; Examiner’s note: after receiving a classification probability, a proper composite loss to a negative of the output is applied to minimize the logistic loss .) It would have been obvious to one having ordinary skill in the art before the effective filing date of the invention was made to modify the combination of Madras, Mozannar and Charusaie by applying a proper composite loss to a negative of the output as taught in Verma to the classifier machine learning model in Madras in order to “minimize[] the logistic loss” (Verma, Section A.3). Regarding Claim 22 , The combination of Madras, Mozannar, Charusaie and Verma teaches: The method of claim 18, wherein the non-deferral confidence score is generated by the output of the rejector function (Charusaie, Equation (9), Examiner’s note: Equation (9) teaches the non-deferral confidence score generated by the output of the rejector function. r(x) = 0 is the non-deferral confidence score generated by the output of the rejector function. PNG media_image10.png 65 329 media_image10.png Greyscale ) applying a proper composite loss to the output (Verma, Section A.3., “if f : X → R is learnt by minimizing the logistic loss, then PNG media_image17.png 16 19 media_image17.png Greyscale = 1 / 1+exp( − f(x)) acts as a class probability estimate. Furthermore, strictly proper composite losses are classification calibrated (Reid & Williamson, 2010). Thus, based on the definition of calibration of the binary surrogate losses (Section A.2), we can write the final predictor learnt by minimizing the logistic loss as: PNG media_image18.png 32 226 media_image18.png Greyscale ”; Examiner’s note: after receiving a classification probability, a proper composite loss to the output is applied to minimize the logistic loss .) The reasons of obviousness have been noted in the rejection of Claim 21 above and applicable herein. Claim 23 recites substantially the same limitations as Claim 1, in the form of a system, therefore it is rejected under the same rationale . Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Geifman et al. teaches a deep neural network with an integrated reject option . Any inquiry concerning this communication or earlier communications from the examiner should be directed to YONG D RHO whose telephone number is (571)270-0194. The examiner can normally be reached 8am-5pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at 571-270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /YONG DOO RHO/Examiner, Art Unit 2147 /VIKER A LAMARDO/Supervisory Patent Examiner, Art Unit 2147 Application/Control Number: 18/488,951 Page 2 Art Unit: 2147 Application/Control Number: 18/488,951 Page 3 Art Unit: 2147 Application/Control Number: 18/488,951 Page 4 Art Unit: 2147 Application/Control Number: 18/488,951 Page 5 Art Unit: 2147 Application/Control Number: 18/488,951 Page 6 Art Unit: 2147 Application/Control Number: 18/488,951 Page 7 Art Unit: 2147 Application/Control Number: 18/488,951 Page 8 Art Unit: 2147 Application/Control Number: 18/488,951 Page 9 Art Unit: 2147 Application/Control Number: 18/488,951 Page 10 Art Unit: 2147 Application/Control Number: 18/488,951 Page 11 Art Unit: 2147 Application/Control Number: 18/488,951 Page 12 Art Unit: 2147 Application/Control Number: 18/488,951 Page 13 Art Unit: 2147 Application/Control Number: 18/488,951 Page 14 Art Unit: 2147 Application/Control Number: 18/488,951 Page 15 Art Unit: 2147 Application/Control Number: 18/488,951 Page 16 Art Unit: 2147 Application/Control Number: 18/488,951 Page 17 Art Unit: 2147 Application/Control Number: 18/488,951 Page 18 Art Unit: 2147 Application/Control Number: 18/488,951 Page 19 Art Unit: 2147 Application/Control Number: 18/488,951 Page 20 Art Unit: 2147 Application/Control Number: 18/488,951 Page 21 Art Unit: 2147 Application/Control Number: 18/488,951 Page 22 Art Unit: 2147 Application/Control Number: 18/488,951 Page 23 Art Unit: 2147 Application/Control Number: 18/488,951 Page 24 Art Unit: 2147 Application/Control Number: 18/488,951 Page 26 Art Unit: 2147 Application/Control Number: 18/488,951 Page 27 Art Unit: 2147 Application/Control Number: 18/488,951 Page 28 Art Unit: 2147 Application/Control Number: 18/488,951 Page 29 Art Unit: 2147 Application/Control Number: 18/488,951 Page 30 Art Unit: 2147 Application/Control Number: 18/488,951 Page 31 Art Unit: 2147 Application/Control Number: 18/488,951 Page 34 Art Unit: 2147 Application/Control Number: 18/488,951 Page 35 Art Unit: 2147
Prosecution Insights