Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
Acknowledgment is made of the Information Disclosure Statement dated 9/21/2023 and 10/17/2023. All of the cited references have been considered.
Drawings
The drawings have been received on 9/21/2023. These drawings are accepted.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1,
Claim 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 1 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“determining, [by a machine learning model and] based on the input data, a feature map that represents learned features present in the input data;”
“for each respective inlier class of a plurality of inlier classes, determining, [by the machine learning model and] based on the feature map, a corresponding inlier score indicative of a probability that the input data belongs to the respective inlier class, [wherein the machine learning model has been trained using at least a threshold number of training samples for each respective inlier class;]”
“for each respective outlier class of a plurality of outlier classes, determining, [by the machine learning model and] based on the feature map, a corresponding outlier score indicative of a probability that the input data belongs to the respective outlier class, [wherein the machine learning model has been trained using fewer than the threshold number of training samples for each respective outlier class; and]”
“determining, based on (i) the corresponding inlier score for each respective inlier class and (ii) the corresponding outlier score for each respective outlier class, whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., determining). The above limitations in the context of this claim encompass, inter alia, determining a feature map, determining a corresponding outlier score, determining whether the input data corresponds (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“[for each respective inlier class of a plurality of inlier classes, determining,] by the machine learning model and [based on the feature map, a corresponding inlier score indicative of a probability that the input data belongs to the respective inlier class,] wherein the machine learning model has been trained using at least a threshold number of training samples for each respective inlier class;”
“[for each respective outlier class of a plurality of outlier classes, determining,] by the machine learning model and [based on the feature map, a corresponding outlier score indicative of a probability that the input data belongs to the respective outlier class,] wherein the machine learning model has been trained using fewer than the threshold number of training samples for each respective outlier class; and”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a machine-learning based model (e.g., by using these elements as tools).
The limitations
“obtaining input data;”
As drafted, amount to insignificant extra-solution activities, which do not integrate a judicial exception into a practical application. For example, the additional elements of "obtaining input data" amount to mere data gathering and data storage, respectively, which are insignificant extra-solution activities that do not integrate a judicial exception into a practical application. See MPEP 2106.05(g).
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“[for each respective inlier class of a plurality of inlier classes, determining,] by the machine learning model and [based on the feature map, a corresponding inlier score indicative of a probability that the input data belongs to the respective inlier class,] wherein the machine learning model has been trained using at least a threshold number of training samples for each respective inlier class;”
“[for each respective outlier class of a plurality of outlier classes, determining,] by the machine learning model and [based on the feature map, a corresponding outlier score indicative of a probability that the input data belongs to the respective outlier class,] wherein the machine learning model has been trained using fewer than the threshold number of training samples for each respective outlier class; and”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a machine-learning based model (e.g., by using these elements as tools).
As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are insignificant extra-solution activities or mere instructions to apply an exception. (i.e., the additional element describes a unit for applying the abstract ideas). Insignificant extra-solution activities and mere instructions to apply an exception cannot provide an inventive concept. Moreover, receiving, communicating, and storing data are insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP 2106.05(d)(II) ("The courts have recognized the following computer functions as well-understood, routine, and conventional functions ... i. Receiving or transmitting data over a network) (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)).
The claim is not patent eligible.
Regarding Claim 2,
Claim 2 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 2 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“[when the input data belongs to a class that is not part of the plurality of outlier classes and the plurality of inlier classes,] [the machine learning model is configured to] determine corresponding inlier scores and corresponding outlier scores indicating that the input data corresponds to the plurality of outlier classes.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., determining). The above limitations in the context of this claim encompass, inter alia, determine corresponding inlier scores and corresponding outlier scores (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“[when the input data belongs to a class that is not part of the plurality of outlier classes and the plurality of inlier classes,] the machine learning model is configured to [determine corresponding inlier scores and corresponding outlier scores indicating that the input data corresponds to the plurality of outlier classes.]”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a machine-learning based model (e.g., by using these elements as tools).
The limitations:
“when the input data belongs to a class that is not part of the plurality of outlier classes and the plurality of inlier classes, [the machine learning model is configured to] [determine corresponding inlier scores and corresponding outlier scores indicating that the input data corresponds to the plurality of outlier classes.]”
As drafted, amount to insignificant extra-solution activities, which do not integrate a judicial exception into a practical application. For example, the additional elements of "obtaining input data" amount to mere data gathering and data storage, respectively, which are insignificant extra-solution activities that do not integrate a judicial exception into a practical application. See MPEP 2106.05(g).
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“[when the input data belongs to a class that is not part of the plurality of outlier classes and the plurality of inlier classes,] the machine learning model is configured to [determine corresponding inlier scores and corresponding outlier scores indicating that the input data corresponds to the plurality of outlier classes.]”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a machine-learning based model (e.g., by using these elements as tools).
As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are insignificant extra-solution activities or mere instructions to apply an exception. (i.e., the additional element describes a unit for applying the abstract ideas). Insignificant extra-solution activities and mere instructions to apply an exception cannot provide an inventive concept. Moreover, receiving, communicating, and storing data are insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP 2106.05(d)(II) ("The courts have recognized the following computer functions as well-understood, routine, and conventional functions ... i. Receiving or transmitting data over a network") (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)).
The claim is not patent eligible.
Regarding Claim 3,
Claim 3 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 3 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes comprises:
determining (i) a first sum of the corresponding inlier score for each respective inlier class and (ii) a second sum of the corresponding outlier score for each respective outlier class;”
“determining a disparity between the second sum and the first sum; and”
“based on determining the disparity, determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., determining). The above limitations in the context of this claim encompass, inter alia, determining a first and second sum, determining a disparity, determining whether the input data corresponds (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 1.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 4,
Claim 4 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 4 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“determining the disparity between the second sum and the first sum comprises determining whether the first sum exceeds the second sum or the second sum exceeds the first sum, and”
“determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes comprises:
based on determining that the first sum exceeds the second sum, determining that the input data corresponds to the plurality of inlier classes; or”
“based on determining that the second sum exceeds the first sum, determining that the input data corresponds to the plurality of outlier classes.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., determining). The above limitations in the context of this claim encompass, inter alia, determining the disparity, determining whether the input data corresponds (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 1.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 5,
Claim 5 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 5 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“determining that the input data corresponds to the plurality of inlier classes;
based on determining that the input data corresponds to the plurality of inlier classes, determining, based on the corresponding inlier score for each respective inlier class, a particular inlier class to which the input data belongs; and”
“generating an indication of the particular inlier class to which the input data belongs.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., determining, generating). The above limitations in the context of this claim encompass, inter alia, determining that the input data corresponds, generating an indication (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 1.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 6,
Claim 6 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 6 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes comprises:
determining that the input data corresponds to the plurality of outlier classes; and”
“based on determining that the input data corresponds to the plurality of outlier classes, generating an indication that the machine learning model is untrained to classify the input data with at least a threshold accuracy.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., determining, generating). The above limitations in the context of this claim encompass, inter alia, determining that the input data corresponds, generating an indication (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 1.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 7,
Claim 7 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 7 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: Please see the corresponding analysis of Claim1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“wherein the input data comprises one or more of: image data, audio data, waveform data, point cloud data, or text data.”
As drafted, amount to insignificant extra-solution activities, which do not integrate a judicial exception into a practical application. For example, the additional elements of "obtaining input data" amount to mere data gathering and data storage, respectively, which are insignificant extra-solution activities that do not integrate a judicial exception into a practical application. See MPEP 2106.05(g).
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are insignificant extra-solution activities or mere instructions to apply an exception. (i.e., the additional element describes a unit for applying the abstract ideas). Insignificant extra-solution activities and mere instructions to apply an exception cannot provide an inventive concept. Moreover, receiving, communicating, and storing data are insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP 2106.05(d)(II) ("The courts have recognized the following computer functions as well-understood, routine, and conventional functions ... i. Receiving or transmitting data over a network") (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)).
The claim is not patent eligible.
Regarding Claim 8,
Claim 8 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 8 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“[wherein the input data comprises a medical image, and] wherein determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes comprises: determining whether the machine learning model is qualified to generate a medical diagnosis based on the medical image, wherein the medical diagnosis comprises a classification of the medical image into a particular inlier class of the plurality of inlier classes.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., determining). The above limitations in the context of this claim encompass, inter alia, determining whether the input data corresponds, determining whether the machine learning model is qualified to generate (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“wherein the input data comprises a medical image, and [wherein determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes comprises: determining whether the machine learning model is qualified to generate a medical diagnosis based on the medical image, wherein the medical diagnosis comprises a classification of the medical image into a particular inlier class of the plurality of inlier classes.]”
As drafted, amount to insignificant extra-solution activities, which do not integrate a judicial exception into a practical application. For example, the additional elements of "obtaining input data" amount to mere data gathering and data storage, respectively, which are insignificant extra-solution activities that do not integrate a judicial exception into a practical application. See MPEP 2106.05(g).
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are insignificant extra-solution activities or mere instructions to apply an exception. (i.e., the additional element describes a unit for applying the abstract ideas). Insignificant extra-solution activities and mere instructions to apply an exception cannot provide an inventive concept. Moreover, receiving, communicating, and storing data are insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP 2106.05(d)(II) ("The courts have recognized the following computer functions as well-understood, routine, and conventional functions ... i. Receiving or transmitting data over a network ... iv. Storing and retrieving information in memory") (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)).
The claim is not patent eligible.
Regarding Claim 9,
Claim 9 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 9 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“[wherein the machine learning model comprises:]
[one or more encoders configured to] generate the feature map by processing the input data;”
“[a plurality of neurons connected to the one or more encoders and configured to] generate, based on the feature map, a vector comprising a plurality of values, wherein each respective neuron of the plurality of neurons comprises a plurality of trainable weights; and”
“[a softmax operator configured to] generate, based on the vector, the corresponding inlier score for each respective inlier class and the corresponding outlier score for each respective outlier class.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., generating). The above limitations in the context of this claim encompass, inter alia, generating the feature map, generating a vector, generating corresponding inlier score (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“wherein the machine learning model comprises:
one or more encoders configured to[ generate the feature map by processing the input data;]”
“a plurality of neurons connected to the one or more encoders and configured to [generate, based on the feature map, a vector comprising a plurality of values, wherein each respective neuron of the plurality of neurons comprises a plurality of trainable weights; and]”
“a softmax operator configured to [generate, based on the vector, the corresponding inlier score for each respective inlier class and the corresponding outlier score for each respective outlier class.]”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a machine-learning based model (e.g., by using these elements as tools).
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“wherein the machine learning model comprises:
one or more encoders configured to[ generate the feature map by processing the input data;]”
“a plurality of neurons connected to the one or more encoders and configured to [generate, based on the feature map, a vector comprising a plurality of values, wherein each respective neuron of the plurality of neurons comprises a plurality of trainable weights; and]”
“a softmax operator configured to [generate, based on the vector, the corresponding inlier score for each respective inlier class and the corresponding outlier score for each respective outlier class.]”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a machine-learning based model (e.g., by using these elements as tools).
The claim is not patent eligible.
Regarding Claim 10,
Claim 10 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 10 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“[wherein the machine learning model comprises an ensemble of a plurality of sub-models, wherein each respective sub-model of the plurality of sub-models comprises: (i) corresponding one or more encoders, (ii) a corresponding plurality of neurons, and (iii) a corresponding softmax operator, wherein each respective sub-model has been trained using a different corresponding training procedure, wherein each respective sub-model is configured] to generate a corresponding set of inlier scores for the plurality of inlier classes and a corresponding set of outlier scores for the plurality of outlier classes, and wherein determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes comprises:”
“determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes based on (i) the corresponding set of inlier scores generated by each respective sub-model and (ii) the corresponding set of outlier scores generated by each respective sub-model.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., generating, determining). The above limitations in the context of this claim encompass, inter alia, generating a corresponding set of inlier scores, determining whether the input data corresponds (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“wherein the machine learning model comprises an ensemble of a plurality of sub-models, wherein each respective sub-model of the plurality of sub-models comprises: (i) corresponding one or more encoders, (ii) a corresponding plurality of neurons, and (iii) a corresponding softmax operator, wherein each respective sub-model has been trained using a different corresponding training procedure, wherein each respective sub-model is configured to [generate a corresponding set of inlier scores for the plurality of inlier classes and a corresponding set of outlier scores for the plurality of outlier classes, and wherein determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes comprises:]”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a machine-learning based model (e.g., by using these elements as tools).
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“wherein the machine learning model comprises an ensemble of a plurality of sub-models, wherein each respective sub-model of the plurality of sub-models comprises: (i) corresponding one or more encoders, (ii) a corresponding plurality of neurons, and (iii) a corresponding softmax operator, wherein each respective sub-model has been trained using a different corresponding training procedure, wherein each respective sub-model is configured to [generate a corresponding set of inlier scores for the plurality of inlier classes and a corresponding set of outlier scores for the plurality of outlier classes, and wherein determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes comprises:]”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a machine-learning based model (e.g., by using these elements as tools).
The claim is not patent eligible.
Regarding Claim 11,
Claim 11 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 11 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: Please see the corresponding analysis of Claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“wherein a first sub-model of the plurality of sub-models has been trained using a contrastive training process, and wherein a second sub-model of the plurality of sub-models has been trained using a transfer learning training process.”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a machine-learning based model (e.g., by using these elements as tools).
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“wherein a first sub-model of the plurality of sub-models has been trained using a contrastive training process, and wherein a second sub-model of the plurality of sub-models has been trained using a transfer learning training process.”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a machine-learning based model (e.g., by using these elements as tools).
The claim is not patent eligible.
Regarding Claim 12,
Claim 12 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 12 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“determining, [by the machine learning model and] based on the training input data, a training feature map that represents learned features present in the training input data;”
“for each respective inlier class of the plurality of inlier classes, determining, [by the machine learning model and] based on the training feature map, a corresponding inlier training score indicative of a probability that the training input data belongs to the respective inlier class;”
“for each respective outlier class of the plurality of outlier classes, determining, by the machine learning model and based on the training feature map, a corresponding outlier training score indicative of a probability that the training input data belongs to the respective outlier class;”
“determining a fine-grained loss value based on a training score of the ground-truth class, wherein the training score is the corresponding inlier training score for an inlier class corresponding to the ground-truth class or the corresponding outlier training score for an outlier class corresponding to the ground-truth class;”
“determining a coarse-grained loss value based on (i) a first training sum of the corresponding inlier training score for each respective inlier class when the ground-truth class is an inlier or (ii) a second training sum of the corresponding outlier training score for each respective outlier class when the ground-truth class is an outlier; and”
“adjusting one or more parameters of the machine learning model based on the fine- grained loss value and the coarse-grained loss value.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., determining, adjusting). The above limitations in the context of this claim encompass, inter alia, determining a training feature map, determining a corresponding inlier training score, determining a corresponding outlier training score, determining a fine-grained loss, determining a coarse-grained loss, adjusting parameters of the machine learning model (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“[determining,] by the machine learning model and [based on the training input data, a training feature map that represents learned features present in the training input data;]”
“[for each respective inlier class of the plurality of inlier classes, determining,] by the machine learning model and [based on the training feature map, a corresponding inlier training score indicative of a probability that the training input data belongs to the respective inlier class;]”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a machine-learning based model (e.g., by using these elements as tools).
The limitations:
“obtaining training input data associated with a ground-truth class;”
As drafted, amount to insignificant extra-solution activities, which do not integrate a judicial exception into a practical application. For example, the additional elements of "obtaining training input data" amount to mere data gathering and data storage, respectively, which are insignificant extra-solution activities that do not integrate a judicial exception into a practical application. See MPEP 2106.05(g).
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“[determining,] by the machine learning model and [based on the training input data, a training feature map that represents learned features present in the training input data;]”
“[for each respective inlier class of the plurality of inlier classes, determining,] by the machine learning model and [based on the training feature map, a corresponding inlier training score indicative of a probability that the training input data belongs to the respective inlier class;]”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a machine-learning based model (e.g., by using these elements as tools).
As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are insignificant extra-solution activities or mere instructions to apply an exception. (i.e., the additional element describes a unit for applying the abstract ideas). Insignificant extra-solution activities and mere instructions to apply an exception cannot provide an inventive concept. Moreover, receiving, communicating, and storing data are insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP 2106.05(d)(II) ("The courts have recognized the following computer functions as well-understood, routine, and conventional functions ... i. Receiving or transmitting data over a network") (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)).
The claim is not patent eligible.
Regarding Claim 13,
Claim 13 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 13 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein determining the fine-grained loss value comprises:
determining a negative logarithm of the training score of the ground-truth class.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., determining). The above limitations in the context of this claim encompass, inter alia, determining the fine-grained loss value (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 1.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 14,
Claim 14 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 14 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein determining the coarse-grained loss value comprises:
determining (i) a negative logarithm of the first training sum when the ground-truth class is an inlier or (ii) a negative logarithm of the second training sum when the ground-truth class is an outlier.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., determining). The above limitations in the context of this claim encompass, inter alia, determining the corase-grained loss value (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 1.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 15,
Claim 15 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 15 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein adjusting the one or more parameters of the machine learning model comprises:
determining a weighted sum of the fine-grained loss value and the coarse-grained loss value; and”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., adjusting, determining). The above limitations in the context of this claim encompass, inter alia, adjusting the one or more parameters, determining a weighted sum (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: Please see the corresponding analysis of Claim 1.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 16,
Claim 16 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 16 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: Please see the corresponding analysis of Claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“wherein the training input data forms part of a training data set that forms a long-tailed distribution of training samples representing more outlier classes than inlier classes.”
As drafted, amount to insignificant extra-solution activities, which do not integrate a judicial exception into a practical application. For example, the additional elements of "obtaining training input data" amount to mere data gathering and data storage, respectively, which are insignificant extra-solution activities that do not integrate a judicial exception into a practical application. See MPEP 2106.05(g).
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are insignificant extra-solution activities or mere instructions to apply an exception. (i.e., the additional element describes a unit for applying the abstract ideas). Insignificant extra-solution activities and mere instructions to apply an exception cannot provide an inventive concept. Moreover, receiving, communicating, and storing data are insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP 2106.05(d)(II) ("The courts have recognized the following computer functions as well-understood, routine, and conventional functions ... i. Receiving or transmitting data over a network") (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)).
The claim is not patent eligible.
Regarding Claim 17,
Claim 17 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 17 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“determining the plurality of inlier classes by identifying, within the training data set, a first plurality of classes each of which is associated with at least the threshold number of training samples; and”
“determining the plurality of outlier classes by identifying, within the training data set, a second plurality of classes each of which is associated with fewer than the threshold number of training samples.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., determining). The above limitations in the context of this claim encompass, inter alia, determining the plurality of inlier classes by identifying, determining the plurality of outlier classes by identifying (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“obtaining a training data set comprising a plurality of training samples, wherein each respective training sample of the plurality of training samples comprises training input data associated with a corresponding ground-truth class;”
As drafted, amount to insignificant extra-solution activities, which do not integrate a judicial exception into a practical application. For example, the additional elements of "obtaining a training data set" amount to mere data gathering and data storage, respectively, which are insignificant extra-solution activities that do not integrate a judicial exception into a practical application. See MPEP 2106.05(g).
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are insignificant extra-solution activities or mere instructions to apply an exception. (i.e., the additional element describes a unit for applying the abstract ideas). Insignificant extra-solution activities and mere instructions to apply an exception cannot provide an inventive concept. Moreover, receiving, communicating, and storing data are insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP 2106.05(d)(II) ("The courts have recognized the following computer functions as well-understood, routine, and conventional functions ... i. Receiving or transmitting data over a network") (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)).
The claim is not patent eligible.
Regarding Claim 18,
Claim 18 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 18 is directed to a method, i.e., a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“partitioning the second plurality of classes into a first set of outlier classes and a second set of outlier classes that is disjoint from the first set of outlier classes;”
“[training the machine learning model based on the first set of outlier classes,] wherein the plurality of outlier classes is equivalent to the first set of outlier classes; and”
“after training the machine learning model based on the first set of outlier classes, evaluating performance of the machine learning model based on the second set of outlier classes, wherein the plurality of outlier classes excludes the second set of outlier classes.”
As drafted, under their broadest reasonable interpretation, cover concepts performed in human mind (including an observation, evaluation, judgement, or opinion, e.g., partitioning, evaluating). The above limitations in the context of this claim encompass, inter alia, partitioning the second plurality of classes, evaluating performance of the machine learning model (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application.
The limitations:
“training the machine learning model [based on the first set of outlier classes, wherein the plurality of outlier classes is equivalent to the first set of outlier classes; and]”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a machine-learning based model (e.g., by using these elements as tools).
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The limitations:
“training the machine learning model [based on the first set of outlier classes, wherein the plurality of outlier classes is equivalent to the first set of outlier classes; and]”
As drafted, are additional elements that amount to no more than mere instructions to apply the exception for the abstract ideas. See MPEP 2106.05(f). Specifically, they amount to mere instructions to apply the exception using a machine-learning based model (e.g., by using these elements as tools).
The claim is not patent eligible.
Regarding Claim 19,
Claim 19 recites a system for performing steps similar of claim 1 and is rejected with the same rationale, mutatis mutandis, in view of the following additional elements, considered individually and as an ordered combination with the additional elements identified above, failing to integrate the abstract idea into a practical application or amount to significantly more than the abstract idea:
“a processor; and”
“a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations comprising:”
This is a recitation of generic computing components to be used in performing the abstract idea, which does not integrate the abstract idea into a practical application or amount to significantly more than the abstract idea. See MPEP 2106.05(f).
Regarding Claim 20,
Claim 20 recites a system for performing steps similar of claim 1 and is rejected with the same rationale, mutatis mutandis, in view of the following additional elements, considered individually and as an ordered combination with the additional elements identified above, failing to integrate the abstract idea into a practical application or amount to significantly more than the abstract idea:
“a non-transitory computer-readable medium having stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations comprising:”
This is a recitation of generic computing components to be used in performing the abstract idea, which does not integrate the abstract idea into a practical application or amount to significantly more than the abstract idea. See MPEP 2106.05(f).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 3, 4, 5, 7, 9, 10, 11, 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al. (Large-Scale Long-Tailed Recognition in an Open World); hereinafter Liu in view of Sharma et al. (Long-Tailed Recognition Using Class-Balanced Experts); hereinafter Sharma and in view of Li et al. (Overcoming Classifier Imbalance for Long-tail Object Detection with Balanced Group Softmax); hereinafter Li
Claim 1 is rejected over Liu, Sharma and Li.
Regarding claim 1, Liu teaches a computer-implemented method comprising:
obtaining input data; (Liu [page 3]: “Let vdirect denote the direct feature extracted from an input image. The final classification accuracy largely depends on the quality of this direct feature.)
determining, by a machine learning model and based on the input data, a feature map that represents learned features present in the input data; (Liu [page 3]: “Consider a convolutional neural network (CNN) with a softmax output layer for classification. The second-to-the last layer can be viewed as the feature and the last layer a linear classifier”)
determining, based on (i) the corresponding inlier score for each respective inlier class and (ii) the corresponding outlier score for each respective outlier class, whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes. (Liu [page 5]: “This helps us understand the detailed characteristics of each method. For the open-set setting, the F-measure is also reported for a balanced treatment of precision and recall following [3]. For determining open classes, the softmax probability threshold is initially set as 0:1, while a more detailed analysis is provided in Sec. 4.3.”)
Liu does not appear to explicitly teach for each respective inlier class of a plurality of inlier classes, determining, by the machine learning model and based on the feature map, a corresponding inlier score indicative of a probability that the input data belongs to the respective inlier class,
However, Sharma teaches for each respective inlier class of a plurality of inlier classes, determining, by the machine learning model and based on the feature map, a corresponding inlier score indicative of a probability that the input data belongs to the respective inlier class, (Sharma [page 1]: “To address this practical challenge, in this work, we focus on the problem of long-tailed recognition, wherein datasets exhibit a natural power-law distribution [32], allowing us to assess model performance on four folds: Manyshot classes (≥ 100 samples), Mediumshot classes (20 ~ 100 samples), Fewshot classes (< 20 samples), and All classes.”)
It would have been obvious before the effective filing date to combine the long-tail distributed data of Liu with the thresholds of Sharma to increase accuracy (Sharma, page 10). Liu and Sharma are analogous art because they both concern separating high and low class samples.
Liu does not appear to explicitly teach for each respective outlier class of a plurality of outlier classes, determining, by the machine learning model and based on the feature map, a corresponding outlier score indicative of a probability that the input data belongs to the respective outlier class, wherein the machine learning model has been trained using fewer than the threshold number of training samples for each respective outlier class; and wherein the machine learning model has been trained using fewer than the threshold number of training samples for each respective outlier class; and
However, Li teaches for each respective outlier class of a plurality of outlier classes, determining, by the machine learning model and based on the feature map, a corresponding outlier score indicative of a probability that the input data belongs to the respective outlier class, wherein the machine learning model has been trained using fewer than the threshold number of training samples for each respective outlier class; and wherein the machine learning model has been trained using fewer than the threshold number of training samples for each respective outlier class; and (Li [page 5]: “During inference, we first generate z with the trained model, and apply softmax in each group using Eqn. (4). Except for G0, all nodes of others are ignored, and probabilities of all categories are ordered by the original category IDs.”; [page 7]: “Re-weighting method suppresses the weights of head classes and lifts weights of the tail classes. For ours, since we decouple the relationships of different group of categories, weights of G1; G2 and G3 are almost at the same level. Though weights of G4 are still smaller, they have been better balanced than the original model.”; Note: Each tail category has its own score and was trained with < threshold samples)
It would have been obvious before the effective filing date to combine the long-tail distributed data of Liu with the reweighting of Li to improve performance (Li [page 6]). Liu and Li are analogous art because they both concern long tail distribution.
Claim 2 is rejected over Liu, Sharma and Li with the incorporation of claim 1.
Regarding claim 2, Liu teaches wherein, when the input data belongs to a class that is not part of the plurality of outlier classes and the plurality of inlier classes, the machine learning model is configured to determine corresponding inlier scores and corresponding outlier scores indicating that the input data corresponds to the plurality of outlier classes. (Liu [page 1]: “A practical system shall be able to classify among a few common and many rare categories, to generalize the concept of a single category from only a few known instances, and to acknowledge novelty upon an instance of a never seen category. We define OLTR as learning from long-tail and open-end distributed data and evaluating the classification accuracy over a balanced test set which include head, tail, and open classes in a continuous spectrum (Fig. 1).”)
Claim 3 is rejected over Liu, Sharma and Li with the incorporation of claim 1.
Regarding claim 3, Liu does not appear to explicitly teach wherein determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes comprises: determining (i) a first sum of the corresponding inlier score for each respective inlier class and (ii) a second sum of the corresponding outlier score for each respective outlier class;
determining a disparity between the second sum and the first sum; and
based on determining the disparity, determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes.
However, Sharma teaches wherein determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes comprises: determining (i) a first sum of the corresponding inlier score for each respective inlier class and (ii) a second sum of the corresponding outlier score for each respective outlier class;
determining a disparity between the second sum and the first sum; and
based on determining the disparity, determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes. (Sharma [page 6]: “Soft-voting. We find the full posterior by summing up the partial posteriors directly and normalising the sum to 1 … Here g(.) is a function that converts an expert's partial posterior into a full posterior. Since experts are trained with a reject class, g(.) averages reject class probability score across out-of-distribution classes”)
Claim 4 is rejected over Liu, Sharma and Li with the incorporation of claim 1.
Regarding claim 4, Liu teaches determining the disparity between the second sum and the first sum comprises determining whether the first sum exceeds the second sum or the second sum exceeds the first sum, and
determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes comprises:
based on determining that the first sum exceeds the second sum, determining that the input data corresponds to the plurality of inlier classes; or based on determining that the second sum exceeds the first sum, determining that the input data corresponds to the plurality of outlier classes. (Liu [page 5]: “We evaluate the performance of each method under both the closed-set (test set contains no unknown classes) and open-set (test set contains unknown classes) settings to highlight their differences. Under each setting, besides the overall top-1 classification accuracy [15] over all classes, we also calculate the accuracy of three disjoint subsets: many-shot classes (classes each with over training 100 samples), medium-shot classes (classes each with 20 ~100 training samples) and few-shot classes (classes under 20 training samples). This helps us understand the detailed characteristics of each method. For the open-set setting, the F-measure is also reported for a balanced treatment of precision and recall following [3]. For determining open classes, the softmax probability threshold is initially set as 0.1”)
Claim 5 is rejected over Liu, Sharma and Li with the incorporation of claim 1.
Regarding claim 5, Liu teaches wherein determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes comprises:
based on determining that the input data corresponds to the plurality of inlier classes, determining, based on the corresponding inlier score for each respective inlier class, a particular inlier class to which the input data belongs; and
generating an indication of the particular inlier class to which the input data belongs. (Liu [page 5]: “We evaluate the performance of each method under both the closed-set (test set contains no unknown classes) and open-set (test set contains unknown classes) settings to highlight their differences. Under each setting, besides the overall top-1 classification accuracy [15] over all classes, we also calculate the accuracy of three disjoint subsets: many-shot classes (classes each with over training 100 samples), medium-shot classes (classes each with 20 ~100 training samples) and few-shot classes (classes under 20 training samples). This helps us understand the detailed characteristics of each method. For the open-set setting, the F-measure is also reported for a balanced treatment of precision and recall following [3]. For determining open classes, the softmax probability threshold is initially set as 0.1”)
Claim 7 is rejected over Liu, Sharma and Li with the incorporation of claim 1.
Regarding claim 7, Liu teaches wherein the input data comprises one or more of: image data, audio data, waveform data, point cloud data, or text data. (Liu [page 5]: “Datasets. We curate three open long-tailed benchmarks, ImageNet-LT (object-centric), Places-LT (scene-centric), and MS1M-LT (face-centric), respectively.”)
Claim 9 is rejected over Liu, Sharma and Li with the incorporation of claim 1.
Regarding claim 9, Liu does not appear to explicitly teach wherein the machine learning model comprises: one or more encoders configured to generate the feature map by processing the input data;
a plurality of neurons connected to the one or more encoders and configured to generate, based on the feature map, a vector comprising a plurality of values, wherein each respective neuron of the plurality of neurons comprises a plurality of trainable weights; and
a softmax operator configured to generate, based on the vector, the corresponding inlier score for each respective inlier class and the corresponding outlier score for each respective outlier class.
However, Li teaches wherein the machine learning model comprises: one or more encoders configured to generate the feature map by processing the input data; (Li [page 3]: “The backbone network fback takes an image I as input, and generates a feature map F = fback(I). The feature map is then passed to ROI-align [17] or ROI-pooling [12] to produce K proposals with their own feature Fk = ROIAlign(F; bk).”)
a plurality of neurons connected to the one or more encoders and configured to generate, based on the feature map, a vector comprising a plurality of values, wherein each respective neuron of the plurality of neurons comprises a plurality of trainable weights; and (Li [page 3]: “the network losses are reweighted at category level by multiplying different weights on different categories to enlarge the influence of tail-class training samples [6, 2, 19] or at instance level by multiplying different weights on different training samples for more fine-grained control”)
a softmax operator configured to generate, based on the vector, the corresponding inlier score for each respective inlier class and the corresponding outlier score for each respective outlier class. (See equation (2) on page 3 of Li to see the softmax)
It would have been obvious before the effective filing date to combine the long-tail distributed data of Liu with the reweighting of Li to improve performance (Li [page 6]). Liu and Li are analogous art because they both concern long tail distribution.
Claim 10 is rejected over Liu, Sharma and Li with the incorporation of claim 1.
Regarding claim 10, Liu does not appear to explicitly teach wherein the machine learning model comprises an ensemble of a plurality of sub-models, wherein each respective sub-model of the plurality of sub-models comprises: (i) corresponding one or more encoders, (ii) a corresponding plurality of neurons, and (iii) a corresponding softmax operator, wherein each respective sub-model has been trained using a different corresponding
training procedure, wherein each respective sub-model is configured to generate a corresponding set of inlier scores for the plurality of inlier classes and a corresponding set of outlier scores for the plurality of outlier classes, and wherein determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes comprises:
determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes based on (i) the corresponding set of inlier scores generated by each respective sub-model and (ii) the corresponding set of outlier scores generated by each respective sub-model.
However, Sharma teaches wherein the machine learning model comprises an ensemble of a plurality of sub-models, wherein each respective sub-model of the plurality of sub-models comprises: (i) corresponding one or more encoders, (ii) a corresponding plurality of neurons, and (iii) a corresponding softmax operator, wherein each respective sub-model has been trained using a different corresponding (Sharma [page 11]: “we take our ensemble of class-balanced experts and plot a confusion matrix, each entry showing the percentage of samples from dataset D that are classified by expert model E . For the preliminary analysis we use Soft-voting for fusing expert posteriors. Fig. 4a shows the result for Places-LT. The plot shows there is significant confusion amongst experts; experts aren't selected optimally for classes to which a test sample belongs.”)
training procedure, wherein each respective sub-model is configured to generate a corresponding set of inlier scores for the plurality of inlier classes and a corresponding set of outlier scores for the plurality of outlier classes, and wherein determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes comprises: (Sharma [page 2]: “Fig. 1: Our pipeline for long-tailed recognition: an ensemble of experts trained on class-balanced subsets of Manyshot, Mediumshot, and Fewshot data. We transfer knowledge from Manyshot to Mediumshot and Fewshot classes by initializing experts with a Baseline model trained on all the data. Expert models classify samples outside their subset as out-of-distribution and output partial posteriors that are fused into a full posterior to obtain the final prediction.”)
determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes based on (i) the corresponding set of inlier scores generated by each respective sub-model and (ii) the corresponding set of outlier scores generated by each respective sub-model. (Sharma [page 12]: “This article presented an ensemble of class-balanced experts framework for long-tailed recognition. Our effective and modular strategy explicitly tackles relative imbalance without resorting to complex models or sophisticated loss objectives. We decompose the imbalanced classification problem into balanced classification problems that are more tractable, and train separate expert models for Manyshot, Mediumshot and Fewshot subsets of the data with a reject class for samples lying outside an expert's class-balanced subset. We scale and shift experts' partial posteriors to jointly calibrate experts' predictions, and our ensemble of class-balanced experts reaches close to state-of-the-art performance on two long-tailed benchmarks.”)
It would have been obvious before the effective filing date to combine the long-tail distributed data of Liu with the thresholds of Sharma to increase accuracy (Sharma, page 10). Liu and Sharma are analogous art because they both concern separating high and low class samples.
Claim 11 is rejected over Liu, Sharma and Li with the incorporation of claim 1.
Regarding claim 11, Liu does not appear to explicitly teach wherein a first sub-model of the plurality of sub-models has been trained using a contrastive training process, and wherein a second sub-model of the plurality of sub-models has been trained using a transfer learning training process.
However, Sharma teaches wherein a first sub-model of the plurality of sub-models has been trained using a contrastive training process, and wherein a second sub-model of the plurality of sub-models has been trained using a transfer learning training process. (Sharma [page 12]: “This article presented an ensemble of class-balanced experts framework for long-tailed recognition. Our effective and modular strategy explicitly tackles relative imbalance without resorting to complex models or sophisticated loss objectives. We decompose the imbalanced classification problem into balanced classification problems that are more tractable, and train separate expert models for Manyshot, Mediumshot and Fewshot subsets of the data with a reject class for samples lying outside an expert's class-balanced subset. We scale and shift experts' partial posteriors to jointly calibrate experts' predictions, and our ensemble of class-balanced experts reaches close to state-of-the-art performance on two long-tailed benchmarks.”)
Claim 19 is rejected over Liu, Sharma and Li.
Regarding claim 19, Liu teaches a system comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations comprising: (Liu [page 3]: “Let vdirect denote the direct feature extracted from an input image. The final classification accuracy largely depends on the quality of this direct feature.)
The remainder of claim 19 is claim 1 in the form of a system and is rejected for the same reasons as claim 1 stated above.
Claim 20 is rejected over Liu, Sharma and Li.
Regarding claim 20, Liu teaches a non-transitory computer-readable medium having stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations comprising: obtaining input data; (Liu [page 3]: “Let vdirect denote the direct feature extracted from an input image. The final classification accuracy largely depends on the quality of this direct feature.)
The remainder of claim 20 is claim 1 in the form of a non-transitory computer-readable medium and is rejected for the same reasons as claim 1 stated above.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Liu, Sharma and Li in view of Thulasidasan et al. (A SIMPLE AND EFFECTIVE BASELINE FOR OUT-OF DISTRIBUTION DETECTION USING AN ABSTENTION CLASS); hereinafter Thulasidasan
Claim 6 is rejected over Liu, Sharma, Li and Thulasidasan with the incorporation of claim 1.
Regarding claim 6, Liu does not appear to explicitly teach wherein determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes comprises: determining that the input data corresponds to the plurality of outlier classes; and
based on determining that the input data corresponds to the plurality of outlier classes, generating an indication that the machine learning model is untrained to classify the input data with at least a threshold accuracy.
However, Thulasidasan teaches wherein determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes comprises: determining that the input data corresponds to the plurality of outlier classes; and
based on determining that the input data corresponds to the plurality of outlier classes, generating an indication that the machine learning model is untrained to classify the input data with at least a threshold accuracy. (Thulasidasan [page 3]: “Our approach uses a DNN trained with an extra abstention class for detecting out-of-distribution and novel samples; from here on, we will refer to this as the deep abstaining classifier (DAC). We augment our training set of in-distribution samples (Din) with an auxiliary dataset of known out-of-distribution samples (~D out), that are known to be mostly disjoint from the main training set (we will use Dout to
denote unknown out-of-distribution samples that we use for testing). We assign the training label of K + 1 to all the outlier samples in ~D out (where K is the number of known classes) and train with cross-entropy; the minimization problem then becomes:”; Note: see the logarithm in equation (1))
It would have been obvious before the effective filing date to combine the long-tail distributed data of Liu with the logarithm of Thulasidasan to improve predictive uncertainty (Thulasidasan, page 5). Liu and Thulasidasan are analogous art because they both determine out of distribution data.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Liu, Sharma and Li in view of Murphy et al. (FRODO: Free rejection of out-of-distribution samples: application to chest x-ray analysis); hereinafter Murphy
Claim 8 is rejected over Liu, Sharma, Li and Murphy with the incorporation of claim 1.
Regarding claim 8, Liu dos not appear to explicitly teach wherein the input data comprises a medical image, and wherein determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes comprises: determining whether the machine learning model is qualified to generate a medical diagnosis based on the medical image, wherein the medical diagnosis comprises a classification of the medical image into a particular inlier class of the plurality of inlier classes.
However, Murphy teaches wherein the input data comprises a medical image, and wherein determining whether the input data corresponds to the plurality of inlier classes or to the plurality of outlier classes comprises: (Murphy [page 1]: “trained with frontal chest x-rays
(CXR) to demonstrate that this metric can be used to detect non-CXR and lateral CXR
(i.e. OOD samples) successfully. We compare this method to a baseline OOD detection
method (Hendrycks and Gimpel, 2017). We show that our method outperforms the baseline
by a significant margin.”)
determining whether the machine learning model is qualified to generate a medical diagnosis based on the medical image, wherein the medical diagnosis comprises a classification of the medical image into a particular inlier class of the plurality of inlier classes. (Murphy [page 3]: “Rejection of OOD samples is a key step towards moving medical image analysis into the clinic, ensuring that algorithms do not provide meaningless scores and instead reject un-suitable data for manual review. In this study, we have demonstrated FRODO, a method to classify OOD samples obtaining 0:99 AUC.”; and [page 1]: “train a standard ResNet-50 model to detect emphysema.”)
It would have been obvious before the effective filing date to combine the long-tail distributed data of Liu with the rejection of out of distribution samples of Murphy to increase performance (Murphy, page 3). Liu and Murphy are analogous art because they both concept detecting out of distribution inputs.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Liu, Sharma and Li in view of Bertinetto et al. (Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks); hereinafter Bertinetto
Claim 12 is rejected over Liu, Sharma, Li and Bertinetto with the incorporation of claim 1.
Regarding claim 12, Liu does not appear to explicitly teach wherein the machine learning model has been trained using a training process comprising:
obtaining training input data associated with a ground-truth class;
determining, by the machine learning model and based on the training input data, a training feature map that represents learned features present in the training input data;
for each respective inlier class of the plurality of inlier classes, determining, by the machine learning model and based on the training feature map, a corresponding inlier training score indicative of a probability that the training input data belongs to the respective inlier class;
for each respective outlier class of the plurality of outlier classes, determining, by the machine learning model and based on the training feature map, a corresponding outlier training score indicative of a probability that the training input data belongs to the respective outlier class;
determining a fine-grained loss value based on a training score of the ground-truth class, wherein the training score is the corresponding inlier training score for an inlier class corresponding to the ground-truth class or the corresponding outlier training score for an outlier class corresponding to the ground-truth class;
determining a coarse-grained loss value based on (i) a first training sum of the corresponding inlier training score for each respective inlier class when the ground-truth class is an inlier or (ii) a second training sum of the corresponding outlier training score for each respective outlier class when the ground-truth class is an outlier; and
adjusting one or more parameters of the machine learning model based on the fine- grained loss value and the coarse-grained loss value.
However, Bertinetto teaches wherein the machine learning model has been trained using a training process comprising:
obtaining training input data associated with a ground-truth class; (Bertinetto [page 2]: “Consider a training set … which pairs N images … with class labels)
determining, by the machine learning model and based on the training input data, a training feature map that represents learned features present in the training input data; (Bertinetto [page 2]: “A network architecture implements the predictor function”)
for each respective inlier class of the plurality of inlier classes, determining, by the machine learning model and based on the training feature map, a corresponding inlier training score indicative of a probability that the training input data belongs to the respective inlier class; (Bertinetto [page 2]: “is a categorical distribution over classes for each input image and denote the corresponding distribution as p”)
for each respective outlier class of the plurality of outlier classes, determining, by the machine learning model and based on the training feature map, a corresponding outlier training score indicative of a probability that the training input data belongs to the respective outlier class; (Bertinetto [page 4]: “When the hierarchy H is a tree, it corresponds to a unique factorisation of the categorical distribution p(C) over classes in terms of the conditional probabilities along the path connecting each class to the root of the tree. Denoting the path from a leaf node C to the root R as C(0) = C; … ; C(h) = R, the probability of class C can be factorized as”)
determining a fine-grained loss value based on a training score of the ground-truth class, wherein the training score is the corresponding inlier training score for an inlier class corresponding to the ground-truth class or the corresponding outlier training score for an outlier class corresponding to the ground-truth class; (Bertinetto [page 4]: “A direct way to incorporate hierarchical information in the loss is to hierarchically factorise the output of the classifier according to Eqn. 2 and define the total loss as the reweighted sum of the cross-entropies of the conditional probabilities. This leads us to define the hierarchical crossentropy (HXE) as”; Note: See Eq (4))
determining a coarse-grained loss value based on (i) a first training sum of the corresponding inlier training score for each respective inlier class when the ground-truth class is an inlier or (ii) a second training sum of the corresponding outlier training score for each respective outlier class when the ground-truth class is an outlier; and (Bertinetto [page 4]: “A direct way to incorporate hierarchical information in the loss is to hierarchically factorise the output of the classifier according to Eqn. 2 and define the total loss as the reweighted sum of the cross-entropies of the conditional probabilities. This leads us to define the hierarchical crossentropy (HXE) as”; Note: See Eq (4))
adjusting one or more parameters of the machine learning model based on the fine- grained loss value and the coarse-grained loss value. (Bertinetto [page 4]: “A direct way to incorporate hierarchical information in the loss is to hierarchically factorise the output of the classifier according to Eqn. 2 and define the total loss as the reweighted sum of the cross-entropies of the conditional probabilities. This leads us to define the hierarchical crossentropy (HXE) as”; Note: See Eq (5) for weighting)
It would have been obvious before the effective filing date to combine the long-tail distributed data of Liu with the hierarchical cross entropy of Bertinetto to improve image classification (Bertinetto, 5. Conclusion). Liu and Bertinetto are analogous art because they both concern image classification.
Claims 13, 14, 15, 16, 17 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Liu, Sharma, Li, Bertinetto and Thulasidasan
Claim 13 is rejected over Liu, Sharma, Li, Bertinetto and Thulasidasan with the incorporation of claim 1.
Regarding claim 13, Liu does not appear to explicitly teach wherein determining the fine-grained loss value comprises: determining a negative logarithm of the training score of the ground-truth class.
However, Thulasidasan teaches wherein determining the fine- grained loss value comprises: determining a negative logarithm of the training score of the ground-truth class. (Thulasidasan [page 3]: “Our approach uses a DNN trained with an extra abstention class for detecting out-of-distribution and novel samples; from here on, we will refer to this as the deep abstaining classifier (DAC). We augment our training set of in-distribution samples (Din) with an auxiliary dataset of known out-of-distribution samples (~D out), that are known to be mostly disjoint from the main training set (we will use Dout to denote unknown out-of-distribution samples that we use for testing). We assign the training label of K + 1 to all the outlier samples in ~Dout (where K is the number of known classes) and train with cross-entropy; the minimization problem then becomes:”; Note: see the logarithm in equation (1))
It would have been obvious before the effective filing date to combine the long-tail distributed data of Liu with the logarithm of Thulasidasan to improve predictive uncertainty (Thulasidasan, page 5). Liu and Thulasidasan are analogous art because they both determine out of distribution data.
Claim 14 is rejected over Liu, Sharma, Li, Bertinetto and Thulasidasan with the incorporation of claim 1.
Regarding claim 14, Liu does not appear to explicitly teach determining (i) a negative logarithm of the first training sum when the ground-truth class is an inlier or (ii) a negative logarithm of the second training sum when the ground-truth class is an outlier.
However, Bertinetto teaches determining (i) a negative logarithm of the first training sum when the ground-truth class is an inlier or (ii) a negative logarithm of the second training sum when the ground-truth class is an outlier. (Bertinetto [page 4]: “where Leaves(C) denotes the set of leaf nodes of the subtree starting at node C.”; Note: See Eq (3) for the ratio and (4) for the log.
It would have been obvious before the effective filing date to combine the long-tail distributed data of Liu with the hierarchical cross entropy of Bertinetto to improve image classification (Bertinetto, 5. Conclusion). Liu and Bertinetto are analogous art because they both concern image classification.
Claim 15 is rejected over Liu, Sharma, Li, Bertinetto and Thulasidasan with the incorporation of claim 1.
Regarding claim 15, Liu does not appear to explicitly teach wherein adjusting the one or more parameters of the machine learning model comprises: determining a weighted sum of the fine-grained loss value and the coarse-grained loss value; and
adjusting the one or more parameters of the machine learning model based on the weighted sum.
However, Bertinetto teaches wherein adjusting the one or more parameters of the machine learning model comprises: determining a weighted sum of the fine-grained loss value and the coarse-grained loss value; and (Bertinetto [3.1. Hierarchical cross-entropy]: “where λ (C(l)) is the weight associated with the edge node C(l+1) -> C(l),”)
adjusting the one or more parameters of the machine learning model based on the weighted sum. (Bertinetto [4.3. Experimental results]: “In Fig. 3 and 4 we show how it is possible to effectively trade off top-1 error to reduce hierarchical error, by simply adjusting the hyperparameters α and β in Eqn. 5 and 7.”)
It would have been obvious before the effective filing date to combine the long-tail distributed data of Liu with the hierarchical cross entropy of Bertinetto to improve image classification (Bertinetto, 5. Conclusion). Liu and Bertinetto are analogous art because they both concern image classification.
Claim 16 is rejected over Liu, Sharma, Li, Bertinetto and Thulasidasan with the incorporation of claim 1.
Regarding claim 16, Liu does not appear to explicitly teach wherein the training input data forms part of a training data set that forms a long-tailed distribution of training samples representing more outlier classes than inlier classes.
However, Li teaches wherein the training input data forms part of a training data set that forms a long-tailed distribution of training samples representing more outlier classes than inlier classes. (Li [page 3]: “Long-tail classification is attracting increasing attention due to its realistic applications. Current works leverage data re-sampling, cost-sensitive learning, or other techniques. For data re-sampling methods, training samples are either over-sampled (adding copies of training samples for tail classes) [16], undersampled (deleting training samples for head classes) [8], or class-balanced sampled [34, 27], which motivates RFS [15].”)
It would have been obvious before the effective filing date to combine the long-tail distributed data of Liu with the reweighting of Li to improve performance (Li [page 6]). Liu and Li are analogous art because they both concern long tail distribution.
Claim 17 is rejected over Liu, Sharma, Li, Bertinetto and Thulasidasan with the incorporation of claim 1.
Regarding claim 17, Liu does not appear to explicitly teach wherein the training process further comprises: obtaining a training data set comprising a plurality of training samples, wherein each respective training sample of the plurality of training samples comprises training input data associated with a corresponding ground-truth class;
determining the plurality of inlier classes by identifying, within the training data set, a first plurality of classes each of which is associated with at least the threshold number of training samples; and
determining the plurality of outlier classes by identifying, within the training data set, a second plurality of classes each of which is associated with fewer than the threshold number of training samples.
However, Sharma teaches wherein the training process further comprises: obtaining a training data set comprising a plurality of training samples, wherein each respective training sample of the plurality of training samples comprises training input data associated with a corresponding ground-truth class;
determining the plurality of inlier classes by identifying, within the training data set, a first plurality of classes each of which is associated with at least the threshold number of training samples; and
determining the plurality of outlier classes by identifying, within the training data set, a second plurality of classes each of which is associated with fewer than the threshold number of training samples. (Sharma [page 1]: “To address this practical challenge, in this work, we focus on the problem of long-tailed recognition, wherein datasets exhibit a natural power-law distribution [32], allowing us to assess model performance on four folds: Manyshot classes (≥ 100 samples), Mediumshot classes (20 ~ 100 samples), Fewshot classes (< 20 samples), and All classes.”)
It would have been obvious before the effective filing date to combine the long-tail distributed data of Liu with the thresholds of Sharma to increase accuracy (Sharma, page 10). Liu and Sharma are analogous art because they both concern separating high and low class samples.
Claim 18 is rejected over Liu, Sharma, Li, Bertinetto and Thulasidasan with the incorporation of claim 1.
Regarding claim 18, Liu teaches partitioning the second plurality of classes into a first set of outlier classes and a second set of outlier classes that is disjoint from the first set of outlier classes;
training the machine learning model based on the first set of outlier classes, wherein the plurality of outlier classes is equivalent to the first set of outlier classes; and
after training the machine learning model based on the first set of outlier classes, evaluating performance of the machine learning model based on the second set of outlier classes, wherein the plurality of outlier classes excludes the second set of outlier classes. (Liu [page 5]: “We evaluate the performance of each method under both the closed-set (test set contains no unknown classes) and open-set (test set contains unknown classes) settings to highlight their differences. Under each setting, besides the overall top-1 classification accuracy [15] over all classes, we also calculate the accuracy of three disjoint subsets: many-shot classes (classes each with over training 100 samples), medium-shot classes (classes each with 20 ~100 training samples) and few-shot classes (classes under 20 training samples). This helps us understand the detailed characteristics of each method. For the open-set setting, the F-measure is also reported for a balanced treatment of precision and recall following [3]. For determining open classes, the softmax probability threshold is initially set as 0.1,”)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID H TRAN whose telephone number is (703)756-1525. The examiner can normally be reached M-F 9:30 am - 5:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID H TRAN/Examiner, Art Unit 2147
/VIKER A LAMARDO/Supervisory Patent Examiner, Art Unit 2147
Prosecution Insights