Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner’s Note
The Examiner encourages Applicant to schedule an interview to discuss issues related to, for example, the rejections noted below under 35 U.S.C § 112, § 101 and § 103, for moving forward allowance.
Providing supporting paragraph(s) for each limitation of amended/new claim(s) in Remarks is strongly requested for clear and definite claim interpretations by Examiner.
Priority
Acknowledgment is made of applicant's claim for the provisional application filed on 07/30/2021.
Allowable Subject Matter
Claims 17 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) and 35 U.S.C. 101 set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
The following is an examiner’s statement of reasons for allowance:
The prior art of record, Boulanger et al. (US 2021/0133510 A1) teaches that a CNN training process tracks image-related metrics and other informative metrics, and the trained inference CNN may then be tested to generate performance results. The training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates a new training dataset.
Hall et al. (US 2023/0162049 A1) teaches a histogram of the number of images per strictness threshold for test and train sets in normal and pneumonia labeled images. In addition, the system shows a set of histogram plots showing balanced accuracy and cross-entropy, or log loss, for various model architectures, where balanced accuracy is calculated as the average of the class-based accuracy for all classes.
Franc et al. (US 20170316342 A1) teaches thresholds of a data representation of data features where the thresholds specify pre-defined bins (e.g., uniform and equidistant bins). Adjacent bins of the pre-defined bins having substantially similar weights can be reciprocally merged, and the merging results in a number of refined bins.
Coover et al. (US 20210073275 A1) teaches that the number of bin values in the sub-fingerprint can be based on a threshold number of bin values. The number of bin values in the sub-fingerprint can be based on a trained neural network to determine which extrema are most likely to contribute to a match. The example sub-fingerprint includes 20 bin values representative of extrema, and the number of bin values in the sub-fingerprint can be variable.
SHAVIT et al. (US 20170368413 A1) teaches that each attribute possible range may be divided into a bin. A coverage measure to end training may be at least one measurement in each bin, or at least one measurement in certain selected bins or in some number of bins defined as threshold. So the ending criteria may be all or a certain number of the attributes bin coverage.
Yao et al. (US 20180137383 A1) teaches that a fern may be used to perform data partition as in ESR (Explicit Shape Regression). Each fern is a binary decision tree structure with a set of binary tests. Each image is passed down all of the ferns as it traverses through each stage of the regression. The fern is a composition of some number (N) of features and thresholds (N) that divide the features from all of the training samples into some number of bins, such as 2N. Each bin corresponds to a regression output that best matches the new image that is to be identified.
However, the prior art of record, taken either alone or in combination, fails to teach or fairly suggest claim 17 in combination with the remaining features and elements of the claimed invention.
Claim Objections
Claim(s) 5, 14 is/are objected to because of the following informalities.
Claim(s) 5 is/are objected to because of the following informalities: it appears that “the updated machine learning model” (line 2) needs to read “the updated ML model” or something else. Appropriate correction is required.
Claim(s) 14 is/are objected to because of the following informalities: it appears that “generated” (2nd last line) needs to read “generate” or something else. Appropriate correction is required.
Claim(s) 5, 14 each recite(s) limitations that raise issues of indefiniteness as set forth above, and their dependent claims are objected to at least based on their direct and/or indirect dependency from the claims listed above. Appropriate explanation and/or amendment is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim(s) 1 recite(s) the limitation “the measure” (line 16). There is insufficient antecedent basis for this limitation in the claim. It is not clear if it indicates “a measure” (line 9) or not since “a measure” (line 9) is based on “first plurality of inputs” while “the measure” (line 16) is based on “second plurality of inputs”. Thus, it appears it may need to read “a measure”, or something else. For the purposes of examination, “a measure” is used. In addition, claim(s) 5, 10, 19, 20 is/are rejected for the same reason.
Claim(s) 17 recite(s) the limitation “the measures” (line 5). There is insufficient antecedent basis for this limitation in the claim. It is not clear if it indicates “measures” (line 4) or not since “measures” (line 4) is based on “the first performance data and the second performance data” while “the measures” (line 5) is based on “across the plurality of bins”. Thus, it appears it may need to read “measures”, or something else. For the purposes of examination, “measures” is used.
Claim(s) 18 recite(s) the limitation “the plurality of bins” (line 5). There is insufficient antecedent basis for this limitation in the claim. It is not clear if it indicates “a plurality of bins” (line 3) or not since “a plurality of bins” (line 3) is based on “first histogram” while “the plurality of bins” (line 5) is based on “second histogram”. Thus, it appears it may need to read “a plurality of bins”, or something else. For the purposes of examination, “a plurality of bins” is used.
Claim(s) 1, 5, 10, 17-20 each recite(s) limitations that raise issues of indefiniteness as set forth above, and their dependent claims are rejected at least based on their direct and/or indirect dependency from the claims listed above. Appropriate explanation and/or amendment is required.
Examiner’s Remarks
“CONTINGENT LIMITATIONS” of MPEP 2111.04.II says “The broadest reasonable interpretation of a method (or process) claim having contingent limitations requires only those steps that must be performed and does not include steps that are not required to be performed because the condition(s) precedent are not met. … The system claim interpretation differs from a method claim interpretation because the claimed structure must be present in the system regardless of whether the condition is met and the function is actually performed.” Thus, based on the “CONTINGENT LIMITATIONS” of MPEP 2111.04.II, in claim 1, “or to generate a supplemental ML model to use with the trained ML model” and “or to generate the supplemental ML model to use with the trained ML model” are not examined. Thus, claims 6-10 are not examined since they are not required to be performed because the condition(s) precedent is/are not met.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“…, comprising:
… to perform:
(A) …, …;
(B) …, …;
(C) determining, using the first representation, the second representation, the first performance data, and the second performance data, whether to update the trained ML model or to generate a supplemental ML model to use with the trained ML model; and
(D) when it is determined to update the trained ML model or to generate the supplemental ML model to use with the trained ML model, ...”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). In particular, the claim recites an additional element(s) (“using at least one computer hardware processor to perform”, “to which the trained ML model was applied”) – using a device and/or a model to process data. The device and the model in each step are recited at a high-level of generality (i.e., as a generic computer performing a generic computer function of processing data) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element(s) (“obtaining information about training data used to …”, “obtaining information about new data …”) – the act of receiving data. The claim is adding an insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g). The act of receiving data is recited at a high-level of generality (i.e., as a generic act of receiving performing a generic act function of receiving data) such that it amounts no more than a mere act to apply the exception using a generic act of receiving. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element(s) (“generate a trained machine learning (ML) model”, “updating the trained ML model to generate an updated ML model or generating the supplemental ML model to use with the trained ML model”). The additional element is recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element (“the training data comprising a first plurality of inputs and a corresponding first plurality of outputs, the information about the training data comprising: a first representation of a first distribution of the first plurality of inputs, and first performance data indicative of a measure of performance of the trained ML model on the first plurality of inputs”, “the new data comprising a second plurality of inputs and a corresponding second plurality of outputs, the information about the new data comprising: a second representation of a second distribution of the second plurality of inputs, and second performance data indicative of the measure of performance of the trained ML model on the second plurality of inputs”). This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, with respect to integration of the abstract idea into a practical application, the additional elements of using a generic computer component to perform each step amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claim is not patent eligible. MPEP 2106.05(f).
As discussed above, the claim recites the additional element(s) of receiving data at a high-level of generality and is adding an insignificant extra-solution activity – see MPEP 2106.05(g). However, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood, routine, and conventional. See MPEP 2106.05(d)(II) – “Receiving or transmitting data over a network” or “Storing and retrieving information in memory”. Accordingly, this additional element does not provide an inventive concept and significantly more than the abstract idea. Thus, the claim is not patent eligible.
The additional elements regarding training are recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not amount to significantly more than the abstract idea. The claim is directed to an abstract idea.
This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not amount to significantly more than the abstract idea. See MPEP 2106.05(h).
Regarding claim 2
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“determining to update the trained ML model”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element(s) (“updating the trained ML model to generate the updated ML model”). The additional element is recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The additional elements regarding training are recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not amount to significantly more than the abstract idea. The claim is directed to an abstract idea.
Regarding claim 3
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1: The claim recites the abstract idea identified above regarding claim 2.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element(s) (“training, using at least some of the new data, a second trained ML model; generating the updated ML model as an ensemble of the trained ML model and the second trained ML model”). The additional element is recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The additional elements regarding training are recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not amount to significantly more than the abstract idea. The claim is directed to an abstract idea.
Regarding claim 4
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“determining weights for the trained ML model and the second trained ML model in the ensemble”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. In particular, the claim does not recite additional elements. Thus, the claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, the claim is not patent eligible.
Regarding claim 5
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“…, …;
…, …;
determining, using the third representation, the fourth representation, the third performance data, and the fourth performance data, whether to further update the updated ML model or to generate a further supplemental ML model to use with the updated ML model; and
when it is determined to further update the updated ML model or to generate the further supplemental ML model to use with the updated ML model, …”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element(s) (“obtaining information about second training data used to …”, “obtaining information about second new data”) – the act of receiving data. The claim is adding an insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g). The act of receiving data is recited at a high-level of generality (i.e., as a generic act of receiving performing a generic act function of receiving data) such that it amounts no more than a mere act to apply the exception using a generic act of receiving. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
The claim recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). In particular, the claim recites an additional element(s) (“to which the updated ML model was applied”) – using a device and/or a model to process data. The device and the model in each step are recited at a high-level of generality (i.e., as a generic computer performing a generic computer function of processing data) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element(s) (“generate the updated machine learning model”, “updating the updated ML model or generating the further supplemental ML model”). The additional element is recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element (“the second training data comprising a third plurality of inputs and a corresponding third plurality of outputs, the information about the second training data comprising: a third representation of a third distribution of the third plurality of inputs, and third performance data indicative of a measure of performance of the updated ML model on the third plurality of inputs”, “the second new data comprising a fourth plurality of inputs and a corresponding fourth plurality of outputs, the information about the second new data comprising: a fourth representation of a fourth distribution of the fourth plurality of inputs, and fourth performance data indicative of the measure of performance of the updated ML model on the fourth plurality of inputs”). This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, the claim recites the additional element(s) of receiving data at a high-level of generality and is adding an insignificant extra-solution activity – see MPEP 2106.05(g). However, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood, routine, and conventional. See MPEP 2106.05(d)(II) – “Receiving or transmitting data over a network” or “Storing and retrieving information in memory”. Accordingly, this additional element does not provide an inventive concept and significantly more than the abstract idea. Thus, the claim is not patent eligible.
As discussed above, with respect to integration of the abstract idea into a practical application, the additional elements of using a generic computer component to perform each step amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claim is not patent eligible. MPEP 2106.05(f).
The additional elements regarding training are recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not amount to significantly more than the abstract idea. The claim is directed to an abstract idea.
This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not amount to significantly more than the abstract idea. See MPEP 2106.05(h).
Regarding claim 6
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“determining to generate the supplemental ML model to use with the trained ML model”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element(s) (“generating the supplemental ML model”). The additional element is recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The additional elements regarding training are recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not amount to significantly more than the abstract idea. The claim is directed to an abstract idea.
Regarding claim 7
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“…;
associating the trained ML model with a first portion of an input data domain; and
associating the supplemental ML model with a second portion of the input data domain”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element(s) (“training, using at least some of the new data, the supplemental ML model”). The additional element is recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The additional elements regarding training are recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not amount to significantly more than the abstract idea. The claim is directed to an abstract idea.
Regarding claim 8
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“…;
determining whether the new input data is in the first portion of the input data domain or in the second portion of the input data domain;
when it is determined that the new input data is in the first portion of the input data domain, … to obtain a first corresponding output; and
when it is determined that the new input data is in the second portion of the input data domain, … to obtain a second corresponding output”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element(s) (“obtaining new input data”) – the act of receiving data. The claim is adding an insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g). The act of receiving data is recited at a high-level of generality (i.e., as a generic act of receiving performing a generic act function of receiving data) such that it amounts no more than a mere act to apply the exception using a generic act of receiving. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element(s) (“providing the new input data as input to the trained ML model”, “providing the new input data as input to the supplemental ML model”) – the act of providing (i.e. inputting) data. The claim is adding an insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g). The act of inputting data is recited at a high-level of generality (i.e., as a generic act of inputting performing a generic act function of inputting data) such that it amounts no more than a mere act to apply the exception using a generic act of inputting. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, the claim recites the additional element(s) of receiving data at a high-level of generality and is adding an insignificant extra-solution activity – see MPEP 2106.05(g). However, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood, routine, and conventional. See MPEP 2106.05(d)(II) – “Receiving or transmitting data over a network” or “Storing and retrieving information in memory”. Accordingly, this additional element does not provide an inventive concept and significantly more than the abstract idea. Thus, the claim is not patent eligible.
As discussed above, the claim recites the additional element(s) of inputting data at a high-level of generality and is adding an insignificant extra-solution activity – see MPEP 2106.05(g) – “Mere Data Gathering”. However, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood, routine, and conventional. See MPEP 2106.05(d)(II) – “Receiving or transmitting data over a network” or “Storing and retrieving information in memory”. Accordingly, this additional element does not provide an inventive concept and significantly more than the abstract idea. Thus, the claim is not patent eligible.
Regarding claim 9
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“monitoring performance of the trained ML model and the supplemental ML model; and
adjusting portions of the input data domain associated with the trained ML model and the supplemental ML model based on results of the monitoring,
wherein adjusting the portions comprises adjusting a boundary between the first portion of the input data domain and the second portion of the input data domain thereby changing the first portion and the second portion of the input data domain”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. In particular, the claim does not recite additional elements. Thus, the claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, the claim is not patent eligible.
Regarding claim 10
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“…, …;
…, …;
determining, using the third representation, the fourth representation, the third performance data, and the fourth performance data, whether to further update the supplemental ML model or to generate a second supplemental ML model to use with the trained ML model and the supplemental ML model; and
when it is determined to further update the supplemental ML model or to generate the second supplemental ML model to use with the trained ML model and the supplemental ML model, …”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element(s) (“obtaining information about second training data used to”, “obtaining information about second new data”) – the act of receiving data. The claim is adding an insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g). The act of receiving data is recited at a high-level of generality (i.e., as a generic act of receiving performing a generic act function of receiving data) such that it amounts no more than a mere act to apply the exception using a generic act of receiving. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element(s) (“generate the supplemental ML model”, “updating the supplemental ML model or generating the second supplemental ML model”). The additional element is recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
The claim recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). In particular, the claim recites an additional element(s) (“to which the supplemental ML model was applied”) – using a device and/or a model to process data. The device and the model in each step are recited at a high-level of generality (i.e., as a generic computer performing a generic computer function of processing data) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element (“the second training data comprising a third plurality of inputs and a corresponding third plurality of outputs, the information about the second training data comprising: a third representation of a third distribution of the third plurality of inputs, and third performance data indicative of a measure of performance of the supplemental ML model on the third plurality of inputs”, “the second new data comprising a fourth plurality of inputs and a corresponding fourth plurality of outputs, the information about the second new data comprising: a fourth representation of a fourth distribution of the fourth plurality of inputs, and fourth performance data indicative of the measure of performance of the supplemental ML model on the fourth plurality of inputs”). This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, the claim recites the additional element(s) of receiving data at a high-level of generality and is adding an insignificant extra-solution activity – see MPEP 2106.05(g). However, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood, routine, and conventional. See MPEP 2106.05(d)(II) – “Receiving or transmitting data over a network” or “Storing and retrieving information in memory”. Accordingly, this additional element does not provide an inventive concept and significantly more than the abstract idea. Thus, the claim is not patent eligible.
The additional elements regarding training are recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not amount to significantly more than the abstract idea. The claim is directed to an abstract idea.
As discussed above, with respect to integration of the abstract idea into a practical application, the additional elements of using a generic computer component to perform each step amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claim is not patent eligible. MPEP 2106.05(f).
This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not amount to significantly more than the abstract idea. See MPEP 2106.05(h).
Regarding claim 11
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1: The claim recites the abstract idea identified above regarding claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element (“wherein the first representation of the first distribution of the first plurality of inputs comprises a histogram having a plurality of bins and a plurality of counts corresponding to the plurality of bins, each of the plurality of counts indicating how many of the first plurality of inputs fall into a respective bin in the plurality of bins”). This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not amount to significantly more than the abstract idea. See MPEP 2106.05(h).
Regarding claim 12
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1: The claim recites the abstract idea identified above regarding claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element (“wherein the first performance data indicative of the measure of performance of the trained ML model on the first plurality of inputs comprises: for each bin of at least some of the plurality of bins, a measure of average error by the trained ML model when applied to inputs, among the first plurality of inputs, that fall in the bin”). This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not amount to significantly more than the abstract idea. See MPEP 2106.05(h).
Regarding claim 13
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1: The claim recites the abstract idea identified above regarding claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element(s) (“prior to obtaining the information about the new data”) – the act of receiving data. The claim is adding an insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g). The act of receiving data is recited at a high-level of generality (i.e., as a generic act of receiving performing a generic act function of receiving data) such that it amounts no more than a mere act to apply the exception using a generic act of receiving. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element(s) (“applying the trained ML model to the second plurality of inputs to obtain the second plurality of outputs”). The additional element is recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, the claim recites the additional element(s) of receiving data at a high-level of generality and is adding an insignificant extra-solution activity – see MPEP 2106.05(g). However, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood, routine, and conventional. See MPEP 2106.05(d)(II) – “Receiving or transmitting data over a network” or “Storing and retrieving information in memory”. Accordingly, this additional element does not provide an inventive concept and significantly more than the abstract idea. Thus, the claim is not patent eligible.
The additional elements regarding training are recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not amount to significantly more than the abstract idea. The claim is directed to an abstract idea.
Regarding claim 14
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1: The claim recites the abstract idea identified above regarding claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element(s) (“prior to obtaining the information about the training data”) – the act of receiving data. The claim is adding an insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g). The act of receiving data is recited at a high-level of generality (i.e., as a generic act of receiving performing a generic act function of receiving data) such that it amounts no more than a mere act to apply the exception using a generic act of receiving. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element(s) (“training, using the first plurality of inputs and the first plurality of outputs, an untrained ML model to generated the trained ML model”). The additional element is recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, the claim recites the additional element(s) of receiving data at a high-level of generality and is adding an insignificant extra-solution activity – see MPEP 2106.05(g). However, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood, routine, and conventional. See MPEP 2106.05(d)(II) – “Receiving or transmitting data over a network” or “Storing and retrieving information in memory”. Accordingly, this additional element does not provide an inventive concept and significantly more than the abstract idea. Thus, the claim is not patent eligible.
The additional elements regarding training are recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not amount to significantly more than the abstract idea. The claim is directed to an abstract idea.
Regarding claim 15
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“determining a first value based on a comparison between the first representation and the second representation;
determining a second value based on a comparison between the first performance data and the second performance data; and
determining, based on a weighted combination of the first value and the second value, whether to update the trained ML model or to generate a supplemental ML model to use with the trained ML model”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. In particular, the claim does not recite additional elements. Thus, the claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, the claim is not patent eligible.
Regarding claim 16
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1: The claim recites the abstract idea identified above regarding claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element (“wherein the first representation comprises a first histogram having a first plurality of counts for a plurality of bins, and the second representation comprises a second histogram having a second plurality of counts for the plurality of bins, wherein the first performance data comprises, for each bin of at least some of the plurality of bins, a measure of error incurred by the trained ML model when applied to inputs, among the first plurality of inputs, that fall in the bin, and wherein the second performance data comprises, for each bin of at least some of the plurality of bins, a measure of error incurred by the trained ML model when applied to inputs, among the second plurality of inputs, that fall in the bin”). This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not amount to significantly more than the abstract idea. See MPEP 2106.05(h).
Regarding claim 17
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“determining a number of bins, among the plurality of bins, for which a difference between measures of error specified by the first performance data and the second performance data exceeds an average difference between the measures of error across the plurality of bins;
determining to update the trained ML model when the number of bins exceeds a predetermined threshold number of bins; and
determining to generate a supplemental ML model when the number of bins is less or equal to the pre-determined threshold number of bins”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. In particular, the claim does not recite additional elements. Thus, the claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, the claim is not patent eligible.
Regarding claim 18
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“… ,
wherein the method further comprises:
identifying, using the first representation, the second representation, the first performance data and the second performance data, one or more bins for which to obtain additional inputs and corresponding ground truth values for improving performance of the trained ML model”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element (“wherein the first representation of the first distribution of the first plurality of inputs comprises a first histogram having a plurality of bins, wherein the second representation of the second distribution of the second plurality of inputs comprises a second histogram having the plurality of bins”). This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not amount to significantly more than the abstract idea. See MPEP 2106.05(h).
Regarding claim 19
The claim recites “A system, comprising: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:” to perform precisely the method of Claim 1. As performance of an abstract idea on generic computer components (see MPEP 2106.05(f)) cannot integrate the abstract idea into a practical application nor provide significantly more than the abstract idea itself, the claim is rejected for reasons set forth in the rejection of Claim 1.
Regarding claim 20
The claim recites “At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform:” to perform precisely the method of Claim 1. As performance of an abstract idea on generic computer components (see MPEP 2106.05(f)) and “Storing and retrieving information in memory” (see MPEP 2106.05(g) on Insignificant Extra-Solution Activity, and MPEP 2106.05(d) on Well-Understood, Routine, Conventional Activity) cannot integrate the abstract idea into a practical application nor provide significantly more than the abstract idea itself, the claim is rejected for reasons set forth in the rejection of Claim 1.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-5, 13-14, 19-20 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Boulanger et al. (US 2021/0133510 A1)
Regarding claim 1
Boulanger teaches
A method, comprising:
using at least one computer hardware processor to perform:
(Boulanger [fig(s) 3] [par(s) 39] “As illustrated, the image classification system 200 includes one or more processors 204 that perform data processing and/or other software execution operations for the image classification system 200. The processor 204 may include logic devices, microcontrollers, processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other devices that may be used by the image classification system 200 to execute appropriate instructions, such as software instructions stored in memory 206 including 3D simulation and image capture component 208, training dataset generation component 210, and image classification component 212 (e.g., a neural network trained by the training dataset), and/or other applications. The memory 206 may be implemented in one or more memory devices (e.g., memory components) that store executable instructions, data and information, including image data, video data, audio data, network information. The memory devices may include various types of memory for information storage including volatile and non-volatile memory devices, such as RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically-Erasable Read-Only Memory), flash memory, a disk drive, and other types of memory described herein.”;)
(A) obtaining information about training data used to generate a trained machine learning (ML) model, the training data comprising a first plurality of inputs and a corresponding first plurality of outputs, the information about the training data comprising:
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label. If the neural network 330 mislabels the input image (e.g., determines that the image is a “rhinoceros” instead of an “elephant”), then a backward pass through the neural network 330 may be used to adjust the neural network to correct for the misclassification. Referring to FIG. 4D, a trained neural network 340, may then be implemented in an application (i.e., a neural network inference application) on a run time environment to classify thermal images 342.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.”;)
a first representation of a first distribution of the first plurality of inputs, and
(Boulanger [fig(s) 1] [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.” [par(s) 28-33] “The scene generator 104 is operable to build a virtual 3D environment utilizing data from an object database 112, which stores 3D models and other object data allowing a modeled 3D object to be placed in the scene. The scene generator 104 may also apply environmental effects 114 (such as weather conditions, temperature, time of day, etc.). The environment simulation system 102 may optionally include an infrared sensor simulator/image capture component 106 for capturing infrared images of a scene and/or an optical image capture component 108 for capturing visible images of a generated scene.” [par(s) 22-23] “The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. … image characteristics (e.g., image size, object size, features extracted)” [par(s) 43] “The training dataset includes synthetic images as described herein, and may also include real images captured from an infrared, visible light, or other type of camera.”;)
first performance data indicative of a measure of performance of the trained ML model on the first plurality of inputs;
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.”;)
(B) obtaining information about new data to which the trained ML model was applied, the new data comprising a second plurality of inputs and a corresponding second plurality of outputs, the information about the new data comprising:
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label. If the neural network 330 mislabels the input image (e.g., determines that the image is a “rhinoceros” instead of an “elephant”), then a backward pass through the neural network 330 may be used to adjust the neural network to correct for the misclassification. Referring to FIG. 4D, a trained neural network 340, may then be implemented in an application (i.e., a neural network inference application) on a run time environment to classify thermal images 342.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results”; e.g., newly-generated synthetic images along with fig 1 read(s) on “new data to which the trained ML model was applied”.)
a second representation of a second distribution of the second plurality of inputs, and
(Boulanger [fig(s) 1] [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.” [par(s) 28-33] “The scene generator 104 is operable to build a virtual 3D environment utilizing data from an object database 112, which stores 3D models and other object data allowing a modeled 3D object to be placed in the scene. The scene generator 104 may also apply environmental effects 114 (such as weather conditions, temperature, time of day, etc.). The environment simulation system 102 may optionally include an infrared sensor simulator/image capture component 106 for capturing infrared images of a scene and/or an optical image capture component 108 for capturing visible images of a generated scene.” [par(s) 22-23] “The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. … image characteristics (e.g., image size, object size, features extracted)” [par(s) 43] “The training dataset includes synthetic images as described herein, and may also include real images captured from an infrared, visible light, or other type of camera.”;)
second performance data indicative of the measure of performance of the trained ML model on the second plurality of inputs;
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.”;)
(C) determining, using the first representation, the second representation, the first performance data, and the second performance data, whether to update the trained ML model or to generate a supplemental ML model to use with the trained ML model; and
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”;)
(D) when it is determined to update the trained ML model or to generate the supplemental ML model to use with the trained ML model, updating the trained ML model to generate an updated ML model or generating the supplemental ML model to use with the trained ML model.
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”;)
Regarding claim 2
Boulanger teaches claim 1.
Boulanger further teaches
wherein (C) comprises determining to update the trained ML model and (D) comprises updating the trained ML model to generate the updated ML model.
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”;)
Regarding claim 3
Boulanger teaches claim 2.
wherein updating the trained ML model comprises: (See claim 1)
Boulanger further teaches
training, using at least some of the new data, a second trained ML model;
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”;)
generating the updated ML model as an ensemble of the trained ML model and the second trained ML model.
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”; e.g., “The process repeats in an iterative fashion … in training an inference CNN” along with fig 1 read(s) on “an ensemble of the trained ML model and the second trained ML model” since a current ML model is used for generating a next ML model iteratively and together.)
Regarding claim 4
Boulanger teaches claim 3.
wherein generating the updated ML model further comprises (See claim 1)
Boulanger further teaches
determining weights for the trained ML model and the second trained ML model in the ensemble.
(Boulanger [fig(s) 1] [par(s) 43] “the training starts with a forward pass through the neural network 300 including feature extraction 304 in a plurality of convolution layers 306 and pooling layers 308, followed by image classification 310 in a plurality of fully connected layers 312 and an output layer 314. Next, a backward pass through the neural network 300 may be used to update the CNN parameters in view of errors produced in the forward pass (e.g., misclassified objects).” [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”; e.g., “The process repeats in an iterative fashion … in training an inference CNN” along with fig 1 read(s) on “the trained ML model and the second trained ML model in the ensemble” since a current ML model is used for generating a next ML model iteratively and together. In addition, e.g., updating “CNN parameters” based on the forward pass and backward pass read(s) on “determining weights”.)
Regarding claim 5
Boulanger teaches claim 2.
Boulanger further teaches
obtaining information about second training data used to generate the updated machine learning model, the second training data comprising a third plurality of inputs and a corresponding third plurality of outputs, the information about the second training data comprising:
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label.” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.”;)
a third representation of a third distribution of the third plurality of inputs, and
(Boulanger [fig(s) 1] [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.” [par(s) 28-33] “The scene generator 104 is operable to build a virtual 3D environment utilizing data from an object database 112, which stores 3D models and other object data allowing a modeled 3D object to be placed in the scene. The scene generator 104 may also apply environmental effects 114 (such as weather conditions, temperature, time of day, etc.). The environment simulation system 102 may optionally include an infrared sensor simulator/image capture component 106 for capturing infrared images of a scene and/or an optical image capture component 108 for capturing visible images of a generated scene.” [par(s) 22-23] “The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. … image characteristics (e.g., image size, object size, features extracted)” [par(s) 43] “The training dataset includes synthetic images as described herein, and may also include real images captured from an infrared, visible light, or other type of camera.”;)
third performance data indicative of a measure of performance of the updated ML model on the third plurality of inputs;
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.”;)
obtaining information about second new data to which the updated ML model was applied, the second new data comprising a fourth plurality of inputs and a corresponding fourth plurality of outputs, the information about the second new data comprising:
(Boulanger [fig(s) 1] [par(s) 45] “Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label. If the neural network 330 mislabels the input image (e.g., determines that the image is a “rhinoceros” instead of an “elephant”), then a backward pass through the neural network 330 may be used to adjust the neural network to correct for the misclassification. Referring to FIG. 4D, a trained neural network 340, may then be implemented in an application (i.e., a neural network inference application) on a run time environment to classify thermal images 342.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”; e.g., newly-generated synthetic images along with fig 1 read(s) on “second new data to which the updated ML model was applied”.)
a fourth representation of a fourth distribution of the fourth plurality of inputs, and
(Boulanger [fig(s) 1] [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.” [par(s) 28-33] “The scene generator 104 is operable to build a virtual 3D environment utilizing data from an object database 112, which stores 3D models and other object data allowing a modeled 3D object to be placed in the scene. The scene generator 104 may also apply environmental effects 114 (such as weather conditions, temperature, time of day, etc.). The environment simulation system 102 may optionally include an infrared sensor simulator/image capture component 106 for capturing infrared images of a scene and/or an optical image capture component 108 for capturing visible images of a generated scene.” [par(s) 22-23] “The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. … image characteristics (e.g., image size, object size, features extracted)” [par(s) 43] “The training dataset includes synthetic images as described herein, and may also include real images captured from an infrared, visible light, or other type of camera.”;)
fourth performance data indicative of the measure of performance of the updated ML model on the fourth plurality of inputs;
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.”;)
determining, using the third representation, the fourth representation, the third performance data, and the fourth performance data, whether to further update the updated ML model or to generate a further supplemental ML model to use with the updated ML model; and
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”;)
when it is determined to further update the updated ML model or to generate the further supplemental ML model to use with the updated ML model, updating the updated ML model or generating the further supplemental ML model.
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”;)
Regarding claim 13
Boulanger teaches claim 1.
Boulanger further teaches
prior to obtaining the information about the new data, applying the trained ML model to the second plurality of inputs to obtain the second plurality of outputs.
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label. If the neural network 330 mislabels the input image (e.g., determines that the image is a “rhinoceros” instead of an “elephant”), then a backward pass through the neural network 330 may be used to adjust the neural network to correct for the misclassification. Referring to FIG. 4D, a trained neural network 340, may then be implemented in an application (i.e., a neural network inference application) on a run time environment to classify thermal images 342.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results”; e.g., fig 1 read(s) on “prior to obtaining the information about the new data, applying the trained ML model to the second plurality of inputs to obtain the second plurality of outputs” since “Training Dataset Analysis Engine 70” obtains “Informative metrics” and “Performance results” after training and validation.)
Regarding claim 14
Boulanger teaches claim 1.
Boulanger further teaches
prior to obtaining the information about the training data, training, using the first plurality of inputs and the first plurality of outputs, an untrained ML model to generated the trained ML model.
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label. If the neural network 330 mislabels the input image (e.g., determines that the image is a “rhinoceros” instead of an “elephant”), then a backward pass through the neural network 330 may be used to adjust the neural network to correct for the misclassification. Referring to FIG. 4D, a trained neural network 340, may then be implemented in an application (i.e., a neural network inference application) on a run time environment to classify thermal images 342.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.”; e.g., fig 1 read(s) on “prior to obtaining the information about the training data, training, using the first plurality of inputs and the first plurality of outputs, an untrained ML model to generated the trained ML model” since “Training Dataset Analysis Engine 70” obtains “Informative metrics” and “Performance results” after training.)
Regarding claim 19
The claim is a system claim corresponding to the method claim 1, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 20
The claim is a computer-readable storage medium claim corresponding to the method claim 1, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 11-12, 16, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Boulanger et al. (US 2021/0133510 A1) in view of Hall et al. (US 20230162049 A1)
Regarding claim 11
Boulanger teaches claim 1.
However, Boulanger does not appear to explicitly teach:
wherein the first representation of the first distribution of the first plurality of inputs comprises a histogram having a plurality of bins and a plurality of counts corresponding to the plurality of bins, each of the plurality of counts indicating how many of the first plurality of inputs fall into a respective bin in the plurality of bins.
Hall teaches
wherein the first representation of the first distribution of the first plurality of inputs comprises a histogram having a plurality of bins and a plurality of counts corresponding to the plurality of bins, each of the plurality of counts indicating how many of the first plurality of inputs fall into a respective bin in the plurality of bins.
(Hall [fig(s) 7] [par(s) 130] “FIG. 7 is a histogram of the number of images per strictness threshold for test and train sets in normal and pneumonia labeled images according to an embodiment;” [par(s) 244] “FIG. 7 is a histogram of the number of images per strictness threshold for test and train sets in normal and pneumonia labeled images according to an embodiment. FIG. 7 highlights two important effects of label noise in the set. 1) Though representing only 12% of the aggregated dataset, the test set increases the number of noisy labels identified by 100% when compared with the number for the training set alone, underlining the knock-on effect that label noise can have on model performance. 2) This shows how false negatives added to a training set “confuses” the model, causing a counter-intuitive increase in the number of false positives.” [par(s) 203] “Using Algorithm 3, a histogram is generated that bins together images with the same number of successful predictions hl←∥{sj|sj=l}∥0, where bin l contains images that were successfully predicted by l models (0<l<n×k). A cumulative histogram H←Σi=0lhi is then used to calculate a percentage difference operator.;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Boulanger with the histogram bins of Hall.
One of ordinary skill in the art would have been motived to combine in order to provide an improvement in the AI performance of prediction by creating a clean training dataset for AI training based on removal of bad data.
(WERKHEISER [par(s) 337-338] “This indicates that the Untrainable Data Cleansing technique has in fact removed the mid-labeled and noisy data from the dataset, ultimately improving the data quality and thus the AI model performance.” [par(s) 359] “The above case studies demonstrated that embodiments of the UDC method can be used to effectively identify bad data ( or bad labels) even for "hard" classification problems such as detection of pneumonia from pediatric x-rays. The removal of the bad data was used to create a clean training dataset for AI training and resulted in an improvement in the AI performance for identifying pneumonia in x-rays. Additionally, UDC found bad data present in the test dataset used by AI practitioners to test and report on their AI performance for analyzing x-ray images for pneumonia. This means that the AI performance that is reported may not be the true performance of the AI, and potentially (unintentionally) misleading. Lastly, bad data that was identified by UDC for this particular problem was of the Noisy category, which both the AI and radiologist found difficult to label with confidence using the available x-ray image. This suggests that these images alone contain limited information to make a conclusive diagnosis with certainty, and thus have a higher probability of being mislabeled or mis-diagnosed.”)
Regarding claim 12
The combination of Boulanger, HALL teaches claim 11.
wherein the first performance data indicative of the measure of performance of the trained ML model on the first plurality of inputs comprises: (See claim 1)
Hall further teaches
for each bin of at least some of the plurality of bins, a measure of average error by the trained ML model when applied to inputs, among the first plurality of inputs, that fall in the bin.
(Hall [fig(s) 5-6, 11] “balanced accuracy” [par(s) 128] “FIG. 5 is a set of histogram plots showing balanced accuracy (top) and cross-entropy, or log loss, (bottom) (left) for various model architectures Wore UDC and (right) for the ResNet-50 architecture after UDC for varying strictness thresholds 1 according to an embodiment;” [par(s) 176-180] “(Mean) accuracy A: is the proportion of predictions for which the model was correct (NT) compared with the total number of predictions (N). Formally, accuracy has the following definition: … Balanced accuracy(or F1 Score) Abal: is more suitable in cases where the data class distribution is unbalanced. Balanced accuracy is calculated as the average of the class-based accuracy for all classes c
∈
{1... C}:
PNG
media_image1.png
105
467
media_image1.png
Greyscale
” [par(s) 240] “FIG. 5 shows the balanced accuracy (top) and cross-entropy, or log loss, (bottom) (left) for various model architectures before UDC and (right) for the ResNet-50 architecture after UDC for varying strictness thresholds 1 for the test set. The shading of the bars represent the performance of the model on the test set if the epoch (or model) chosen is that which resulted in the lowest log loss as measured against (diagonal lines) the test set and (black) the validation ("val") set. The discrepancy between these two values is indicative of the generalisability of the model; i.e. models that perform well one but not the other are not expected to generalise well. This discrepancy is shown to improve with UDC. Case Study 2B: Test Set Treated as Additional Data Source Case Study 2A shows that UDC improves model performance even on a blind test set, which is a measure of the power of the UDC method. In this section, the effect of treating the test set as a different data source is investigated. To this end, the test set is included (or "injected") into the training set and the resulting effect on model performance is noted.”;)
The combination of Boulanger, Hall is combinable with Hall for the same rationale as set forth above with respect to claim 11.
Regarding claim 16
Boulanger teaches claim 1.
However, Boulanger does not appear to explicitly teach:
wherein the first representation comprises a first histogram having a first plurality of counts for a plurality of bins, and the second representation comprises a second histogram having a second plurality of counts for the plurality of bins,
wherein the first performance data comprises, for each bin of at least some of the plurality of bins, a measure of error incurred by the trained ML model when applied to inputs, among the first plurality of inputs, that fall in the bin, and
wherein the second performance data comprises, for each bin of at least some of the plurality of bins, a measure of error incurred by the trained ML model when applied to inputs, among the second plurality of inputs, that fall in the bin.
Hall teaches
wherein the first representation comprises a first histogram having a first plurality of counts for a plurality of bins, and the second representation comprises a second histogram having a second plurality of counts for the plurality of bins,
(Hall [fig(s) 7] [par(s) 130] “FIG. 7 is a histogram of the number of images per strictness threshold for test and train sets in normal and pneumonia labeled images according to an embodiment;” [par(s) 244] “FIG. 7 is a histogram of the number of images per strictness threshold for test and train sets in normal and pneumonia labeled images according to an embodiment. FIG. 7 highlights two important effects of label noise in the set. 1) Though representing only 12% of the aggregated dataset, the test set increases the number of noisy labels identified by 100% when compared with the number for the training set alone, underlining the knock-on effect that label noise can have on model performance. 2) This shows how false negatives added to a training set “confuses” the model, causing a counter-intuitive increase in the number of false positives.” [par(s) 203] “Using Algorithm 3, a histogram is generated that bins together images with the same number of successful predictions hl←∥{sj|sj=l}∥0, where bin l contains images that were successfully predicted by l models (0<l<n×k). A cumulative histogram H←Σi=0lhi is then used to calculate a percentage difference operator.;)
wherein the first performance data comprises, for each bin of at least some of the plurality of bins, a measure of error incurred by the trained ML model when applied to inputs, among the first plurality of inputs, that fall in the bin, and
(Hall [fig(s) 5-6, 11] “balanced accuracy” [par(s) 128] “FIG. 5 is a set of histogram plots showing balanced accuracy (top) and cross-entropy, or log loss, (bottom) (left) for various model architectures Wore UDC and (right) for the ResNet-50 architecture after UDC for varying strictness thresholds 1 according to an embodiment;” [par(s) 176-180] “(Mean) accuracy A: is the proportion of predictions for which the model was correct (NT) compared with the total number of predictions (N). Formally, accuracy has the following definition: … Balanced accuracy(or F1 Score) Abal: is more suitable in cases where the data class distribution is unbalanced. Balanced accuracy is calculated as the average of the class-based accuracy for all classes c
∈
{1... C}:
PNG
media_image1.png
105
467
media_image1.png
Greyscale
” [par(s) 240] “FIG. 5 shows the balanced accuracy (top) and cross-entropy, or log loss, (bottom) (left) for various model architectures before UDC and (right) for the ResNet-50 architecture after UDC for varying strictness thresholds 1 for the test set. The shading of the bars represent the performance of the model on the test set if the epoch (or model) chosen is that which resulted in the lowest log loss as measured against (diagonal lines) the test set and (black) the validation ("val") set. The discrepancy between these two values is indicative of the generalisability of the model; i.e. models that perform well one but not the other are not expected to generalise well. This discrepancy is shown to improve with UDC. Case Study 2B: Test Set Treated as Additional Data Source Case Study 2A shows that UDC improves model performance even on a blind test set, which is a measure of the power of the UDC method. In this section, the effect of treating the test set as a different data source is investigated. To this end, the test set is included (or "injected") into the training set and the resulting effect on model performance is noted.”;)
wherein the second performance data comprises, for each bin of at least some of the plurality of bins, a measure of error incurred by the trained ML model when applied to inputs, among the second plurality of inputs, that fall in the bin.
(Hall [fig(s) 5-6, 11] “balanced accuracy” [par(s) 128] “FIG. 5 is a set of histogram plots showing balanced accuracy (top) and cross-entropy, or log loss, (bottom) (left) for various model architectures Wore UDC and (right) for the ResNet-50 architecture after UDC for varying strictness thresholds 1 according to an embodiment;” [par(s) 176-180] “(Mean) accuracy A: is the proportion of predictions for which the model was correct (NT) compared with the total number of predictions (N). Formally, accuracy has the following definition: … Balanced accuracy(or F1 Score) Abal: is more suitable in cases where the data class distribution is unbalanced. Balanced accuracy is calculated as the average of the class-based accuracy for all classes c
∈
{1... C}:
PNG
media_image1.png
105
467
media_image1.png
Greyscale
” [par(s) 240] “FIG. 5 shows the balanced accuracy (top) and cross-entropy, or log loss, (bottom) (left) for various model architectures before UDC and (right) for the ResNet-50 architecture after UDC for varying strictness thresholds 1 for the test set. The shading of the bars represent the performance of the model on the test set if the epoch (or model) chosen is that which resulted in the lowest log loss as measured against (diagonal lines) the test set and (black) the validation ("val") set. The discrepancy between these two values is indicative of the generalisability of the model; i.e. models that perform well one but not the other are not expected to generalise well. This discrepancy is shown to improve with UDC. Case Study 2B: Test Set Treated as Additional Data Source Case Study 2A shows that UDC improves model performance even on a blind test set, which is a measure of the power of the UDC method. In this section, the effect of treating the test set as a different data source is investigated. To this end, the test set is included (or "injected") into the training set and the resulting effect on model performance is noted.”;)
Boulanger is combinable with Hall for the same rationale as set forth above with respect to claim 11.
Regarding claim 18
Boulanger teaches claim 1.
Boulanger further teaches
identifying, using the first representation, the second representation, the first performance data and the second performance data, one or more bins for which to obtain additional inputs and corresponding ground truth values for improving performance of the trained ML model.
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN” [par(s) 45] “Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label.”; e.g., “The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56.” read(s) on “bins”.)
However, Boulanger does not appear to explicitly teach:
wherein the first representation of the first distribution of the first plurality of inputs comprises a first histogram having a plurality of bins,
wherein the second representation of the second distribution of the second plurality of inputs comprises a second histogram having the plurality of bins, wherein the method further comprises:
Hall teaches
wherein the first representation of the first distribution of the first plurality of inputs comprises a first histogram having a plurality of bins,
(Hall [fig(s) 7] [par(s) 130] “FIG. 7 is a histogram of the number of images per strictness threshold for test and train sets in normal and pneumonia labeled images according to an embodiment;” [par(s) 244] “FIG. 7 is a histogram of the number of images per strictness threshold for test and train sets in normal and pneumonia labeled images according to an embodiment. FIG. 7 highlights two important effects of label noise in the set. 1) Though representing only 12% of the aggregated dataset, the test set increases the number of noisy labels identified by 100% when compared with the number for the training set alone, underlining the knock-on effect that label noise can have on model performance. 2) This shows how false negatives added to a training set “confuses” the model, causing a counter-intuitive increase in the number of false positives.” [par(s) 203] “Using Algorithm 3, a histogram is generated that bins together images with the same number of successful predictions hl←∥{sj|sj=l}∥0, where bin l contains images that were successfully predicted by l models (0<l<n×k). A cumulative histogram H←Σi=0lhi is then used to calculate a percentage difference operator.;)
wherein the second representation of the second distribution of the second plurality of inputs comprises a second histogram having the plurality of bins,
(Hall [fig(s) 7] [par(s) 130] “FIG. 7 is a histogram of the number of images per strictness threshold for test and train sets in normal and pneumonia labeled images according to an embodiment;” [par(s) 244] “FIG. 7 is a histogram of the number of images per strictness threshold for test and train sets in normal and pneumonia labeled images according to an embodiment. FIG. 7 highlights two important effects of label noise in the set. 1) Though representing only 12% of the aggregated dataset, the test set increases the number of noisy labels identified by 100% when compared with the number for the training set alone, underlining the knock-on effect that label noise can have on model performance. 2) This shows how false negatives added to a training set “confuses” the model, causing a counter-intuitive increase in the number of false positives.” [par(s) 203] “Using Algorithm 3, a histogram is generated that bins together images with the same number of successful predictions hl←∥{sj|sj=l}∥0, where bin l contains images that were successfully predicted by l models (0<l<n×k). A cumulative histogram H←Σi=0lhi is then used to calculate a percentage difference operator.;)
Boulanger is combinable with Hall for the same rationale as set forth above with respect to claim 11.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Boulanger et al. (US 2021/0133510 A1) in view of Frosch et al. (US 20220335328 A1)
Regarding claim 15
Boulanger teaches claim 1.
wherein the determining comprises: (See claim 1)
determining a first value based on a comparison between the first representation and the second representation;
(Frosch [par(s) 20] “The accuracy of the determinations produced by the machine learning model when deployed in the field can differ from that exhibited by the machine learning model during training and/or validation. In various cases, this performance disparity can be due to differences between the input data encountered by the machine learning model during deployment and the input data encountered by the machine learning model during training and/or validation. For example, a machine learning model can be configured to receive as input an MRI scanned image associated with a patient and to produce as output a diagnosis based on the MRI scanned image. In various cases, the machine learning model can be trained and/or validated on datasets that comprise MRI scanned images associated with patients having certain demographics (e.g., patients within a certain age range, patients of certain ethnicities, patients without comorbidities). In various aspects, once deployed in the field, the machine leaning model can encounter MRI scanned images associated with patients that fall outside of these certain demographics (e.g., patients outside of the certain age range, patients not belonging to the certain ethnicities, patients with comorbidities). Because the machine learning model did not encounter such MRI scanned images during training and/or validation, the machine learning model can exhibit reduced diagnosis accuracy when encountering such MRI scanned images during deployment. Such differences in input data can exist because it can be impracticable to represent within a training dataset and/or a validation dataset the full range of all possible input data variations which can be encountered in the field.” [par(s) 80] “In various instances, the augmentation component 122 can apply any suitable augmentations to the one or more anonymized copies of the data candidate 302. For example, if the data candidate 302 is an image, the augmentation component 122 can modify the one or more anonymized copies of the data candidate 302 by randomly rotating various copies, randomly altering the zoom-level and/or magnification level of various copies, randomly applying a Gaussian blur to various copies, randomly inserting images of various objects and/or artefacts into various copies, and/or randomly applying various optical distortions to various copies. In various instances, the one or more augmented and anonymized copies of the data candidate 302 can be considered as the set of synthetic data candidates 502.”; e.g., “set of synthetic data candidates” read(s) on “first value”.)
determining a second value based on a comparison between the first performance data and the second performance data; and
(Frosch [par(s) 20] “The accuracy of the determinations produced by the machine learning model when deployed in the field can differ from that exhibited by the machine learning model during training and/or validation. In various cases, this performance disparity can be due to differences between the input data encountered by the machine learning model during deployment and the input data encountered by the machine learning model during training and/or validation. For example, a machine learning model can be configured to receive as input an MRI scanned image associated with a patient and to produce as output a diagnosis based on the MRI scanned image. In various cases, the machine learning model can be trained and/or validated on datasets that comprise MRI scanned images associated with patients having certain demographics (e.g., patients within a certain age range, patients of certain ethnicities, patients without comorbidities). In various aspects, once deployed in the field, the machine leaning model can encounter MRI scanned images associated with patients that fall outside of these certain demographics (e.g., patients outside of the certain age range, patients not belonging to the certain ethnicities, patients with comorbidities). Because the machine learning model did not encounter such MRI scanned images during training and/or validation, the machine learning model can exhibit reduced diagnosis accuracy when encountering such MRI scanned images during deployment. Such differences in input data can exist because it can be impracticable to represent within a training dataset and/or a validation dataset the full range of all possible input data variations which can be encountered in the field.” [par(s) 80] “In various instances, the augmentation component 122 can apply any suitable augmentations to the one or more anonymized copies of the data candidate 302. For example, if the data candidate 302 is an image, the augmentation component 122 can modify the one or more anonymized copies of the data candidate 302 by randomly rotating various copies, randomly altering the zoom-level and/or magnification level of various copies, randomly applying a Gaussian blur to various copies, randomly inserting images of various objects and/or artefacts into various copies, and/or randomly applying various optical distortions to various copies. In various instances, the one or more augmented and anonymized copies of the data candidate 302 can be considered as the set of synthetic data candidates 502.”; e.g., “set of synthetic data candidates” read(s) on “second value”.)
determining, based on a weighted combination of the first value and the second value, whether to update the trained ML model or to generate a supplemental ML model to use with the trained ML model.
(Frosch [par(s) 39] “In various embodiments, the training component can electronically train (e.g., via supervised training) the particular deployed machine learning model based on the one or more synthetic data candidates. More specifically, the training component can electronically feed to the particular deployed machine learning model the one or more synthetic data candidates, thereby causing the particular deployed machine learning model to output results. In various cases, the training component can compute errors and/or losses based on differences between the particular expert conclusion and such outputted results (e.g., the particular expert conclusion can be considered as the annotation associated with the one or more synthetic data candidates). In various instances, the training component can electronically update (e.g., via backpropagation) parameters (e.g., weights and/or biases) of the particular deployed machine learning model based on such computed errors and/or losses. In this way, the training component can iteratively improve the performance of the particular deployed machine learning model based on the one or more synthetic data candidates. Those having ordinary skill in the art will appreciate that, in some cases, unsupervised training and/or reinforcement learning can be implemented, rather than supervised training.”; e.g., “train (e.g., via supervised training) the particular deployed machine learning model based on the one or more synthetic data candidates” read(s) on “weighted combination of the first value and the second value” since each image is used for training together in combination.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Boulanger with the weighted combination of Frosch.
One of ordinary skill in the art would have been motived to combine in order to improve performance of the deployed machine learning model by leveraging failure data.
(Frosch [par(s) 23] “the inventors devised various embodiments of the subject innovation so as to automatically capture input data candidates which cause a deployed machine learning model to generate inaccurate determinations. This can be beneficial because failure data (e.g., input data candidates which cause a deployed machine learning model to produce incorrect inferences) can be valuable from a development and/or training perspective. Specifically, such failure data can be leveraged so as to improve the performance of the deployed machine learning model.”)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SEHWAN KIM whose telephone number is (571)270-7409. The examiner can normally be reached Mon - Fri 9:00 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J Huntley can be reached on (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SEHWAN KIM/Examiner, Art Unit 2129
12/23/2025