Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 08/13/2026 has been entered.
Examiner’s Note
Under Allowable Subject Matter, “and to include all of the limitations of the base claim and any intervening claims” has been removed. (emphasis added) In other words, if claim 17 is integrated into the independent claims, the claim rejections under 35 USC § 103 will be withdrawn.
Providing supporting paragraph(s) for each limitation of amended/new claim(s) in Remarks is strongly requested for clear and definite claim interpretations by Examiner (e.g., to avoid rejections under 35 U.S.C § 112(a) “Lack of written description”)
Applicant can schedule an interview at any stage of the prosecution (e.g., Non-Final, Final, and After-Final) to discuss any issues related to, for example, rejections under 35 U.S.C § 101 and § 103, for moving toward allowance.
Priority
Acknowledgment is made of applicant's claim for the provisional application filed on 07/30/2021.
Allowable Subject Matter
Claims 17 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) and 35 U.S.C. 101 set forth in this Office action.
The following is an examiner’s statement of reasons for allowance:
The prior art of record, Boulanger et al. (US 2021/0133510 A1) teaches that a CNN training process tracks image-related metrics and other informative metrics, and the trained inference CNN may then be tested to generate performance results. The training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates a new training dataset.
Hall et al. (US 2023/0162049 A1) teaches a histogram of the number of images per strictness threshold for test and train sets in normal and pneumonia labeled images. In addition, the system shows a set of histogram plots showing balanced accuracy and cross-entropy, or log loss, for various model architectures, where balanced accuracy is calculated as the average of the class-based accuracy for all classes.
Franc et al. (US 20170316342 A1) teaches thresholds of a data representation of data features where the thresholds specify pre-defined bins (e.g., uniform and equidistant bins). Adjacent bins of the pre-defined bins having substantially similar weights can be reciprocally merged, and the merging results in a number of refined bins.
Coover et al. (US 20210073275 A1) teaches that the number of bin values in the sub-fingerprint can be based on a threshold number of bin values. The number of bin values in the sub-fingerprint can be based on a trained neural network to determine which extrema are most likely to contribute to a match. The example sub-fingerprint includes 20 bin values representative of extrema, and the number of bin values in the sub-fingerprint can be variable.
SHAVIT et al. (US 20170368413 A1) teaches that each attribute possible range may be divided into a bin. A coverage measure to end training may be at least one measurement in each bin, or at least one measurement in certain selected bins or in some number of bins defined as threshold. So the ending criteria may be all or a certain number of the attributes bin coverage.
Yao et al. (US 20180137383 A1) teaches that a fern may be used to perform data partition as in ESR (Explicit Shape Regression). Each fern is a binary decision tree structure with a set of binary tests. Each image is passed down all of the ferns as it traverses through each stage of the regression. The fern is a composition of some number (N) of features and thresholds (N) that divide the features from all of the training samples into some number of bins, such as 2N. Each bin corresponds to a regression output that best matches the new image that is to be identified.
However, the prior art of record, taken either alone or in combination, fails to teach or fairly suggest claim 17 in combination with the remaining features and elements of the claimed invention.
Response to Arguments
Applicant's arguments filed on 08/13/2026 have been fully considered but they are not persuasive.
In Remarks regarding 112(b), Applicant contends:
The term "the plurality of bins" in line 5 refers back to the same plurality as in line 3 as the claim is written to cover the first and second histograms having the same plurality of bins.
Examiner’s response:
The examiner understands the applicant’s assertion.
However, if the term "the plurality of bins" in line 5 referred back to the same plurality as in line 3, the second histogram and the first histogram are the same. But the second histogram and the first histogram may not be the same. For at least these reasons, Applicant's arguments are not convincing.
In Remarks regarding 101, Applicant contends:
In particular, the inclusion of (amended) limitations (D) and (E) into claim 1 ties the claim to concrete technological improvements expressly described in the Specification. Indeed, the Specification (par. 53) teaches that the techniques developed by the inventors "improve ML technology by enabling automatic adapting, without manual intervention, of ML models in different ways depending on the source of the new data." Additionally, "the developed techniques save time and cost compared to conventional techniques where data analysts manually review and process new data and create an entirely new ML model using the manually processed data." Id. These are improvements to machine learning technology and how it operate; it is not merely using a computer as a tool to perform an abstract idea. As amended, acts (D) and (E) explicitly recite - the adapting and training steps as well as the use of different ML models (e.g., trained or supplemental) based on the source of new data - that achieve this improvement. Because the claim recites a technological improvement - which is described in the Specification - the claim is patent eligible under Step 2A, Prong 2 of the Alice/Mayo analysis.
Examiner’s response:
The examiner understands the applicant’s assertion.
However, it appears that each processing step is just applying the abstract idea to a general field of endeavor with additional elements. In addition, improvements to technology or technical field are not necessarily reflected in the claims. Thus, the claim does not integrate the judicial exception into a practical application, and the claim does not amount to significantly more than the judicial exception.
The examiner understands the applicant’s assertion regarding improvements.
It appears that par 51 states problems of the conventional approaches and pars 52-53 provide improvements over the conventional approaches. However, it is not clear how the steps (D) and (E) reflect the asserted improvements. It appears that the step (D) just updates the ML model or generates a supplemental ML model based on the two alternatives. In addition, the step (E) provides a new input data to either the trained ML model or the supplemental ML model depending on the property of the new input data. However, it is not clear how the steps provide improvements. Elaborating the claimed steps in more detail so that the claims reflect the asserted improvements may overcome the existing rejection.
For now, the limitations do not clearly show e.g., improvements in computer technology and improvements to other technical fields. Rather, the improvements in Remarks are about just improving the abstract ideas of the independent claims. It doesn’t seem that the specification and/or the independent claims clearly show how the inventive concept of the claims enables improvements and how they are tied together. The applicant may need to amend the claims to show how the claim languages and improvements are tied together.
To find a valid improvement to a technology, MPEP 2106.04(d)(1) says the specification must explain the improvement and that the claim must reflect the disclosed improvement. Furthermore, the improvement should not be merely a consequence of the abstract idea. See MPEP 2106.05(a). An improvement in the abstract idea itself is not an improvement to technology.
For at least these reasons, Applicant's arguments are not convincing.
The Examiner encourages Applicant to schedule an interview to discuss issues related to, for example, the rejections noted below under 35 U.S.C § 101.
Applicant’s arguments regarding 35 USC § 102/103 with respect to the independent claims have been considered but are moot because the arguments are directed to amended limitation(s) that has/have not been previously examined.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 18 is/are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim(s) 18 recite(s) the limitation “the plurality of bins” (line 5). There is insufficient antecedent basis for this limitation in the claim. It is not clear if it indicates “a plurality of bins” (line 3) or not since “a plurality of bins” (line 3) is based on “first histogram” while “the plurality of bins” (line 5) is based on “second histogram”. Thus, it appears it may need to read “a plurality of bins”, or something else. For the purposes of examination, “a plurality of bins” is used.
Claim(s) 18 each recite(s) limitations that raise issues of indefiniteness as set forth above, and their dependent claims are rejected at least based on their direct and/or indirect dependency from the claims listed above. Appropriate explanation and/or amendment is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-5, 7, 9-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“…, comprising:
… to perform:
(A) …, …;
(B) …, …;
(C) determining, using the first representation, the second representation, the first performance data, and the second performance data, whether to update the trained ML model or to generate a supplemental ML model to use with the trained ML model, the determining of step (C) comprising:
comparing the first representation to the second representation to assess a difference between the first representation and the second representation;
comparing the first performance data to the second performance data to assess a difference between the first performance data and the second performance data; and
determining, based on the assessed differences, whether to update the trained ML model or to generate the supplemental ML model to use with the trained ML model;
(D) …; and
(E) determining whether to provide new input data as input to the trained ML model or the supplemental ML model, the determining of step (E) comprising:
determining whether the new input data is in a first portion of the input data domain or in a second portion of the input data domain;
…”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). In particular, the claim recites an additional element(s) (“using at least one computer hardware processor to perform”, “to which the trained ML model was applied”) – using a device and/or a model to process data. The device and the model in each step are recited at a high-level of generality (i.e., as a generic computer performing a generic computer function of processing data) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element(s) (“obtaining information about training data used to …”, “obtaining information about new data …”) – the act of receiving data. The claim is adding an insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g). The act of receiving data is recited at a high-level of generality (i.e., as a generic act of receiving performing a generic act function of receiving data) such that it amounts no more than a mere act to apply the exception using a generic act of receiving. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element(s) (“generate a trained machine learning (ML) model”, “adapting the trained ML model based on the determining of step (C), wherein the adapting comprises: when it is determined to update the trained ML model or to generate the supplemental ML model to use with the trained ML model, either updating the trained ML model through additional training to generate an updated ML model or generating the supplemental ML model, by training the supplemental ML model, to use with the trained ML model, wherein the adapting further comprises generating the supplemental ML model”). The additional element is recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element (“the training data comprising a first plurality of inputs and a corresponding first plurality of outputs, the first plurality of inputs being from a first portion of an input data domain, the information about the training data comprising: a first representation of a first distribution of the first plurality of inputs, and first performance data indicative of a measure of performance of the trained ML model on the first plurality of inputs”, “the new data comprising a second plurality of inputs and a corresponding second plurality of outputs, the information about the new data comprising: a second representation of a second distribution of the second plurality of inputs, and second performance data indicative of a measure of performance of the trained ML model on the second plurality of inputs”). This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)
In particular, the claim recites an additional element(s) (“when it is determined that the new input data is in the first portion of the input data domain, providing the new input data as input to the trained ML model to obtain a first corresponding output; and when it is determined that the new input data is in the second portion of the input data domain, providing the new input data as input to the supplemental ML model to obtain a second corresponding output”) – the act of providing (i.e. inputting) data. The claim is adding an insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g). The act of inputting data is recited at a high-level of generality (i.e., as a generic act of inputting performing a generic act function of inputting data) such that it amounts no more than a mere act to apply the exception using a generic act of inputting. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, with respect to integration of the abstract idea into a practical application, the additional elements of using a generic computer component to perform each step amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claim is not patent eligible. MPEP 2106.05(f).
As discussed above, the claim recites the additional element(s) of receiving data at a high-level of generality and is adding an insignificant extra-solution activity – see MPEP 2106.05(g). However, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood, routine, and conventional. See MPEP 2106.05(d)(II) – “Receiving or transmitting data over a network” or “Storing and retrieving information in memory”. Accordingly, this additional element does not provide an inventive concept and significantly more than the abstract idea. Thus, the claim is not patent eligible.
The additional elements regarding training are recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not amount to significantly more than the abstract idea. The claim is directed to an abstract idea.
This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not amount to significantly more than the abstract idea. See MPEP 2106.05(h).
As discussed above, the claim recites the additional element(s) of inputting data at a high-level of generality and is adding an insignificant extra-solution activity – see MPEP 2106.05(g) – “Mere Data Gathering”. However, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood, routine, and conventional. See MPEP 2106.05(d)(II) – “Receiving or transmitting data over a network” or “Storing and retrieving information in memory”. Accordingly, this additional element does not provide an inventive concept and significantly more than the abstract idea. Thus, the claim is not patent eligible.
Regarding claim 2
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“determining to update the trained ML model”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element(s) (“updating the trained ML model to generate the updated ML model”). The additional element is recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The additional elements regarding training are recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not amount to significantly more than the abstract idea. The claim is directed to an abstract idea.
Regarding claim 3
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1: The claim recites the abstract idea identified above regarding claim 2.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element(s) (“training, using at least some of the new data, a second trained ML model; generating the updated ML model as an ensemble of the trained ML model and the second trained ML model”). The additional element is recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The additional elements regarding training are recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not amount to significantly more than the abstract idea. The claim is directed to an abstract idea.
Regarding claim 4
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“determining weights for the trained ML model and the second trained ML model in the ensemble”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. In particular, the claim does not recite additional elements. Thus, the claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, the claim is not patent eligible.
Regarding claim 5
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“…, …;
…, …;
determining, using the third representation, the fourth representation, the third performance data, and the fourth performance data, whether to further update the updated ML model or to generate a further supplemental ML model to use with the updated ML model; and
when it is determined to further update the updated ML model or to generate the further supplemental ML model to use with the updated ML model, …”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element(s) (“obtaining information about second training data used to …”, “obtaining information about second new data”) – the act of receiving data. The claim is adding an insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g). The act of receiving data is recited at a high-level of generality (i.e., as a generic act of receiving performing a generic act function of receiving data) such that it amounts no more than a mere act to apply the exception using a generic act of receiving. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
The claim recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). In particular, the claim recites an additional element(s) (“to which the updated ML model was applied”) – using a device and/or a model to process data. The device and the model in each step are recited at a high-level of generality (i.e., as a generic computer performing a generic computer function of processing data) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element(s) (“generate the updated ML model”, “updating the updated ML model or generating the further supplemental ML model”). The additional element is recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element (“the second training data comprising a third plurality of inputs and a corresponding third plurality of outputs, the information about the second training data comprising: a third representation of a third distribution of the third plurality of inputs, and third performance data indicative of a measure of performance of the updated ML model on the third plurality of inputs”, “the second new data comprising a fourth plurality of inputs and a corresponding fourth plurality of outputs, the information about the second new data comprising: a fourth representation of a fourth distribution of the fourth plurality of inputs, and fourth performance data indicative of a measure of performance of the updated ML model on the fourth plurality of inputs”). This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, the claim recites the additional element(s) of receiving data at a high-level of generality and is adding an insignificant extra-solution activity – see MPEP 2106.05(g). However, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood, routine, and conventional. See MPEP 2106.05(d)(II) – “Receiving or transmitting data over a network” or “Storing and retrieving information in memory”. Accordingly, this additional element does not provide an inventive concept and significantly more than the abstract idea. Thus, the claim is not patent eligible.
As discussed above, with respect to integration of the abstract idea into a practical application, the additional elements of using a generic computer component to perform each step amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claim is not patent eligible. MPEP 2106.05(f).
The additional elements regarding training are recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not amount to significantly more than the abstract idea. The claim is directed to an abstract idea.
This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not amount to significantly more than the abstract idea. See MPEP 2106.05(h).
Regarding claim 7
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“…;
associating the trained ML model with the first portion of an input data domain; and
associating the supplemental ML model with the second portion of the input data domain”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element(s) (“training, using at least some of the new data, the supplemental ML model”). The additional element is recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The additional elements regarding training are recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not amount to significantly more than the abstract idea. The claim is directed to an abstract idea.
Regarding claim 9
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“monitoring performance of the trained ML model and the supplemental ML model; and
adjusting portions of the input data domain associated with the trained ML model and the supplemental ML model based on results of the monitoring,
wherein adjusting the portions comprises adjusting a boundary between the first portion of the input data domain and the second portion of the input data domain thereby changing the first portion and the second portion of the input data domain”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. In particular, the claim does not recite additional elements. Thus, the claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, the claim is not patent eligible.
Regarding claim 10
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“…, …;
…, …;
determining, using the third representation, the fourth representation, the third performance data, and the fourth performance data, whether to further update the supplemental ML model or to generate a second supplemental ML model to use with the trained ML model and the supplemental ML model; and
when it is determined to further update the supplemental ML model or to generate the second supplemental ML model to use with the trained ML model and the supplemental ML model, …”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element(s) (“obtaining information about second training data used to”, “obtaining information about second new data”) – the act of receiving data. The claim is adding an insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g). The act of receiving data is recited at a high-level of generality (i.e., as a generic act of receiving performing a generic act function of receiving data) such that it amounts no more than a mere act to apply the exception using a generic act of receiving. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element(s) (“generate the supplemental ML model”, “updating the supplemental ML model or generating the second supplemental ML model”). The additional element is recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
The claim recites additional elements that are mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See MPEP 2106.05(f). In particular, the claim recites an additional element(s) (“to which the supplemental ML model was applied”) – using a device and/or a model to process data. The device and the model in each step are recited at a high-level of generality (i.e., as a generic computer performing a generic computer function of processing data) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element (“the second training data comprising a third plurality of inputs and a corresponding third plurality of outputs, the information about the second training data comprising: a third representation of a third distribution of the third plurality of inputs, and third performance data indicative of a measure of performance of the supplemental ML model on the third plurality of inputs”, “the second new data comprising a fourth plurality of inputs and a corresponding fourth plurality of outputs, the information about the second new data comprising: a fourth representation of a fourth distribution of the fourth plurality of inputs, and fourth performance data indicative of a measure of performance of the supplemental ML model on the fourth plurality of inputs”). This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, the claim recites the additional element(s) of receiving data at a high-level of generality and is adding an insignificant extra-solution activity – see MPEP 2106.05(g). However, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood, routine, and conventional. See MPEP 2106.05(d)(II) – “Receiving or transmitting data over a network” or “Storing and retrieving information in memory”. Accordingly, this additional element does not provide an inventive concept and significantly more than the abstract idea. Thus, the claim is not patent eligible.
The additional elements regarding training are recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not amount to significantly more than the abstract idea. The claim is directed to an abstract idea.
As discussed above, with respect to integration of the abstract idea into a practical application, the additional elements of using a generic computer component to perform each step amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claim is not patent eligible. MPEP 2106.05(f).
This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not amount to significantly more than the abstract idea. See MPEP 2106.05(h).
Regarding claim 11
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1: The claim recites the abstract idea identified above regarding claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element (“wherein the first representation of the first distribution of the first plurality of inputs comprises a histogram having a plurality of bins and a plurality of counts corresponding to the plurality of bins, each of the plurality of counts indicating how many of the first plurality of inputs fall into a respective bin in the plurality of bins”). This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not amount to significantly more than the abstract idea. See MPEP 2106.05(h).
Regarding claim 12
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1: The claim recites the abstract idea identified above regarding claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element (“wherein the first performance data indicative of the measure of performance of the trained ML model on the first plurality of inputs comprises: for each bin of at least some of the plurality of bins, a measure of average error by the trained ML model when applied to inputs, among the first plurality of inputs, that fall in the bin”). This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not amount to significantly more than the abstract idea. See MPEP 2106.05(h).
Regarding claim 13
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1: The claim recites the abstract idea identified above regarding claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element(s) (“prior to obtaining the information about the new data”) – the act of receiving data. The claim is adding an insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g). The act of receiving data is recited at a high-level of generality (i.e., as a generic act of receiving performing a generic act function of receiving data) such that it amounts no more than a mere act to apply the exception using a generic act of receiving. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element(s) (“applying the trained ML model to the second plurality of inputs to obtain the second plurality of outputs”). The additional element is recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, the claim recites the additional element(s) of receiving data at a high-level of generality and is adding an insignificant extra-solution activity – see MPEP 2106.05(g). However, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood, routine, and conventional. See MPEP 2106.05(d)(II) – “Receiving or transmitting data over a network” or “Storing and retrieving information in memory”. Accordingly, this additional element does not provide an inventive concept and significantly more than the abstract idea. Thus, the claim is not patent eligible.
The additional elements regarding training are recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not amount to significantly more than the abstract idea. The claim is directed to an abstract idea.
Regarding claim 14
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1: The claim recites the abstract idea identified above regarding claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element(s) (“prior to obtaining the information about the training data”) – the act of receiving data. The claim is adding an insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g). The act of receiving data is recited at a high-level of generality (i.e., as a generic act of receiving performing a generic act function of receiving data) such that it amounts no more than a mere act to apply the exception using a generic act of receiving. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
In particular, the claim recites an additional element(s) (“training, using the first plurality of inputs and the first plurality of outputs, an untrained ML model to generate the trained ML model”). The additional element is recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, the claim recites the additional element(s) of receiving data at a high-level of generality and is adding an insignificant extra-solution activity – see MPEP 2106.05(g). However, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood, routine, and conventional. See MPEP 2106.05(d)(II) – “Receiving or transmitting data over a network” or “Storing and retrieving information in memory”. Accordingly, this additional element does not provide an inventive concept and significantly more than the abstract idea. Thus, the claim is not patent eligible.
The additional elements regarding training are recited at such a high level without any details as to how a model is trained such that it amounts to only the idea of a solution or outcome because it fails to recite details of how a solution to a problem is accomplished, and, therefore, represents no more than mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Accordingly, this additional element does not amount to significantly more than the abstract idea. The claim is directed to an abstract idea.
Regarding claim 15
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“determining a first value based on the comparison between the first representation and the second representation;
determining a second value based on the comparison between the first performance data and the second performance data; and
determining, based on a weighted combination of the first value and the second value, whether to update the trained ML model or to generate a supplemental ML model to use with the trained ML model”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. In particular, the claim does not recite additional elements. Thus, the claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, the claim is not patent eligible.
Regarding claim 16
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1: The claim recites the abstract idea identified above regarding claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element (“wherein the first representation comprises a first histogram having a first plurality of counts for a plurality of bins, and the second representation comprises a second histogram having a second plurality of counts for the plurality of bins, wherein the first performance data comprises, for each bin of at least some of the plurality of bins, a measure of error incurred by the trained ML model when applied to inputs, among the first plurality of inputs, that fall in the bin, and wherein the second performance data comprises, for each bin of at least some of the plurality of bins, a measure of error incurred by the trained ML model when applied to inputs, among the second plurality of inputs, that fall in the bin”). This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not amount to significantly more than the abstract idea. See MPEP 2106.05(h).
Regarding claim 17
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“determining a number of bins, among the plurality of bins, for which a difference between measures of error specified by the first performance data and the second performance data exceeds an average difference between measures of error across the plurality of bins;
determining to update the trained ML model when the number of bins exceeds a predetermined threshold number of bins; and
determining to generate a supplemental ML model when the number of bins is less or equal to the pre-determined threshold number of bins”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. In particular, the claim does not recite additional elements. Thus, the claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, the claim is not patent eligible.
Regarding claim 18
Step 1: The claim recites a method; therefore, it falls into the statutory category of processes.
Step 2A Prong 1:
The limitations of
“… ,
wherein the method further comprises:
identifying, using the first representation, the second representation, the first performance data and the second performance data, one or more bins for which to obtain additional inputs and corresponding ground truth values for improving performance of the trained ML model”, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, the limitations in the context of this claim encompass the user mentally thinking with a physical aid (e.g., pencil and paper).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
In particular, the claim recites an additional element (“wherein the first representation of the first distribution of the first plurality of inputs comprises a first histogram having a plurality of bins, wherein the second representation of the second distribution of the second plurality of inputs comprises a second histogram having the plurality of bins”). This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not integrate the abstract idea into a practical application. See MPEP 2106.05(h)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
This is a recitation of a particular type or source of model/data to be used in performing the abstract idea. Limiting the abstract idea to a particular type or source of model/data is an attempt to limit the abstract idea to a particular field of use or technological environment, which does not amount to significantly more than the abstract idea. See MPEP 2106.05(h).
Regarding claim 19
The claim recites “A system, comprising: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:” to perform precisely the method of Claim 1. As performance of an abstract idea on generic computer components (see MPEP 2106.05(f)) cannot integrate the abstract idea into a practical application nor provide significantly more than the abstract idea itself, the claim is rejected for reasons set forth in the rejection of Claim 1.
Regarding claim 20
The claim recites “At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform:” to perform precisely the method of Claim 1. As performance of an abstract idea on generic computer components (see MPEP 2106.05(f)) and “Storing and retrieving information in memory” (see MPEP 2106.05(g) on Insignificant Extra-Solution Activity, and MPEP 2106.05(d) on Well-Understood, Routine, Conventional Activity) cannot integrate the abstract idea into a practical application nor provide significantly more than the abstract idea itself, the claim is rejected for reasons set forth in the rejection of Claim 1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 13-15, 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Boulanger et al. (US 2021/0133510 A1) in view of Frosch et al. (US 20220335328 A1)
Regarding claim 1
Boulanger teaches
A method, comprising:
using at least one computer hardware processor to perform:
(Boulanger [fig(s) 3] [par(s) 39] “As illustrated, the image classification system 200 includes one or more processors 204 that perform data processing and/or other software execution operations for the image classification system 200. The processor 204 may include logic devices, microcontrollers, processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other devices that may be used by the image classification system 200 to execute appropriate instructions, such as software instructions stored in memory 206 including 3D simulation and image capture component 208, training dataset generation component 210, and image classification component 212 (e.g., a neural network trained by the training dataset), and/or other applications. The memory 206 may be implemented in one or more memory devices (e.g., memory components) that store executable instructions, data and information, including image data, video data, audio data, network information. The memory devices may include various types of memory for information storage including volatile and non-volatile memory devices, such as RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically-Erasable Read-Only Memory), flash memory, a disk drive, and other types of memory described herein.”;)
(A) obtaining information about training data used to generate a trained machine learning (ML) model, the training data comprising a first plurality of inputs and a corresponding first plurality of outputs, the first plurality of inputs being from a first portion of an input data domain, the information about the training data comprising:
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label. If the neural network 330 mislabels the input image (e.g., determines that the image is a “rhinoceros” instead of an “elephant”), then a backward pass through the neural network 330 may be used to adjust the neural network to correct for the misclassification. Referring to FIG. 4D, a trained neural network 340, may then be implemented in an application (i.e., a neural network inference application) on a run time environment to classify thermal images 342.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.”;)
a first representation of a first distribution of the first plurality of inputs, and
(Boulanger [fig(s) 1] [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.” [par(s) 28-33] “The scene generator 104 is operable to build a virtual 3D environment utilizing data from an object database 112, which stores 3D models and other object data allowing a modeled 3D object to be placed in the scene. The scene generator 104 may also apply environmental effects 114 (such as weather conditions, temperature, time of day, etc.). The environment simulation system 102 may optionally include an infrared sensor simulator/image capture component 106 for capturing infrared images of a scene and/or an optical image capture component 108 for capturing visible images of a generated scene.” [par(s) 22-23] “The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. … image characteristics (e.g., image size, object size, features extracted)” [par(s) 43] “The training dataset includes synthetic images as described herein, and may also include real images captured from an infrared, visible light, or other type of camera.”;)
first performance data indicative of a measure of performance of the trained ML model on the first plurality of inputs;
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.”;)
(B) obtaining information about new data to which the trained ML model was applied, the new data comprising a second plurality of inputs and a corresponding second plurality of outputs, the information about the new data comprising:
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label. If the neural network 330 mislabels the input image (e.g., determines that the image is a “rhinoceros” instead of an “elephant”), then a backward pass through the neural network 330 may be used to adjust the neural network to correct for the misclassification. Referring to FIG. 4D, a trained neural network 340, may then be implemented in an application (i.e., a neural network inference application) on a run time environment to classify thermal images 342.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results”; e.g., newly-generated synthetic images along with fig 1 read(s) on “new data to which the trained ML model was applied”.)
a second representation of a second distribution of the second plurality of inputs, and
(Boulanger [fig(s) 1] [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.” [par(s) 28-33] “The scene generator 104 is operable to build a virtual 3D environment utilizing data from an object database 112, which stores 3D models and other object data allowing a modeled 3D object to be placed in the scene. The scene generator 104 may also apply environmental effects 114 (such as weather conditions, temperature, time of day, etc.). The environment simulation system 102 may optionally include an infrared sensor simulator/image capture component 106 for capturing infrared images of a scene and/or an optical image capture component 108 for capturing visible images of a generated scene.” [par(s) 22-23] “The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. … image characteristics (e.g., image size, object size, features extracted)” [par(s) 43] “The training dataset includes synthetic images as described herein, and may also include real images captured from an infrared, visible light, or other type of camera.”;)
second performance data indicative of a measure of performance of the trained ML model on the second plurality of inputs;
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.”;)
(C) determining, using the first representation, the second representation, the first performance data, and the second performance data, whether to update the trained ML model or to generate a supplemental ML model to use with the trained ML model, the determining of step (C) comprising:
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”;)
[comparing] the first representation to the second representation to assess [a difference between] the first representation and the second representation;
(Boulanger [fig(s) 1] [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.” [par(s) 28-33] “The scene generator 104 is operable to build a virtual 3D environment utilizing data from an object database 112, which stores 3D models and other object data allowing a modeled 3D object to be placed in the scene. The scene generator 104 may also apply environmental effects 114 (such as weather conditions, temperature, time of day, etc.). The environment simulation system 102 may optionally include an infrared sensor simulator/image capture component 106 for capturing infrared images of a scene and/or an optical image capture component 108 for capturing visible images of a generated scene.” [par(s) 22-23] “The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. … image characteristics (e.g., image size, object size, features extracted)” [par(s) 43] “The training dataset includes synthetic images as described herein, and may also include real images captured from an infrared, visible light, or other type of camera.”;)
[comparing] the first performance data to the second performance data to assess [a difference between] the first performance data and the second performance data; and
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.”;)
determining, based on the assessed [differences], whether to update the trained ML model or to generate the supplemental ML model to use with the trained ML model;
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”;)
(D) adapting the trained ML model based on the determining of step (C), wherein the adapting comprises:
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”;)
when it is determined to update the trained ML model or to generate the supplemental ML model to use with the trained ML model, either updating the trained ML model through additional training to generate an updated ML model or generating the supplemental ML model, by training the supplemental ML model, to use with the trained ML model, wherein the adapting further comprises generating the supplemental ML model; and
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”;)
(E) determining whether to provide new input data as input to the trained ML model or the supplemental ML model, the determining of step (E) comprising:
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56.”;)
determining whether the new input data is in a first portion of the input data domain or in a second portion of the input data domain;
(Boulanger [fig(s) 1] [par(s) 19-22] “The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 25] “In some embodiments, the dataset generator 74 determines a subset of images from the training dataset 56 to maintain in the training dataset and defines new images to be generated. In some embodiments, images from the training dataset 56 may be ranked on performance results by ranking each image's impact based on overall performance. For example, the dataset generator 74 may keep a certain number of top ranked images for each image classification, keep images that contribute above an identified performance threshold, and/or keep a certain number of top ranked images overall. The dataset generation module 74 may also remove images from the training dataset 56 that are lowest ranked and/or contribute negatively or below an identified performance threshold. For example, the training dataset analysis engine 70 may ranked 1000 images, for example, and keep the best 100 images, remove the remaining 900 images. The new training dataset can include the previous 100 images and the newly generated 900 synthetic images which are selected to improve performance. Various parameters may represent a general categories for random scene generation including foreground object, size, type and angle; the position and type of the camera (e.g., visible camera, infrared imaging device); and presence of background assets.”;)
when it is determined that the new input data is in the first portion of the input data domain, providing the new input data as input to the trained ML model to obtain a first corresponding output; and
when it is determined that the new input data is in the second portion of the input data domain, providing the new input data as input to the supplemental ML model to obtain a second corresponding output.
(Boulanger [fig(s) 1] [par(s) 19-22] “The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 25] “In some embodiments, the dataset generator 74 determines a subset of images from the training dataset 56 to maintain in the training dataset and defines new images to be generated. In some embodiments, images from the training dataset 56 may be ranked on performance results by ranking each image's impact based on overall performance. For example, the dataset generator 74 may keep a certain number of top ranked images for each image classification, keep images that contribute above an identified performance threshold, and/or keep a certain number of top ranked images overall. The dataset generation module 74 may also remove images from the training dataset 56 that are lowest ranked and/or contribute negatively or below an identified performance threshold. For example, the training dataset analysis engine 70 may ranked 1000 images, for example, and keep the best 100 images, remove the remaining 900 images. The new training dataset can include the previous 100 images and the newly generated 900 synthetic images which are selected to improve performance. Various parameters may represent a general categories for random scene generation including foreground object, size, type and angle; the position and type of the camera (e.g., visible camera, infrared imaging device); and presence of background assets.”;)
However, the combination of Boulanger does not appear to explicitly teach:
[comparing] the first representation to the second representation to assess [a difference between] the first representation and the second representation;
[comparing] the first performance data to the second performance data to assess [a difference between] the first performance data and the second performance data; and
determining, based on the assessed [differences], whether to update the trained ML model or to generate the supplemental ML model to use with the trained ML model;
Frosch teaches
comparing the first representation to the second representation to assess a difference between the first representation and the second representation;
(Frosch [par(s) 20] “The accuracy of the determinations produced by the machine learning model when deployed in the field can differ from that exhibited by the machine learning model during training and/or validation. In various cases, this performance disparity can be due to differences between the input data encountered by the machine learning model during deployment and the input data encountered by the machine learning model during training and/or validation. For example, a machine learning model can be configured to receive as input an MRI scanned image associated with a patient and to produce as output a diagnosis based on the MRI scanned image. In various cases, the machine learning model can be trained and/or validated on datasets that comprise MRI scanned images associated with patients having certain demographics (e.g., patients within a certain age range, patients of certain ethnicities, patients without comorbidities). In various aspects, once deployed in the field, the machine leaning model can encounter MRI scanned images associated with patients that fall outside of these certain demographics (e.g., patients outside of the certain age range, patients not belonging to the certain ethnicities, patients with comorbidities). Because the machine learning model did not encounter such MRI scanned images during training and/or validation, the machine learning model can exhibit reduced diagnosis accuracy when encountering such MRI scanned images during deployment. Such differences in input data can exist because it can be impracticable to represent within a training dataset and/or a validation dataset the full range of all possible input data variations which can be encountered in the field.” [par(s) 80] “In various instances, the augmentation component 122 can apply any suitable augmentations to the one or more anonymized copies of the data candidate 302. For example, if the data candidate 302 is an image, the augmentation component 122 can modify the one or more anonymized copies of the data candidate 302 by randomly rotating various copies, randomly altering the zoom-level and/or magnification level of various copies, randomly applying a Gaussian blur to various copies, randomly inserting images of various objects and/or artefacts into various copies, and/or randomly applying various optical distortions to various copies. In various instances, the one or more augmented and anonymized copies of the data candidate 302 can be considered as the set of synthetic data candidates 502.”;)
comparing the first performance data to the second performance data to assess a difference between the first performance data and the second performance data; and
(Frosch [par(s) 20] “The accuracy of the determinations produced by the machine learning model when deployed in the field can differ from that exhibited by the machine learning model during training and/or validation. In various cases, this performance disparity can be due to differences between the input data encountered by the machine learning model during deployment and the input data encountered by the machine learning model during training and/or validation. For example, a machine learning model can be configured to receive as input an MRI scanned image associated with a patient and to produce as output a diagnosis based on the MRI scanned image. In various cases, the machine learning model can be trained and/or validated on datasets that comprise MRI scanned images associated with patients having certain demographics (e.g., patients within a certain age range, patients of certain ethnicities, patients without comorbidities). In various aspects, once deployed in the field, the machine leaning model can encounter MRI scanned images associated with patients that fall outside of these certain demographics (e.g., patients outside of the certain age range, patients not belonging to the certain ethnicities, patients with comorbidities). Because the machine learning model did not encounter such MRI scanned images during training and/or validation, the machine learning model can exhibit reduced diagnosis accuracy when encountering such MRI scanned images during deployment. Such differences in input data can exist because it can be impracticable to represent within a training dataset and/or a validation dataset the full range of all possible input data variations which can be encountered in the field.” [par(s) 80] “In various instances, the augmentation component 122 can apply any suitable augmentations to the one or more anonymized copies of the data candidate 302. For example, if the data candidate 302 is an image, the augmentation component 122 can modify the one or more anonymized copies of the data candidate 302 by randomly rotating various copies, randomly altering the zoom-level and/or magnification level of various copies, randomly applying a Gaussian blur to various copies, randomly inserting images of various objects and/or artefacts into various copies, and/or randomly applying various optical distortions to various copies. In various instances, the one or more augmented and anonymized copies of the data candidate 302 can be considered as the set of synthetic data candidates 502.”;)
determining, based on the assessed differences, whether to update the trained ML model or to generate the supplemental ML model to use with the trained ML model;
(Frosch [par(s) 39] “In various embodiments, the training component can electronically train (e.g., via supervised training) the particular deployed machine learning model based on the one or more synthetic data candidates. More specifically, the training component can electronically feed to the particular deployed machine learning model the one or more synthetic data candidates, thereby causing the particular deployed machine learning model to output results. In various cases, the training component can compute errors and/or losses based on differences between the particular expert conclusion and such outputted results (e.g., the particular expert conclusion can be considered as the annotation associated with the one or more synthetic data candidates). In various instances, the training component can electronically update (e.g., via backpropagation) parameters (e.g., weights and/or biases) of the particular deployed machine learning model based on such computed errors and/or losses. In this way, the training component can iteratively improve the performance of the particular deployed machine learning model based on the one or more synthetic data candidates. Those having ordinary skill in the art will appreciate that, in some cases, unsupervised training and/or reinforcement learning can be implemented, rather than supervised training.”;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Boulanger with the data differences of Frosch.
One of ordinary skill in the art would have been motived to combine in order to improve performance of the deployed machine learning model by leveraging failure data.
(Frosch [par(s) 23] “the inventors devised various embodiments of the subject innovation so as to automatically capture input data candidates which cause a deployed machine learning model to generate inaccurate determinations. This can be beneficial because failure data (e.g., input data candidates which cause a deployed machine learning model to produce incorrect inferences) can be valuable from a development and/or training perspective. Specifically, such failure data can be leveraged so as to improve the performance of the deployed machine learning model.”)
Regarding claim 2
The combination of Boulanger, Frosch teaches claim 1.
Boulanger further teaches
wherein (C) comprises determining to update the trained ML model and (D) comprises updating the trained ML model to generate the updated ML model.
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”;)
Regarding claim 3
The combination of Boulanger, Frosch teaches claim 2.
wherein updating the trained ML model comprises: (See claim 1)
Boulanger further teaches
training, using at least some of the new data, a second trained ML model;
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”;)
generating the updated ML model as an ensemble of the trained ML model and the second trained ML model.
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”; e.g., “The process repeats in an iterative fashion … in training an inference CNN” along with fig 1 read(s) on “an ensemble of the trained ML model and the second trained ML model” since a current ML model is used for generating a next ML model iteratively and together.)
Regarding claim 4
The combination of Boulanger, Frosch teaches claim 3.
wherein generating the updated ML model further comprises (See claim 1)
Boulanger further teaches
determining weights for the trained ML model and the second trained ML model in the ensemble.
(Boulanger [fig(s) 1] [par(s) 43] “the training starts with a forward pass through the neural network 300 including feature extraction 304 in a plurality of convolution layers 306 and pooling layers 308, followed by image classification 310 in a plurality of fully connected layers 312 and an output layer 314. Next, a backward pass through the neural network 300 may be used to update the CNN parameters in view of errors produced in the forward pass (e.g., misclassified objects).” [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”; e.g., “The process repeats in an iterative fashion … in training an inference CNN” along with fig 1 read(s) on “the trained ML model and the second trained ML model in the ensemble” since a current ML model is used for generating a next ML model iteratively and together. In addition, e.g., updating “CNN parameters” based on the forward pass and backward pass read(s) on “determining weights”.)
Regarding claim 5
The combination of Boulanger, Frosch teaches claim 2.
Boulanger further teaches
obtaining information about second training data used to generate the updated ML model, the second training data comprising a third plurality of inputs and a corresponding third plurality of outputs, the information about the second training data comprising:
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label.” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.”;)
a third representation of a third distribution of the third plurality of inputs, and
(Boulanger [fig(s) 1] [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.” [par(s) 28-33] “The scene generator 104 is operable to build a virtual 3D environment utilizing data from an object database 112, which stores 3D models and other object data allowing a modeled 3D object to be placed in the scene. The scene generator 104 may also apply environmental effects 114 (such as weather conditions, temperature, time of day, etc.). The environment simulation system 102 may optionally include an infrared sensor simulator/image capture component 106 for capturing infrared images of a scene and/or an optical image capture component 108 for capturing visible images of a generated scene.” [par(s) 22-23] “The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. … image characteristics (e.g., image size, object size, features extracted)” [par(s) 43] “The training dataset includes synthetic images as described herein, and may also include real images captured from an infrared, visible light, or other type of camera.”;)
third performance data indicative of a measure of performance of the updated ML model on the third plurality of inputs;
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.”;)
obtaining information about second new data to which the updated ML model was applied, the second new data comprising a fourth plurality of inputs and a corresponding fourth plurality of outputs, the information about the second new data comprising:
(Boulanger [fig(s) 1] [par(s) 45] “Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label. If the neural network 330 mislabels the input image (e.g., determines that the image is a “rhinoceros” instead of an “elephant”), then a backward pass through the neural network 330 may be used to adjust the neural network to correct for the misclassification. Referring to FIG. 4D, a trained neural network 340, may then be implemented in an application (i.e., a neural network inference application) on a run time environment to classify thermal images 342.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”; e.g., newly-generated synthetic images along with fig 1 read(s) on “second new data to which the updated ML model was applied”.)
a fourth representation of a fourth distribution of the fourth plurality of inputs, and
(Boulanger [fig(s) 1] [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.” [par(s) 28-33] “The scene generator 104 is operable to build a virtual 3D environment utilizing data from an object database 112, which stores 3D models and other object data allowing a modeled 3D object to be placed in the scene. The scene generator 104 may also apply environmental effects 114 (such as weather conditions, temperature, time of day, etc.). The environment simulation system 102 may optionally include an infrared sensor simulator/image capture component 106 for capturing infrared images of a scene and/or an optical image capture component 108 for capturing visible images of a generated scene.” [par(s) 22-23] “The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. … image characteristics (e.g., image size, object size, features extracted)” [par(s) 43] “The training dataset includes synthetic images as described herein, and may also include real images captured from an infrared, visible light, or other type of camera.”;)
fourth performance data indicative of a measure of performance of the updated ML model on the fourth plurality of inputs;
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.”;)
determining, using the third representation, the fourth representation, the third performance data, and the fourth performance data, whether to further update the updated ML model or to generate a further supplemental ML model to use with the updated ML model; and
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”;)
when it is determined to further update the updated ML model or to generate the further supplemental ML model to use with the updated ML model, updating the updated ML model or generating the further supplemental ML model.
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”;)
Regarding claim 13
The combination of Boulanger, Frosch teaches claim 1.
Boulanger further teaches
prior to obtaining the information about the new data, applying the trained ML model to the second plurality of inputs to obtain the second plurality of outputs.
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label. If the neural network 330 mislabels the input image (e.g., determines that the image is a “rhinoceros” instead of an “elephant”), then a backward pass through the neural network 330 may be used to adjust the neural network to correct for the misclassification. Referring to FIG. 4D, a trained neural network 340, may then be implemented in an application (i.e., a neural network inference application) on a run time environment to classify thermal images 342.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results”; e.g., fig 1 read(s) on “prior to obtaining the information about the new data, applying the trained ML model to the second plurality of inputs to obtain the second plurality of outputs” since “Training Dataset Analysis Engine 70” obtains “Informative metrics” and “Performance results” after training and validation.)
Regarding claim 14
The combination of Boulanger, Frosch teaches claim 1.
Boulanger further teaches
prior to obtaining the information about the training data, training, using the first plurality of inputs and the first plurality of outputs, an untrained ML model to generate the trained ML model.
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label. If the neural network 330 mislabels the input image (e.g., determines that the image is a “rhinoceros” instead of an “elephant”), then a backward pass through the neural network 330 may be used to adjust the neural network to correct for the misclassification. Referring to FIG. 4D, a trained neural network 340, may then be implemented in an application (i.e., a neural network inference application) on a run time environment to classify thermal images 342.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.”; e.g., fig 1 read(s) on “prior to obtaining the information about the training data, training, using the first plurality of inputs and the first plurality of outputs, an untrained ML model to generated the trained ML model” since “Training Dataset Analysis Engine 70” obtains “Informative metrics” and “Performance results” after training.)
Regarding claim 15
The combination of Boulanger, Frosch teaches claim 1.
wherein the determining comprises: (See claim 1)
determining a first value based on the comparison between the first representation and the second representation;
(Frosch [par(s) 20] “The accuracy of the determinations produced by the machine learning model when deployed in the field can differ from that exhibited by the machine learning model during training and/or validation. In various cases, this performance disparity can be due to differences between the input data encountered by the machine learning model during deployment and the input data encountered by the machine learning model during training and/or validation. For example, a machine learning model can be configured to receive as input an MRI scanned image associated with a patient and to produce as output a diagnosis based on the MRI scanned image. In various cases, the machine learning model can be trained and/or validated on datasets that comprise MRI scanned images associated with patients having certain demographics (e.g., patients within a certain age range, patients of certain ethnicities, patients without comorbidities). In various aspects, once deployed in the field, the machine leaning model can encounter MRI scanned images associated with patients that fall outside of these certain demographics (e.g., patients outside of the certain age range, patients not belonging to the certain ethnicities, patients with comorbidities). Because the machine learning model did not encounter such MRI scanned images during training and/or validation, the machine learning model can exhibit reduced diagnosis accuracy when encountering such MRI scanned images during deployment. Such differences in input data can exist because it can be impracticable to represent within a training dataset and/or a validation dataset the full range of all possible input data variations which can be encountered in the field.” [par(s) 80] “In various instances, the augmentation component 122 can apply any suitable augmentations to the one or more anonymized copies of the data candidate 302. For example, if the data candidate 302 is an image, the augmentation component 122 can modify the one or more anonymized copies of the data candidate 302 by randomly rotating various copies, randomly altering the zoom-level and/or magnification level of various copies, randomly applying a Gaussian blur to various copies, randomly inserting images of various objects and/or artefacts into various copies, and/or randomly applying various optical distortions to various copies. In various instances, the one or more augmented and anonymized copies of the data candidate 302 can be considered as the set of synthetic data candidates 502.”; e.g., “set of synthetic data candidates” read(s) on “first value”.)
determining a second value based on the comparison between the first performance data and the second performance data; and
(Frosch [par(s) 20] “The accuracy of the determinations produced by the machine learning model when deployed in the field can differ from that exhibited by the machine learning model during training and/or validation. In various cases, this performance disparity can be due to differences between the input data encountered by the machine learning model during deployment and the input data encountered by the machine learning model during training and/or validation. For example, a machine learning model can be configured to receive as input an MRI scanned image associated with a patient and to produce as output a diagnosis based on the MRI scanned image. In various cases, the machine learning model can be trained and/or validated on datasets that comprise MRI scanned images associated with patients having certain demographics (e.g., patients within a certain age range, patients of certain ethnicities, patients without comorbidities). In various aspects, once deployed in the field, the machine leaning model can encounter MRI scanned images associated with patients that fall outside of these certain demographics (e.g., patients outside of the certain age range, patients not belonging to the certain ethnicities, patients with comorbidities). Because the machine learning model did not encounter such MRI scanned images during training and/or validation, the machine learning model can exhibit reduced diagnosis accuracy when encountering such MRI scanned images during deployment. Such differences in input data can exist because it can be impracticable to represent within a training dataset and/or a validation dataset the full range of all possible input data variations which can be encountered in the field.” [par(s) 80] “In various instances, the augmentation component 122 can apply any suitable augmentations to the one or more anonymized copies of the data candidate 302. For example, if the data candidate 302 is an image, the augmentation component 122 can modify the one or more anonymized copies of the data candidate 302 by randomly rotating various copies, randomly altering the zoom-level and/or magnification level of various copies, randomly applying a Gaussian blur to various copies, randomly inserting images of various objects and/or artefacts into various copies, and/or randomly applying various optical distortions to various copies. In various instances, the one or more augmented and anonymized copies of the data candidate 302 can be considered as the set of synthetic data candidates 502.”; e.g., “set of synthetic data candidates” read(s) on “second value”.)
determining, based on a weighted combination of the first value and the second value, whether to update the trained ML model or to generate a supplemental ML model to use with the trained ML model.
(Frosch [par(s) 39] “In various embodiments, the training component can electronically train (e.g., via supervised training) the particular deployed machine learning model based on the one or more synthetic data candidates. More specifically, the training component can electronically feed to the particular deployed machine learning model the one or more synthetic data candidates, thereby causing the particular deployed machine learning model to output results. In various cases, the training component can compute errors and/or losses based on differences between the particular expert conclusion and such outputted results (e.g., the particular expert conclusion can be considered as the annotation associated with the one or more synthetic data candidates). In various instances, the training component can electronically update (e.g., via backpropagation) parameters (e.g., weights and/or biases) of the particular deployed machine learning model based on such computed errors and/or losses. In this way, the training component can iteratively improve the performance of the particular deployed machine learning model based on the one or more synthetic data candidates. Those having ordinary skill in the art will appreciate that, in some cases, unsupervised training and/or reinforcement learning can be implemented, rather than supervised training.”; e.g., “train (e.g., via supervised training) the particular deployed machine learning model based on the one or more synthetic data candidates” read(s) on “weighted combination of the first value and the second value” since each image is used for training together in combination.)
The combination of Boulanger, Frosch is combinable with Frosch for the same rationale as set forth above with respect to claim 1.
Regarding claim 19
The claim is a system claim corresponding to the method claim 1, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 20
The claim is a computer-readable storage medium claim corresponding to the method claim 1, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Claim(s) 7, 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Boulanger et al. (US 2021/0133510 A1) in view of Frosch et al. (US 20220335328 A1) in view of Cruz et al. (META-DES: A dynamic ensemble selection framework using meta-learning)
Regarding claim 7
The combination of Boulanger, Frosch teaches claim 1.
wherein generating the supplemental ML model comprises: (See claim 1)
Boulanger teaches
training, using at least some of the new data, the [supplemental] ML model;
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”;)
associating the trained ML model with the first portion of an input data domain; and
(Boulanger [fig(s) 1] [par(s) 19-22] “The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 25] “In some embodiments, the dataset generator 74 determines a subset of images from the training dataset 56 to maintain in the training dataset and defines new images to be generated. In some embodiments, images from the training dataset 56 may be ranked on performance results by ranking each image's impact based on overall performance. For example, the dataset generator 74 may keep a certain number of top ranked images for each image classification, keep images that contribute above an identified performance threshold, and/or keep a certain number of top ranked images overall. The dataset generation module 74 may also remove images from the training dataset 56 that are lowest ranked and/or contribute negatively or below an identified performance threshold. For example, the training dataset analysis engine 70 may ranked 1000 images, for example, and keep the best 100 images, remove the remaining 900 images. The new training dataset can include the previous 100 images and the newly generated 900 synthetic images which are selected to improve performance. Various parameters may represent a general categories for random scene generation including foreground object, size, type and angle; the position and type of the camera (e.g., visible camera, infrared imaging device); and presence of background assets.”;)
associating the [supplemental] ML model with the second portion of the input data domain.
(Boulanger [fig(s) 1] [par(s) 19-22] “The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 25] “In some embodiments, the dataset generator 74 determines a subset of images from the training dataset 56 to maintain in the training dataset and defines new images to be generated. In some embodiments, images from the training dataset 56 may be ranked on performance results by ranking each image's impact based on overall performance. For example, the dataset generator 74 may keep a certain number of top ranked images for each image classification, keep images that contribute above an identified performance threshold, and/or keep a certain number of top ranked images overall. The dataset generation module 74 may also remove images from the training dataset 56 that are lowest ranked and/or contribute negatively or below an identified performance threshold. For example, the training dataset analysis engine 70 may ranked 1000 images, for example, and keep the best 100 images, remove the remaining 900 images. The new training dataset can include the previous 100 images and the newly generated 900 synthetic images which are selected to improve performance. Various parameters may represent a general categories for random scene generation including foreground object, size, type and angle; the position and type of the camera (e.g., visible camera, infrared imaging device); and presence of background assets.”;)
However, the combination of Boulanger, Frosch does not appear to explicitly teach:
training, using at least some of the new data, the [supplemental] ML model;
associating the [supplemental] ML model with the second portion of the input data domain.
Cruz teaches
training, using at least some of the new data, the supplemental ML model;
associating the supplemental ML model with the second portion of the input data domain.
(Cruz [sec(s) Abs] “Dynamic ensemble selection systems work by estimating the level of competence of each classifier from a pool of classifiers. Only the most competent ones are selected to classify a given test sample. This is achieved by defining a criterion to measure the level of competence of a base classifier, such as, its accuracy in local regions of the feature space around the query instance. … We propose five distinct sets of meta-features, each one corresponding to a different criterion to measure the level of competence of a classifier for the classification of input samples.” [sec(s) 5] “Each meta-feature fi corresponds to a different criterion to measure the level of competence of a base classifier.” [sec(s) 1] “The proposed framework is divided into three phases: over production, meta-training and generalization. In the overproduction stage, a pool of classifiers is generated using the training data. In the meta-training stage, the five sets of meta-features are extracted from the training data, and are used to train the meta classifier that works as the classifier selector. During the generalization phase, the meta-features are extracted from the query instance and passed down as inputs to the meta-classifier. The meta-classifier estimates whether a base classifier is competent enough to classify the given test instance. Thus, the proposed system differs from the current state-of-the-art dynamic selection techniques not only because it uses multiple criteria to perform the classifier selection, but also because the classifier selection rule is learned by the meta-classifier using the training data.”;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Boulanger, Frosch with the supplemental ML model of Cruz.
One of ordinary skill in the art would have been motived to combine in order to provide a significant improvement in performance for datasets with critical training size samples.
(Cruz [sec(s) 5] “In addition, we observed a significant improvement in performance for datasets with critical training size samples. This gain in accuracy can be explained by the fact that during the Meta Training phase of the framework, each training sample generates several meta-feature vectors for the training of the meta-classifier.”)
Regarding claim 10
The combination of Boulanger, Frosch teaches claim 1.
Boulanger teaches
obtaining information about second training data used to generate the [supplemental] ML model, the second training data comprising a third plurality of inputs and a corresponding third plurality of outputs, the information about the second training data comprising:
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label. If the neural network 330 mislabels the input image (e.g., determines that the image is a “rhinoceros” instead of an “elephant”), then a backward pass through the neural network 330 may be used to adjust the neural network to correct for the misclassification. Referring to FIG. 4D, a trained neural network 340, may then be implemented in an application (i.e., a neural network inference application) on a run time environment to classify thermal images 342.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.”;)
a third representation of a third distribution of the third plurality of inputs, and
(Boulanger [fig(s) 1] [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.” [par(s) 28-33] “The scene generator 104 is operable to build a virtual 3D environment utilizing data from an object database 112, which stores 3D models and other object data allowing a modeled 3D object to be placed in the scene. The scene generator 104 may also apply environmental effects 114 (such as weather conditions, temperature, time of day, etc.). The environment simulation system 102 may optionally include an infrared sensor simulator/image capture component 106 for capturing infrared images of a scene and/or an optical image capture component 108 for capturing visible images of a generated scene.” [par(s) 22-23] “The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. … image characteristics (e.g., image size, object size, features extracted)” [par(s) 43] “The training dataset includes synthetic images as described herein, and may also include real images captured from an infrared, visible light, or other type of camera.”;)
third performance data indicative of a measure of performance of the [supplemental] ML model on the third plurality of inputs;
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.”;)
obtaining information about second new data to which the [supplemental] ML model was applied, the second new data comprising a fourth plurality of inputs and a corresponding fourth plurality of outputs, the information about the second new data comprising:
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label. If the neural network 330 mislabels the input image (e.g., determines that the image is a “rhinoceros” instead of an “elephant”), then a backward pass through the neural network 330 may be used to adjust the neural network to correct for the misclassification. Referring to FIG. 4D, a trained neural network 340, may then be implemented in an application (i.e., a neural network inference application) on a run time environment to classify thermal images 342.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results”; e.g., newly-generated synthetic images along with fig 1 read(s) on “new data to which the trained ML model was applied”.)
a fourth representation of a fourth distribution of the fourth plurality of inputs, and
(Boulanger [fig(s) 1] [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.” [par(s) 28-33] “The scene generator 104 is operable to build a virtual 3D environment utilizing data from an object database 112, which stores 3D models and other object data allowing a modeled 3D object to be placed in the scene. The scene generator 104 may also apply environmental effects 114 (such as weather conditions, temperature, time of day, etc.). The environment simulation system 102 may optionally include an infrared sensor simulator/image capture component 106 for capturing infrared images of a scene and/or an optical image capture component 108 for capturing visible images of a generated scene.” [par(s) 22-23] “The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. … image characteristics (e.g., image size, object size, features extracted)” [par(s) 43] “The training dataset includes synthetic images as described herein, and may also include real images captured from an infrared, visible light, or other type of camera.”;)
fourth performance data indicative of a measure of performance of the [supplemental] ML model on the fourth plurality of inputs;
(Boulanger [fig(s) 1] [par(s) 45] “Referring to FIG. 4C, further details of an embodiment for training a neural network utilizing synthetic training data will now be described. A neural network 320, such as a convolutional neural network, is trained using a training dataset 332 that includes synthetic images as described herein. The training includes with a forward pass through the neural network 330 to produce an image classification. In the illustrated embodiment, a thermal image such as a synthetic thermal image of an elephant is fed to the neural network 330 to produce a classification at the output layer. Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label.” [par(s) 21] “The training dataset analysis engine 70 may then analyze the received data to modify the training dataset 56 by identifying images to keep (e.g., images that contribute to proper classification), drop from (e.g., images that do not contributed to proper classification) and/or add to the training dataset 56. In one or more embodiments, the training dataset analysis engine 70 receives the informative metrics and performance results, analyzes the available data in view of the configuration parameters, and instructs the synthetic image generator 52 to produce an updated training dataset 56 that is predicted to train an inference CNN with improved results.”;)
determining, using the third representation, the fourth representation, the third performance data, and the fourth performance data, whether to further update the [supplemental] ML model or to generate a second supplemental ML model to use with the trained ML model and the supplemental ML model; and
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”;)
when it is determined to further update the [supplemental] ML model or to generate the second supplemental ML model to use with the trained ML model and the supplemental ML model, updating the [supplemental] ML model or generating the second supplemental ML model.
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN”;)
However, the combination of Boulanger, Frosch does not appear to explicitly teach:
obtaining information about second training data used to generate the [supplemental] ML model,
third performance data indicative of a measure of performance of the [supplemental] ML model on the third plurality of inputs;
obtaining information about second new data to which the [supplemental] ML model was applied,
fourth performance data indicative of a measure of performance of the [supplemental] ML model on the fourth plurality of inputs;
determining, using the third representation, the fourth representation, the third performance data, and the fourth performance data, whether to further update the [supplemental] ML model or to generate a second supplemental ML model to use with the trained ML model and the supplemental ML model; and
when it is determined to further update the [supplemental] ML model or to generate the second supplemental ML model to use with the trained ML model and the supplemental ML model, updating the [supplemental] ML model or generating the second supplemental ML model.
Cruz teaches
obtaining information about second training data used to generate the supplemental ML model,
third performance data indicative of a measure of performance of the supplemental ML model on the third plurality of inputs;
obtaining information about second new data to which the supplemental ML model was applied,
fourth performance data indicative of a measure of performance of the supplemental ML model on the fourth plurality of inputs;
determining, using the third representation, the fourth representation, the third performance data, and the fourth performance data, whether to further update the supplemental ML model or to generate a second supplemental ML model to use with the trained ML model and the supplemental ML model; and
when it is determined to further update the supplemental ML model or to generate the second supplemental ML model to use with the trained ML model and the supplemental ML model, updating the supplemental ML model or generating the second supplemental ML model.
(Cruz [sec(s) Abs] “Dynamic ensemble selection systems work by estimating the level of competence of each classifier from a pool of classifiers. Only the most competent ones are selected to classify a given test sample. This is achieved by defining a criterion to measure the level of competence of a base classifier, such as, its accuracy in local regions of the feature space around the query instance. … We propose five distinct sets of meta-features, each one corresponding to a different criterion to measure the level of competence of a classifier for the classification of input samples.” [sec(s) 5] “Each meta-feature fi corresponds to a different criterion to measure the level of competence of a base classifier.” [sec(s) 1] “The proposed framework is divided into three phases: over production, meta-training and generalization. In the overproduction stage, a pool of classifiers is generated using the training data. In the meta-training stage, the five sets of meta-features are extracted from the training data, and are used to train the meta classifier that works as the classifier selector. During the generalization phase, the meta-features are extracted from the query instance and passed down as inputs to the meta-classifier. The meta-classifier estimates whether a base classifier is competent enough to classify the given test instance. Thus, the proposed system differs from the current state-of-the-art dynamic selection techniques not only because it uses multiple criteria to perform the classifier selection, but also because the classifier selection rule is learned by the meta-classifier using the training data.”;)
The combination of Boulanger, Frosch is combinable with Cruz for the same rationale as set forth above with respect to claim 7.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Boulanger et al. (US 2021/0133510 A1) in view of Frosch et al. (US 20220335328 A1) in view of Cruz et al. (META-DES: A dynamic ensemble selection framework using meta-learning) in view of Bolukbasi et al. (Adaptive Neural Networks for Efficient Inference)
Regarding claim 9
The combination of Boulanger, Frosch teaches claim 1.
However, the combination of Boulanger, Frosch does not appear to explicitly teach:
monitoring performance of the trained ML model and the supplemental ML model; and
adjusting portions of the input data domain associated with the trained ML model and the supplemental ML model based on results of the monitoring,
wherein adjusting the portions comprises adjusting a boundary between the first portion of the input data domain and the second portion of the input data domain thereby changing the first portion and the second portion of the input data domain.
Cruz teaches
monitoring performance of the trained ML model and the supplemental ML model; and
(Cruz [sec(s) Abs] “Dynamic ensemble selection systems work by estimating the level of competence of each classifier from a pool of classifiers. Only the most competent ones are selected to classify a given test sample. This is achieved by defining a criterion to measure the level of competence of a base classifier, such as, its accuracy in local regions of the feature space around the query instance. … We propose five distinct sets of meta-features, each one corresponding to a different criterion to measure the level of competence of a classifier for the classification of input samples.” [sec(s) 5] “Each meta-feature fi corresponds to a different criterion to measure the level of competence of a base classifier.”;)
adjusting portions of the input data domain associated with the trained ML model and the supplemental ML model based on results of the monitoring,
(Cruz [sec(s) Abs] “During the generalization phase, the meta-features are extracted from the query instance and passed down as input to the meta-classifier. The meta-classifier estimates, whether a base classifier is competent enough to be added to the ensemble.” [sec(s) 1] “The proposed framework is divided into three phases: over production, meta-training and generalization. In the overproduction stage, a pool of classifiers is generated using the training data. In the meta-training stage, the five sets of meta-features are extracted from the training data, and are used to train the meta classifier that works as the classifier selector. During the generalization phase, the meta-features are extracted from the query instance and passed down as inputs to the meta-classifier. The meta-classifier estimates whether a base classifier is competent enough to classify the given test instance. Thus, the proposed system differs from the current state-of-the-art dynamic selection techniques not only because it uses multiple criteria to perform the classifier selection, but also because the classifier selection rule is learned by the meta-classifier using the training data.”;)
The combination of Boulanger, Frosch is combinable with Cruz for the same rationale as set forth above with respect to claim 7.
However, the combination of Boulanger, Frosch, Cruz does not appear to explicitly teach:
wherein adjusting the portions comprises adjusting a boundary between the first portion of the input data domain and the second portion of the input data domain thereby changing the first portion and the second portion of the input data domain.
Bolukbasi teaches
wherein adjusting the portions comprises adjusting a boundary between the first portion of the input data domain and the second portion of the input data domain thereby changing the first portion and the second portion of the input data domain.
(Bolukbasi [sec(s) 1] “One of the main challenges involved in designing these systems is determining whether the machine-learned models will run in the devices or in the cloud. Offloading all of the work to the cloud can be problematic due to network latency, limited cloud ingress bandwidth, cloud availability and reliability issues, and privacy concerns. Our approach can be used to design such a system, by deploying a small inaccurate model and an exit policy on each device and a large accurate model in the cloud. Easy examples would be handled by the devices, while difficult ones would be forwarded to the cloud. Our approach naturally generalizes to a fog computing topology (where resource constrained edge devices are connected to a more powerful local gateway computer, which in turn is connected to a sequence of increasingly powerful computers along the path to the data-center). Such designs allow our method to be used in memory constrained settings as well due to offloading of complex models from the device.” [sec(s) 3] “After computing the kth convolutional layer, we introduce a decision function γk that determines whether the example should exit the network with a label of ˆyk(x) or proceed to the next layer for further evaluation. … Globally, our goal is to minimize the evaluation time of the network such that the error rate of the adaptive system is no more than some user-chosen value B greater than the full network” [sec(s) 4] “Conceptually, we seek to exploit the fact that many examples are correctly classified by relatively efficient networks such as alexnet and googlenet, whereas only a small fraction of examples are correctly classified by computationally expensive networks such as resnet 152 and incorrectly classified by googlenet and alexnet.” [sec(s) 5.3] “Fig. 6 shows the distributions over examples of the networks that correctly label the example. Notably, 50% and 77% of the examples are correctly classified by all networks for top 1 and top 5 error, respectively. Similarly, 18% and 5% of the examples are incorrectly classified by all networks with respect to their top 1 and top 5 error, respectively. These results verify our hypothesis that for a large fraction of data, there is no need for costly networks.”;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Boulanger, Frosch, Cruz with the data domain portion adjusting of Bolukbasi.
One of ordinary skill in the art would have been motived to combine in order to reduce the computational time, and prevent the oracle from “correcting” mistakes made by the most complex networks.
(Bolukbasi [sec(s) 5] “This second constraint is added due to the fact that our goal is not to improve the prediction performance of the system but to reduce the computational time, and therefore we prevent the oracle from “correcting” mistakes made by the most complex networks.”)
Claim(s) 11-12, 16, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Boulanger et al. (US 2021/0133510 A1) in view of Frosch et al. (US 20220335328 A1) in view of Hall et al. (US 20230162049 A1)
Regarding claim 11
The combination of Boulanger, Frosch teaches claim 1.
However, Boulanger does not appear to explicitly teach:
wherein the first representation of the first distribution of the first plurality of inputs comprises a histogram having a plurality of bins and a plurality of counts corresponding to the plurality of bins, each of the plurality of counts indicating how many of the first plurality of inputs fall into a respective bin in the plurality of bins.
Hall teaches
wherein the first representation of the first distribution of the first plurality of inputs comprises a histogram having a plurality of bins and a plurality of counts corresponding to the plurality of bins, each of the plurality of counts indicating how many of the first plurality of inputs fall into a respective bin in the plurality of bins.
(Hall [fig(s) 7] [par(s) 130] “FIG. 7 is a histogram of the number of images per strictness threshold for test and train sets in normal and pneumonia labeled images according to an embodiment;” [par(s) 244] “FIG. 7 is a histogram of the number of images per strictness threshold for test and train sets in normal and pneumonia labeled images according to an embodiment. FIG. 7 highlights two important effects of label noise in the set. 1) Though representing only 12% of the aggregated dataset, the test set increases the number of noisy labels identified by 100% when compared with the number for the training set alone, underlining the knock-on effect that label noise can have on model performance. 2) This shows how false negatives added to a training set “confuses” the model, causing a counter-intuitive increase in the number of false positives.” [par(s) 203] “Using Algorithm 3, a histogram is generated that bins together images with the same number of successful predictions hl←∥{sj|sj=l}∥0, where bin l contains images that were successfully predicted by l models (0<l<n×k). A cumulative histogram H←Σi=0lhi is then used to calculate a percentage difference operator.;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Boulanger, Frosch with the histogram bins of Hall.
One of ordinary skill in the art would have been motived to combine in order to provide an improvement in the AI performance of prediction by creating a clean training dataset for AI training based on removal of bad data.
(WERKHEISER [par(s) 337-338] “This indicates that the Untrainable Data Cleansing technique has in fact removed the mid-labeled and noisy data from the dataset, ultimately improving the data quality and thus the AI model performance.” [par(s) 359] “The above case studies demonstrated that embodiments of the UDC method can be used to effectively identify bad data ( or bad labels) even for "hard" classification problems such as detection of pneumonia from pediatric x-rays. The removal of the bad data was used to create a clean training dataset for AI training and resulted in an improvement in the AI performance for identifying pneumonia in x-rays. Additionally, UDC found bad data present in the test dataset used by AI practitioners to test and report on their AI performance for analyzing x-ray images for pneumonia. This means that the AI performance that is reported may not be the true performance of the AI, and potentially (unintentionally) misleading. Lastly, bad data that was identified by UDC for this particular problem was of the Noisy category, which both the AI and radiologist found difficult to label with confidence using the available x-ray image. This suggests that these images alone contain limited information to make a conclusive diagnosis with certainty, and thus have a higher probability of being mislabeled or mis-diagnosed.”)
Regarding claim 12
The combination of Boulanger, Frosch, HALL teaches claim 11.
wherein the first performance data indicative of the measure of performance of the trained ML model on the first plurality of inputs comprises: (See claim 1)
Hall further teaches
for each bin of at least some of the plurality of bins, a measure of average error by the trained ML model when applied to inputs, among the first plurality of inputs, that fall in the bin.
(Hall [fig(s) 5-6, 11] “balanced accuracy” [par(s) 128] “FIG. 5 is a set of histogram plots showing balanced accuracy (top) and cross-entropy, or log loss, (bottom) (left) for various model architectures Wore UDC and (right) for the ResNet-50 architecture after UDC for varying strictness thresholds 1 according to an embodiment;” [par(s) 176-180] “(Mean) accuracy A: is the proportion of predictions for which the model was correct (NT) compared with the total number of predictions (N). Formally, accuracy has the following definition: … Balanced accuracy(or F1 Score) Abal: is more suitable in cases where the data class distribution is unbalanced. Balanced accuracy is calculated as the average of the class-based accuracy for all classes c
∈
{1... C}:
PNG
media_image1.png
105
467
media_image1.png
Greyscale
” [par(s) 240] “FIG. 5 shows the balanced accuracy (top) and cross-entropy, or log loss, (bottom) (left) for various model architectures before UDC and (right) for the ResNet-50 architecture after UDC for varying strictness thresholds 1 for the test set. The shading of the bars represent the performance of the model on the test set if the epoch (or model) chosen is that which resulted in the lowest log loss as measured against (diagonal lines) the test set and (black) the validation ("val") set. The discrepancy between these two values is indicative of the generalisability of the model; i.e. models that perform well one but not the other are not expected to generalise well. This discrepancy is shown to improve with UDC. Case Study 2B: Test Set Treated as Additional Data Source Case Study 2A shows that UDC improves model performance even on a blind test set, which is a measure of the power of the UDC method. In this section, the effect of treating the test set as a different data source is investigated. To this end, the test set is included (or "injected") into the training set and the resulting effect on model performance is noted.”;)
The combination of Boulanger, Frosch, Hall is combinable with Hall for the same rationale as set forth above with respect to claim 11.
Regarding claim 16
The combination of Boulanger, Frosch teaches claim 1.
However, Boulanger does not appear to explicitly teach:
wherein the first representation comprises a first histogram having a first plurality of counts for a plurality of bins, and the second representation comprises a second histogram having a second plurality of counts for the plurality of bins,
wherein the first performance data comprises, for each bin of at least some of the plurality of bins, a measure of error incurred by the trained ML model when applied to inputs, among the first plurality of inputs, that fall in the bin, and
wherein the second performance data comprises, for each bin of at least some of the plurality of bins, a measure of error incurred by the trained ML model when applied to inputs, among the second plurality of inputs, that fall in the bin.
Hall teaches
wherein the first representation comprises a first histogram having a first plurality of counts for a plurality of bins, and the second representation comprises a second histogram having a second plurality of counts for the plurality of bins,
(Hall [fig(s) 7] [par(s) 130] “FIG. 7 is a histogram of the number of images per strictness threshold for test and train sets in normal and pneumonia labeled images according to an embodiment;” [par(s) 244] “FIG. 7 is a histogram of the number of images per strictness threshold for test and train sets in normal and pneumonia labeled images according to an embodiment. FIG. 7 highlights two important effects of label noise in the set. 1) Though representing only 12% of the aggregated dataset, the test set increases the number of noisy labels identified by 100% when compared with the number for the training set alone, underlining the knock-on effect that label noise can have on model performance. 2) This shows how false negatives added to a training set “confuses” the model, causing a counter-intuitive increase in the number of false positives.” [par(s) 203] “Using Algorithm 3, a histogram is generated that bins together images with the same number of successful predictions hl←∥{sj|sj=l}∥0, where bin l contains images that were successfully predicted by l models (0<l<n×k). A cumulative histogram H←Σi=0lhi is then used to calculate a percentage difference operator.;)
wherein the first performance data comprises, for each bin of at least some of the plurality of bins, a measure of error incurred by the trained ML model when applied to inputs, among the first plurality of inputs, that fall in the bin, and
(Hall [fig(s) 5-6, 11] “balanced accuracy” [par(s) 128] “FIG. 5 is a set of histogram plots showing balanced accuracy (top) and cross-entropy, or log loss, (bottom) (left) for various model architectures Wore UDC and (right) for the ResNet-50 architecture after UDC for varying strictness thresholds 1 according to an embodiment;” [par(s) 176-180] “(Mean) accuracy A: is the proportion of predictions for which the model was correct (NT) compared with the total number of predictions (N). Formally, accuracy has the following definition: … Balanced accuracy(or F1 Score) Abal: is more suitable in cases where the data class distribution is unbalanced. Balanced accuracy is calculated as the average of the class-based accuracy for all classes c
∈
{1... C}:
PNG
media_image1.png
105
467
media_image1.png
Greyscale
” [par(s) 240] “FIG. 5 shows the balanced accuracy (top) and cross-entropy, or log loss, (bottom) (left) for various model architectures before UDC and (right) for the ResNet-50 architecture after UDC for varying strictness thresholds 1 for the test set. The shading of the bars represent the performance of the model on the test set if the epoch (or model) chosen is that which resulted in the lowest log loss as measured against (diagonal lines) the test set and (black) the validation ("val") set. The discrepancy between these two values is indicative of the generalisability of the model; i.e. models that perform well one but not the other are not expected to generalise well. This discrepancy is shown to improve with UDC. Case Study 2B: Test Set Treated as Additional Data Source Case Study 2A shows that UDC improves model performance even on a blind test set, which is a measure of the power of the UDC method. In this section, the effect of treating the test set as a different data source is investigated. To this end, the test set is included (or "injected") into the training set and the resulting effect on model performance is noted.”;)
wherein the second performance data comprises, for each bin of at least some of the plurality of bins, a measure of error incurred by the trained ML model when applied to inputs, among the second plurality of inputs, that fall in the bin.
(Hall [fig(s) 5-6, 11] “balanced accuracy” [par(s) 128] “FIG. 5 is a set of histogram plots showing balanced accuracy (top) and cross-entropy, or log loss, (bottom) (left) for various model architectures Wore UDC and (right) for the ResNet-50 architecture after UDC for varying strictness thresholds 1 according to an embodiment;” [par(s) 176-180] “(Mean) accuracy A: is the proportion of predictions for which the model was correct (NT) compared with the total number of predictions (N). Formally, accuracy has the following definition: … Balanced accuracy(or F1 Score) Abal: is more suitable in cases where the data class distribution is unbalanced. Balanced accuracy is calculated as the average of the class-based accuracy for all classes c
∈
{1... C}:
PNG
media_image1.png
105
467
media_image1.png
Greyscale
” [par(s) 240] “FIG. 5 shows the balanced accuracy (top) and cross-entropy, or log loss, (bottom) (left) for various model architectures before UDC and (right) for the ResNet-50 architecture after UDC for varying strictness thresholds 1 for the test set. The shading of the bars represent the performance of the model on the test set if the epoch (or model) chosen is that which resulted in the lowest log loss as measured against (diagonal lines) the test set and (black) the validation ("val") set. The discrepancy between these two values is indicative of the generalisability of the model; i.e. models that perform well one but not the other are not expected to generalise well. This discrepancy is shown to improve with UDC. Case Study 2B: Test Set Treated as Additional Data Source Case Study 2A shows that UDC improves model performance even on a blind test set, which is a measure of the power of the UDC method. In this section, the effect of treating the test set as a different data source is investigated. To this end, the test set is included (or "injected") into the training set and the resulting effect on model performance is noted.”;)
The combination of Boulanger, Frosch is combinable with Hall for the same rationale as set forth above with respect to claim 11.
Regarding claim 18
The combination of Boulanger, Frosch teaches claim 1.
Boulanger further teaches
identifying, using the first representation, the second representation, the first performance data and the second performance data, one or more bins for which to obtain additional inputs and corresponding ground truth values for improving performance of the trained ML model.
(Boulanger [fig(s) 1] [par(s) 19-22] “Referring to FIG. 1, various embodiments of a system for training and validating a neural network will be described. In one or more embodiments, a system 50 generates a training dataset in an iterative process that yields high performance CNN object classification. … The dataset generator 74 uses the results of the data extractor/analyzer 72 and, in view of configuration parameters 64, generates parameters for a new training dataset 56 comprising a subset of the current training dataset 56 images and parameters defining new synthetic images to be generated for the next training dataset. The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56. In some embodiments, the process continues iteratively until a final training dataset 80 that meets certain performance criteria, such as a percentage of correctly classified images during the validation process, performance for various size objects, cost validation and/or other criteria, is generated” [par(s) 5] “a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN” [par(s) 45] “Each synthetic image is labeled with the correct classification and the output of the neural network 330 is compared to the correct label.”; e.g., “The assembler/interface 76 converts the new training dataset parameters into instructions directing the image creation interface 54 to cause the synthetic image generator 52 to generate a new training data set 56.” read(s) on “bins”.)
However, the combination of Boulanger, Frosch does not appear to explicitly teach:
wherein the first representation of the first distribution of the first plurality of inputs comprises a first histogram having a plurality of bins,
wherein the second representation of the second distribution of the second plurality of inputs comprises a second histogram having the plurality of bins, wherein the method further comprises:
Hall teaches
wherein the first representation of the first distribution of the first plurality of inputs comprises a first histogram having a plurality of bins,
(Hall [fig(s) 7] [par(s) 130] “FIG. 7 is a histogram of the number of images per strictness threshold for test and train sets in normal and pneumonia labeled images according to an embodiment;” [par(s) 244] “FIG. 7 is a histogram of the number of images per strictness threshold for test and train sets in normal and pneumonia labeled images according to an embodiment. FIG. 7 highlights two important effects of label noise in the set. 1) Though representing only 12% of the aggregated dataset, the test set increases the number of noisy labels identified by 100% when compared with the number for the training set alone, underlining the knock-on effect that label noise can have on model performance. 2) This shows how false negatives added to a training set “confuses” the model, causing a counter-intuitive increase in the number of false positives.” [par(s) 203] “Using Algorithm 3, a histogram is generated that bins together images with the same number of successful predictions hl←∥{sj|sj=l}∥0, where bin l contains images that were successfully predicted by l models (0<l<n×k). A cumulative histogram H←Σi=0lhi is then used to calculate a percentage difference operator.;)
wherein the second representation of the second distribution of the second plurality of inputs comprises a second histogram having the plurality of bins,
(Hall [fig(s) 7] [par(s) 130] “FIG. 7 is a histogram of the number of images per strictness threshold for test and train sets in normal and pneumonia labeled images according to an embodiment;” [par(s) 244] “FIG. 7 is a histogram of the number of images per strictness threshold for test and train sets in normal and pneumonia labeled images according to an embodiment. FIG. 7 highlights two important effects of label noise in the set. 1) Though representing only 12% of the aggregated dataset, the test set increases the number of noisy labels identified by 100% when compared with the number for the training set alone, underlining the knock-on effect that label noise can have on model performance. 2) This shows how false negatives added to a training set “confuses” the model, causing a counter-intuitive increase in the number of false positives.” [par(s) 203] “Using Algorithm 3, a histogram is generated that bins together images with the same number of successful predictions hl←∥{sj|sj=l}∥0, where bin l contains images that were successfully predicted by l models (0<l<n×k). A cumulative histogram H←Σi=0lhi is then used to calculate a percentage difference operator.;)
The combination of Boulanger, Frosch is combinable with Hall for the same rationale as set forth above with respect to claim 11.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SEHWAN KIM whose telephone number is (571)270-7409. The examiner can normally be reached Mon - Thu 7:00 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J Huntley can be reached on (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SEHWAN KIM/Examiner, Art Unit 2129
