DETAILED ACTION
This Office Action is in response to communications filed on December 18th, 2025 for Application
No. 18/066,079, in which claims 1 and 3-20 are presented for examination. The amendments
filed on December 18th, 2025 have been entered, where claims 1, 3-7, and 9-18 are amended and
claim 2 is canceled.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 16 and 18-20 are objected to because of the following informalities:
“the plurality of data” (Claim 16, ln. 2) should be “the plurality of data”.
“determining ,” (Claim 18, ln. 6) should be “determining,” (objection applies equally to dependent claims 19-20).
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 and 3-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract ideas without significantly more.
Regarding Claim 1:
Step 1: Claim 1 is a process claim. Therefore, claims 1 and 3-10 are directed to a statutory category of eligible subject matter.
Step 2A Prong 1: If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the "Mental Processes" grouping of abstract ideas. Here, steps of the claimed method are mental processes. Specifically, the claim recites
“A method for selection of training data sets, the method comprising . . . computing an . . . evaluation metric that correlates with a clinical outcome for each instance of the machine trained model” (mental process – apart from “the machine trained model” itself, amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a metric, which may be aided by pen and paper) and
“identifying, based on the . . . evaluation metric, one or more data sets for which performance . . . deviates from a defined performance range; and . . . where the one or more data sets are prioritized for use over data sets that do not deviate from the defined performance range” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on identified subset to be prioritized, with reference to a known range, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional elements:
“machine training a model to perform a . . . task . . . the machine trained model performing . . . of the machine trained model . . . retraining the machine learning model” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea) and
“medical imaging . . . deploying the machine trained model to a clinical environment . . . application dependent clinical . . . the medical imaging task for a medical procedure . . . application dependent clinical” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional elements:
“machine training a model to perform a . . . task . . . the machine trained model performing . . . of the machine trained model . . . retraining the machine learning model” (mere instructions to apply the exception using generic computer components does not provide an inventive concept) and
“medical imaging task; deploying the machine trained model to a clinical environment . . . application dependent clinical . . . the medical imaging task for a medical procedure . . . application dependent clinical” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
For the reasons above, Claim 1 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 3-10. The additional limitations of the dependent claims are addressed below.
Regarding Claim 3:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 3 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“wherein prioritizing the one or more data sets comprises wherein the one or more data sets are given priority” (mental process – amounts to exercising judgment to evaluate observed or known information in order to prioritize previously identified samples of information, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“in at least one of data transfer, data anonymization, or preprocessing” (transmitting data, anonymizing data, and preprocessing data all amount to insignificant extra-solution activity that is incidental to the claimed subject matter) and
“during retraining of the machine trained model” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“in at least one of data transfer, data anonymization, or preprocessing” (the recited element is well‐understood, routine, and conventional; whether in the form of transmitting data, see generally Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362; see also buySAFE, Inc. v. Google Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014), or preprocessing health data, Ferrão et al., Preprocessing structured clinical data for predictive modeling and decision support, Pg. 1145, Para. 5-6, “the target audience of this article is not limited to researchers, but also extended to those involved in system implementation, training and use in routine practice . . . Considering the strong trends in clinical informatics towards structured data entry, we believe that the preprocessing tasks addressed in this article increasingly play a major role”, or anonymizing health data, Ghinita et al., Anonymous Publication of Sensitive Transactional Data, Pg. 161, Col. 1, Para. 1, “Organizations, such as hospitals, publish detailed data (microdata) about individuals (e.g., medical records) . . . Existing privacy-preserving techniques focus on anonymizing personal data”, Pg. 165, Col. 1, Para. 3, “It is well understood [24] that publishing privacy-sensitive data is caught between the conflicting requirements of privacy and utility”; therefore, the limitation, which is recited with a high level of generality, remains insignificant extra-solution activity even upon reconsideration) and
“during retraining of the machine trained model” (mere instructions to apply the exception using generic computer components does not provide an inventive concept).
Accordingly, Claim 3 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 4:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 4 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“wherein identifying the one or more data sets comprises automatically assigning a priority score to the one or more data sets and prioritizing the one or more data sets . . . according to their priority score” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a priority score, which may happen automatically for simple scoring methods like as binary, and forming a priority opinion based on these scores, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“comprises pushing the one or more data sets to an annotation queue” (pushing data to a queue amounts to insignificant extra-solution activity that is incidental to the claimed subject matter).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“comprises pushing the one or more data sets to an annotation queue” (pushing data to a queue is well‐understood, routine, and conventional; whether in the form of transmitting data, see generally Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362; see also buySAFE, Inc. v. Google Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014), or storing data in memory, see Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); therefore, the limitation, which is recited with a high level of generality, remains insignificant extra-solution activity even upon reconsideration).
Accordingly, Claim 4 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 5:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 5 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“wherein prioritizing the one or more data sets comprises assigning the one or more data sets to annotators such that expert annotators are assigned data sets that deviate a most from the defined performance range to annotate” (mental process – apart from the “assigning” itself, which may require a particular technological environment, amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a specific subset to be evaluated by annotators, with reference to a known range, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“assigning” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“assigning” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
Accordingly, Claim 5 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 6:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 6 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“wherein prioritizing comprises identifying sites in a . . . setup that include data sets that deviate the most from the defined performance range” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a specific subset to be prioritized, with reference to a known performance range, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“federated learning” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“federated learning” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
Accordingly, Claim 6 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 7:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 7 depends on.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“wherein during machine training of the model, the model is tested using a testing metric that is different than the . . . evaluation metric” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea) and
“application dependent clinical” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“wherein during machine training of the model, the model is tested using a testing metric that is different than the . . . evaluation metric” (mere instructions to apply the exception using generic computer components does not provide an inventive concept) and
“application dependent clinical” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
Accordingly, Claim 7 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 8:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 8 depends on.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“wherein the task comprises image segmentation of medical imaging data acquired using one of MRI, CT, X-ray, or Ultrasound” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“wherein the task comprises image segmentation of medical imaging data acquired using one of MRI, CT, X-ray, or Ultrasound” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
Accordingly, Claim 8 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 9:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 9 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“wherein the . . . evaluation metric comprises an assessment of how well an output” (mental process – amounts to exercising judgment to evaluate observed or known information to form an opinion, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“of the machine trained model” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea) and
“application dependent clinical . . . guided a clinician during the medical procedure subsequent to the performance of the task” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“of the machine trained model” (mere instructions to apply the exception using generic computer components does not provide an inventive concept) and
“application dependent clinical . . . guided a clinician during the medical procedure subsequent to the performance of the task” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
Accordingly, Claim 9 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 10:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 10 depends on. Here, the claim recites additional elements that are mental processes. Specifically, the claim recites
“wherein identifying comprises identifying one or more data sets that are the ten percent of data sets that deviate the most from the defined performance range” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a specific subset percentage of data to be prioritized, with reference to a known performance range, which may be aided by pen and paper).
Step 2A Prong 2 & Step 2B: There are no elements left for consideration of implementation within a practical application or for consideration of significantly more.
Accordingly, Claim 10 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 11:
Step 1: Claim 11 is a machine claim. Therefore, claims 11-20 are directed to a statutory category of eligible subject matter.
Step 2A Prong 1: If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the "Mental Processes" grouping of abstract ideas. Here, steps performed by the claimed system are mental processes. Specifically, the claim recites
“wherein each data set of the plurality of data sets is assigned an . . . evaluation metric . . . prioritize one or more data sets of the plurality of data sets for use in the training based on the . . . evaluation metric” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a metric, which is used to form opinions on priority that may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional elements:
“A system for data selection for training a deep learning network, comprising: a datastore configured to . . . the deep learning network configured to perform a task; and a processor configured to train the deep learning network using the plurality of data sets, the processor configured to . . . ” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea);
“store a plurality of data sets” (storing data amount to insignificant extra-solution activity that is incidental to the claimed subject matter); and
“application dependent clinical . . . that correlates with a clinical outcome associated with a respective data set . . . relating to medical imaging . . . application dependent clinical” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional elements:
“A system for data selection for training a deep learning network, comprising: a datastore configured to . . . the deep learning network configured to perform a task; and a processor configured to train the deep learning network using the plurality of data sets, the processor configured to . . . ” (mere instructions to apply the exception using generic computer components does not provide an inventive concept);
“store a plurality of data sets” (storing data is well‐understood, routine, and conventional, see Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); therefore, the limitation, which is recited with a high level of generality, remains insignificant extra-solution activity even upon reconsideration); and
“application dependent clinical . . . that correlates with a clinical outcome associated with a respective data set . . . relating to medical imaging . . . application dependent clinical” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
For the reasons above, Claim 11 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 12-17. The additional limitations of the dependent claims are addressed below.
Regarding Claim 12:
Step 2A Prong 1: See the rejection of Claim 11 above, which Claim 12 depends on.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“wherein during training of the deep learning network, the deep learning network is tested using a testing metric that is different than the . . . evaluation metric” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea) and
“application dependent clinical” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“wherein during training of the deep learning network, the deep learning network is tested using a testing metric that is different than the . . . evaluation metric” (mere instructions to apply the exception using generic computer components does not provide an inventive concept) and
“application dependent clinical” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
Accordingly, Claim 12 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 13:
Step 2A Prong 1: See the rejection of Claim 11 above, which Claim 13 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“wherein the . . . evaluation metric comprises an assessment of how well an output” (mental process – amounts to exercising judgment to evaluate observed or known information to form an opinion, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“of the deep learning network” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea) and
“application dependent clinical . . . guided a clinician during a medical procedure” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“of the deep learning network” (mere instructions to apply the exception using generic computer components does not provide an inventive concept) and
“application dependent clinical . . . guided a clinician during a medical procedure” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
Accordingly, Claim 13 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 14:
Step 2A Prong 1: See the rejection of Claim 11 above, which Claim 14 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“prioritize the one or more data sets” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on prioritization, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“in at least one of data transfer, data anonymization, or preprocessing” (transmitting data, anonymizing data, and preprocessing data all amount to insignificant extra-solution activity that is incidental to the claimed subject matter) and
“wherein the processor is configured to . . . during training of the deep learning network” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“in at least one of data transfer, data anonymization, or preprocessing” (the recited element is well‐understood, routine, and conventional; whether in the form of transmitting data, see generally Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362; see also buySAFE, Inc. v. Google Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014), or preprocessing health data, Ferrão et al., Preprocessing structured clinical data for predictive modeling and decision support, Pg. 1145, Para. 5-6, “the target audience of this article is not limited to researchers, but also extended to those involved in system implementation, training and use in routine practice . . . Considering the strong trends in clinical informatics towards structured data entry, we believe that the preprocessing tasks addressed in this article increasingly play a major role”, or anonymizing health data, Ghinita et al., Anonymous Publication of Sensitive Transactional Data, Pg. 161, Col. 1, Para. 1, “Organizations, such as hospitals, publish detailed data (microdata) about individuals (e.g., medical records) . . . Existing privacy-preserving techniques focus on anonymizing personal data”, Pg. 165, Col. 1, Para. 3, “It is well understood [24] that publishing privacy-sensitive data is caught between the conflicting requirements of privacy and utility”; therefore, the limitation, which is recited with a high level of generality, remains insignificant extra-solution activity even upon reconsideration) and
“wherein the processor is configured to . . . during training of the deep learning network” (mere instructions to apply the exception using generic computer components does not provide an inventive concept).
Accordingly, Claim 14 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 15:
Step 2A Prong 1: See the rejection of Claim 11 above, which Claim 15 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“prioritize the one or more data sets by assigning the one or more data sets to annotators such that expert annotators are assigned data sets that deviate a most from a predefined performance range to annotate” (mental process – apart from the “assigning” itself, which may require the system processor, amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a specific subset to be evaluated by annotators, with reference to a range-based criteria, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“wherein the processor is configured to . . . assigning” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“wherein the processor is configured to . . . assigning” (mere instructions to apply the exception using generic computer components does not provide an inventive concept).
Accordingly, Claim 15 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 16:
Step 2A Prong 1: See the rejection of Claim 11 above, which Claim 16 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“assign the plurality of data sets a priority score based on the . . . evaluation metric and prioritize the one or more data sets . . . according to their priority score” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a priority score based on a metric, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“wherein the processor is configured to” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea);
“application dependent clinical” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use); and
“by pushing the one or more data sets to an annotation queue” (pushing data to a queue amounts to insignificant extra-solution activity that is incidental to the claimed subject matter).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“wherein the processor is configured to” (mere instructions to apply the exception using generic computer components does not provide an inventive concept);
“application dependent clinical” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept); and
“by pushing the one or more data sets to an annotation queue” (pushing data to a queue is well‐understood, routine, and conventional; whether in the form of transmitting data, see generally Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362; see also buySAFE, Inc. v. Google Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014), or storing data in memory, see Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); therefore, the limitation, which is recited with a high level of generality, remains insignificant extra-solution activity even upon reconsideration).
Accordingly, Claim 16 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 17:
Step 2A Prong 1: See the rejection of Claim 11 above, which Claim 17 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“only use the ten percent of data sets that deviate the most from a predefined performance range based on the . . . evaluation metric for subsequent training” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a specific subset percentage of data to be included in future use, based in turn on a previously determined range, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“wherein the processor is configured to . . . of the deep learning network” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea) and
“application dependent clinical” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“wherein the processor is configured to . . . of the deep learning network” (mere instructions to apply the exception using generic computer components does not provide an inventive concept) and
“application dependent clinical” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
Accordingly, Claim 17 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 18:
Step 1: Claim 18 is a process claim. Therefore, claims 18-20 are directed to a statutory category of eligible subject matter.
Step 2A Prong 1: If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the "Mental Processes" grouping of abstract ideas. Here, steps of the claimed method are mental processes. Specifically, the claim recites
“A method for data collection, the method comprising . . . computing an . . . evaluation metric for the processed medical imaging data . . . determining, based on the . . . evaluation metric, that the medical imaging data comprises a data set that deviates from a defined performance range; and prioritizing the medical imaging data” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a metric, which is based a previously known range, which is then used to form opinions on the associated information and what should be prioritized which may be aided by pen and paper)
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional elements:
“processing the medical imaging data using a machine learned network . . . for retraining of the machine learned network” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea);
“performing a medical imaging procedure . . . application dependent clinical . . . based on a clinical outcome related to the medical imaging procedure . . . application dependent clinical” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use); and
“to generate medical imaging data” (mere data gathering amounts to insignificant extra-solution activity that is incidental to the claimed subject matter).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional elements:
“processing the medical imaging data using a machine learned network . . . for retraining of the machine learned network” (mere instructions to apply the exception using generic computer components does not provide an inventive concept);
“performing a medical imaging procedure . . . application dependent clinical . . . based on a clinical outcome related to the medical imaging procedure . . . application dependent clinical” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept); and
“to generate medical imaging data” (generating medical data through clinical procedures is well‐understood, routine, and conventional, see Genetic Techs. Ltd., 818 F.3d at 1377; 118 USPQ2d at 1546; Sequenom, 788 F.3d at 1377-78, 115 USPQ2d at 1157); Cleveland Clinic Foundation 859 F.3d at 1362, 123 USPQ2d at 1088 (Fed. Cir. 2017); therefore, the limitation, which is recited with a high level of generality, remains insignificant extra-solution activity even upon reconsideration).
For the reasons above, Claim 18 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 19-20. The additional limitations of the dependent claims are addressed below.
Regarding Claim 19:
Step 2A Prong 1: See the rejection of Claim 18 above, which Claim 19 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“wherein prioritizing comprises” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on prioritization, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“at least one of data transfer, data anonymization, or preprocessing of the medical imaging data” (transmitting data, anonymizing data, and preprocessing data all amount to insignificant extra-solution activity that is incidental to the claimed subject matter) and
“during retraining of the machine learned network” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“at least one of data transfer, data anonymization, or preprocessing of the medical imaging data” (the recited element is well‐understood, routine, and conventional; whether in the form of transmitting data, see generally Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362; see also buySAFE, Inc. v. Google Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014), or preprocessing health data, Ferrão et al., Preprocessing structured clinical data for predictive modeling and decision support, Pg. 1145, Para. 5-6, “the target audience of this article is not limited to researchers, but also extended to those involved in system implementation, training and use in routine practice . . . Considering the strong trends in clinical informatics towards structured data entry, we believe that the preprocessing tasks addressed in this article increasingly play a major role”, or anonymizing health data, Ghinita et al., Anonymous Publication of Sensitive Transactional Data, Pg. 161, Col. 1, Para. 1, “Organizations, such as hospitals, publish detailed data (microdata) about individuals (e.g., medical records) . . . Existing privacy-preserving techniques focus on anonymizing personal data”, Pg. 165, Col. 1, Para. 3, “It is well understood [24] that publishing privacy-sensitive data is caught between the conflicting requirements of privacy and utility”; therefore, the limitation, which is recited with a high level of generality, remains insignificant extra-solution activity even upon reconsideration) and
“during retraining of the machine learned network” (mere instructions to apply the exception using generic computer components does not provide an inventive concept).
Accordingly, Claim 19 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 20:
Step 2A Prong 1: See the rejection of Claim 18 above, which Claim 20 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“wherein prioritizing the medical imaging data comprises assigning the medical imaging data to annotators such that an expert annotator gets the medical imaging data to annotate” (mental process – apart from the “assigning” itself, which may require a particular technological environment, amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a specific subset to be evaluated by annotators, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“assigning” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“assigning” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
Accordingly, Claim 20 is rejected as being directed to an abstract idea without significantly more.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 7-10, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt et al. (hereinafter Bernhardt) (“Active label cleaning: Improving dataset quality under resource constraints”) in view of Wong et al. (hereinafter Wong) (“Implementation of deep learning-based auto-segmentation for radiotherapy planning structures: a workfow study at two cancer centers”) and He et al. (hereinafter He) (Pat. Pub. No. US 2022/0347583 A1).
Regarding Claim 1, Bernhardt teaches a method for selection of training data sets, the method comprising (Pg. 1, Abstract, “Imperfections in data annotation, known as label noise, are detrimental to the training of machine learning models . . . This work advocates for a data-driven approach to prioritising samples for re-annotation—which we term “active label cleaning” . . . the proposed active label cleaning enables correcting labels up to 4× more effectively than typical random selection”, where the “data-driven approach” is a method for “selecti[ng]” data sets, “samples”, for “re-annotation” and where the data sets are for “training of machine learning models”):
machine training a model to perform a medical imaging task (Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model, pθ(ˆy|x), parametrised by θ”; Pg. 3, Col. 2, Table 1, “1: θ ← TRAINROBUSTMODEL(D)”, where the task is “classification” for “medical imaging”, see Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”; see also Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”);
. . . ;
computing an application dependent clinical evaluation metric that correlates with a clinical outcome (Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, where the “predicted posteriors” are an evaluation metric because they are used to evaluate the “label[s]”; see also Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy”, where the “predicted posteriors” are correlated with the clinical outcome of result accuracy, and are therefore an application dependent clinical evaluation metric when used for evaluating “labels” from “radiology reports” for “medical imaging”, see Pg. 1, Col. 1, Para. 2, “Diagnostic labels were extracted from the radiology reports via an error-prone automated process”; Pg. 3, Col. 2, Table 1, “5. j ← arg maxi∈Iavail Φ(xi,l^i; θ) Rank (Eq. (2))” and Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”)
for each instance of the machine trained model performing the medical imaging task . . . (Pg. 3, Col. 2, Table 1, “5. j ← arg maxi∈Iavail Φ(xi,l^i; θ) Rank (Eq. (2))” and Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”, where the predicted postures, “Φ(xi,l^i; θ) ”, which as discussed above are the application dependent clinical evaluation metric, are calculated at each iteration, which are within the broadest reasonable interpretation of an instance, of the model performing the “medical imaging” task);
identifying, based on the application dependent clinical evaluation metric, one or more data sets (Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where the “iterate[ve]” “rank[ing]” and selection of a “batch of samples” based on “correctness and ambiguity (i.e. annotation difficulty)” is within the broadest reasonable interpretation of identifying one or more data sets, which are identified based on the “predicted posteriors”, which as discussed above are the application dependent clinical evaluation metric for each model iteration, see Pg. 2-3, Col. 2-1, Para. 3-1, “To this end, we propose to rank available samples by the following scoring function Φ: . . . [equation 2]. The first term . . . corresponds to the estimated noisiness [and the second term] . . . penalises ambiguous cases in the ranking”)
for which performance of the machine trained model deviates . . . (Pg. 2-3, Col. 2-1, Para. 3-1, “To this end, we propose to rank available samples by the following scoring function Φ: . . . [equation 2]. The first term . . . corresponds to the estimated noisiness”, where the “first term” of the application dependent clinical evaluation metrics “corresponds to the estimated noisiness”, which results in reduced performance of the machined trained model, see and Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy”, thus the above discussed identified data sets deviate in model performance); and
retraining the machine learning model where the one or more data sets are prioritized for use over data sets that do not [sufficiently] deviate . . . (Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where the identified data, which as discussed above deviates in model performance, are prioritized for use in active label cleaning over data over data that is not identified because they do not sufficiently deviate; and where active label cleaning is a retraining approach, see Pg. 3, Col. 2, Table 1, “13: θ ← UPDATE(θ, D) . Fine-tune model”, of a machine learned network, see Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”; see Pg. 2, Col. 1, Para. 3, “Prioritising samples for labelling also underpins the paradigm of active learning, whose goal is to select unlabelled samples that would be most beneficial for training in order to improve the performance of a predictive model on a downstream task . . . The key difference here for the proposed approach is that our goal is not only to improve model performance but also to maximise the quality of labels given limited resources, which makes it valuable for both training and evaluation of predictive models. In more detail, we demonstrate how active learning and noiserobust learning (NRL) can play complementary roles in coping with label noise”).
Bernhardt does not explicitly disclose . . . deploying the machine trained model to a clinical environment . . . for a medical procedure . . . (where model deployment in a clinical environment for a medical procedure is not specifically discussed, but see Pg. 7, Col. 1, Para. 1, “extensions to active cleaning will significantly broaden its application scope, enabling more reliable deployment of machine learning systems in resource-constrained settings”, where “deployment” is suggested)
. . . from a defined performance range . . . (where the identifying of data for prioritization is not specifically described as within a defined performance range; subsequent recitations of performance range omitted).
However, Wong teaches . . . deploying the machine trained model to a clinical environment . . . [to conduct a task] for a medical procedure . . . (Pg. 1, Abstract, “DC models were implemented at two cancer centers and used to generate OAR and CTVs for all patients undergoing RT for a central nervous system (CNS), head and neck (H&N), or prostate cancer”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the method of training data selection comprising: training a model, computing an application dependent evaluation metric correlated with a clinical outcome for each instance of the model, and using the metric to identify datasets with performance deviation of Bernhardt with the deployment of a trained machine learning model in a clinical setting to conduct a task for a medical procedure of Wong in order to apply the method to a clinical setting, where its benefits are of significant importance and can be utilized by actual patients (Bernhardt, Pg. 1, Abstract, “employing experts to remove label noise by fully re-annotating large datasets is infeasible in resource-constrained settings, such as healthcare”; Bernhardt, Pg. 2, Col. 2, Para. 2, “This is imperative in safety-critical domains such as healthcare, as model robustness must be validated on clean labels”; Bernhardt, Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy”; Wong, Pg. 1, Abstract, “DC models were implemented at two cancer centers and used to generate OAR and CTVs for all patients undergoing RT for a central nervous system (CNS), head and neck (H&N), or prostate cancer”). Specifically, this modification could benefit model training for a wide variety of clinical tasks (Wong, Pg. 2, Col. 1, Para. 3, “we hope to increase interest in adopting machine learning auto-segmentation applications in other Radiation Oncology clinical practices”, where, while discussed in the context of “machine learning auto-segmentation” and “Radiation Oncology clinical practices”, there is significant motivation to apply this method to any clinical task where a model is trained using labeled data) by allowing resource efficient and accurate fine-tunning of models on actual client data, which will offer an improvement over generic models (compare Wong, Pg. 2, Col. 1, Para. 4, “These models were trained using publicly available data; no local institutional data was used”, where fine-tunning on client data is not performed, with Wong, Pg. 3, Col. 1, Para. 2, “objective comparisons do assist in the identification of consistent DC contouring errors that can then be the target of model training and improvement”, where fine-tunning on client data would be useful).
Additionally, He teaches . . . [identifying and processing one or more data sets for which performance of the data deviates] from a defined performance range . . . (Para. [0050], “the subject-matter . . . can optionally include that filtering the input data further includes: removing data in the consolidated data that correspond to performance values that fall outside a predetermined range of target performance values”, where one or more data sets, “input data”, are identified and processed, “removing data in the consolidated data”, when performance deviates from a defined performance range, “performance values that fall outside a predetermined range of target performance values”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the method for selection of training data sets, wherein the machine trained model and application dependent clinical evaluation metric are used to identify one or more data sets for which performance of the model deviates and wherein the machine learning model is retained in a manner where the one or more data sets are prioritized for use over data sets that do not sufficiently deviate of Bernhardt in view of Wong with the identifying and processing of one or more data sets for which performance of the data deviates from a defined performance range of He in order utilize a simplified and less computationally intensive approach to identifying data for annotation, which improves model accuracy by focusing specifically on data detrimental to model performance (compare Bernhardt, Pg. 2-3, Col. 2-1, Para. 3-1, “To this end, we propose to rank available samples by the following scoring function Φ: . . . [equation 2]. The first term . . . corresponds to the estimated noisiness . . . On the other hand . . . entropy term . . . penalises ambiguous cases in the ranking”, where the ranking requires a computationally intensive calculation of conflicting “term[s]”, which must be further analyzed to balance the interest in identifying “nois[y]” data that is detrimental to model performance, see Bernhardt, Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy”, against the interest of avoiding “ambiguous cases”; with He, Para. [0050], “the subject-matter . . . can optionally include . . . [identifying and performing operation on] a predetermined range of target performance values”, where noisy data detrimental to model performance can be directly identified using a simplified computational operation).
Regarding Claim 7, Bernhardt in view of Wong and He teach the method of claim 1, wherein during machine training of the model, the model is tested using a testing metric that is different than the application dependent clinical evaluation metric (Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model”, where, as discussed above, the “predicted posteriors” are the application dependent clinical evaluation metric, and “accuracy”, a different testing metric, is also computed, see Bernhardt, Pg. 6, Col. 2, Table 3, “We compare classification accuracy on true, noisy, and cleaned labels”; and where both metrics are used during training, see Bernhardt, Pg. 3, Col. 2, Table 1, “5. j ← arg maxi∈Iavail Φ(xi,l^i; θ) Rank (Eq. (2)), where the “predicted posteriors” are calculated at each training iteration, and Bernhardt, Pg. 5, Fig. 3, where “accuracy” is computed for different “Number[s] of collected relabels on the dataset”, where a batch is relabeled at each iteration of training, see Bernhardt, Pg. 3, Col. 2, Table 1, “13: θ ← UPDATE(θ, D) . Fine-tune model”).
Regarding Claim 8, Bernhardt in view of Wong and He teach the method of claim 1, wherein the task comprises image segmentation of medical imaging data acquired using one of MRI, CT, X-ray, or Ultrasound (Bernhardt, Pg. 2, Col. 2, Para. 3, “a trained classification model”, where it is trained on medical imaging data using an X-ray, see Bernhardt, Pg. 9, Col. 1, Para. 7, “For further information on the acquisition and anonymisation of chest X-ray scans”, and where, in view of Wong, the task is segmentation and the medical image data also includes data acquired by CT, see Wong, Pg. 2, Col. 1-2, Para. 3-1, “By sharing our experience implementing machine learning auto-segmentation into the workflow, we hope to increase interest in adopting machine learning auto-segmentation applications in other Radiation Oncology clinical practices . . . Using these CT images, the auto-segmentation software prospectively generated DCs to be reviewed and edited on all patients undergoing RT treatment planning for CNS, H&N, and prostate malignancies”).
The reasons of obviousness are discussed above in regard to the rejection of Claim 1 and remain
applicable here.
Regarding Claim 9, Bernhardt in view of Wong and He teach the method of claim 1, wherein the application dependent clinical evaluation metric (Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, where, as discussed above the “predicted posteriors” are the application dependent clinical evaluation metric)
comprises an assessment of how well an output of the machine trained model guided a clinician during the medical procedure subsequent to the performance of the task (Wong, Pg. 1, Abstract, “Purpose . . . we evaluate the performance of implemented DC models in the clinical radiotherapy (RT) planning workflow and report on user experience”, where, in view of Wong, the evaluation metric includes an assessment of the “models” in “clinical radiotherapy” medical procedures; Wong, Pg. 2, Col. 1-2, Para. 4-1, “the auto-segmentation software prospectively generated DCs to be reviewed and edited on all patients undergoing RT treatment”, where task outputs by the model are subsequently used to guide the clinician in a planning procedure, Wong, Pg. 2, Col. 2, Para. 3, “Generated DCs underwent manual review and were edited as needed prior to being used for RT treatment planning”, and then assessed by how well they performed, Wong, Pg. 1, Abstract, “Methods and materials: DC models were implemented at two cancer centers and used to generate OAR and CTVs for all patients undergoing RT for a central nervous system (CNS), head and neck (H&N), or prostate cancer. Radiation Therapists/Dosimetrists and Radiation Oncologists completed post-contouring surveys rating the degree of edits required for DCs (1 = minimal, 5 = significant) and overall DC satisfaction (1 = poor, 5 = high)”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the use of an evaluation metric that correlates with a clinical outcome to identify challenging machine learning datasets of Bernhardt in view of Wong and He with the use of an evaluation metric based on how well the output of the machine learning model guided a clinician in further view of Wong in order to determine difficult training data based on practical effectiveness instead of just metrics correlated with a clinical outcome (Wong, Pg. 5, Col. 1, Para. 2-3, “than any auto-segmentation time savings. We therefore relied on post-contouring survey feedback as a quantifiable indicator as to whether DCs impeded, rather than streamlined, existing workflow with the presumption that any unusable DCs would result in poor editing scores and overall satisfaction results”; Wong, Pg. 5, Col. 2, Para. 3, “These favourable survey results suggest that the OAR DCs were associated with a clinical workflow benefit”, where “favourable” evaluation metrics are an indication that the data labels do not need to be reviewed).
Regarding Claim 10, Bernhardt in view of Wong and He teach the method of claim 1, wherein identifying comprises identifying the one or more data sets that are the ten percent of data sets that deviate the most from the defined performance range (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where, as discussed above, the “iterate[ve]” “rank[ing]” and selection of a “batch of samples” based on “correctness and ambiguity” is within the broadest reasonable interpretation of identifying data sets, which in view of He, are the data that deviate from the defined performance range, see He, Para. [0050], “the subject-matter . . . can optionally include . . . [identifying and performing operation on] a predetermined range of target performance values”, where, if all data that deviates from “a predetermined range of target performance values” are identified, then the identified data must include the ten percent of data sets that deviate the most).
The reasons for obviousness were discussed in regard to the rejection of claim 1 and remain applicable here.
Regarding Claim 18, Bernhardt in view of Wong and He teach a method for data collection, the method comprising: performing a medical imaging procedure to generate medical imaging data (Bernhardt, Pg. 1, Abstract, “This work advocates for a data-driven approach to prioritising samples for re-annotation—which we term “active label cleaning” . . . the proposed active label cleaning enables correcting labels up to 4× more effectively than typical random selection”, where “data-driven” and “active” selection of training data sets is a method for identifying data for cleaning, which, in view of Wong, includes data collection, see Wong, Pg. 2, Col. 1, Para. 4, “Planning computed tomography (CT) images from both centers were captured using a GE Healthcare Optima CT580 series scanner with the following parameters depending on disease site: 120kVp, 100–700mAs, 1.25–2.5 mm slice thickness, and 0.683–1.270 mm in-plane pixel size. Using these CT images”, where “Planning computed tomography” is a medical imaging procedure that is performed to generate data of “CT images”);
processing the medical imaging data using a machine learned network (Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model, pθ(ˆy|x), parametrised by θ”; Bernhardt, Pg. 3, Col. 2, Table 1, “1: θ ← TRAINROBUSTMODEL(D)”, where the “trained . . .model” is a machine learned network, see Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”, which processes medical imaging data, see Bernhardt, Pg. 3, Col. 1, Para. 2, “we experiment with two imaging datasets, namely CIFAR10H and NoisyCXR . . . We also set up a new benchmark for label cleaning on medical images, NoisyCXR, comprising 26.6k chest radiographs and multiple labels from clinical datasets”, which in view of Wong, is the medical imaging data, see Wong, Pg. 1, Abstract, “DC models were implemented at two cancer centers and used to generate OAR and CTVs for all patients undergoing RT for a central nervous system (CNS), head and neck (H&N), or prostate cancer”);
computing an application dependent clinical evaluation metric for the processed medical imaging data based on a clinical outcome related to the medical imaging procedure (Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, where the “predicted posteriors” are an evaluation metric because they are used to evaluate the “label[s]” of the imaging data; see also Bernhardt, Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy”, where the “predicted posteriors” are correlated with the clinical outcome of result accuracy, and are therefore an application dependent clinical evaluation metric when used for evaluating “labels” from “radiology reports” for “medical imaging”, see Bernhardt, Pg. 1, Col. 1, Para. 2, “Diagnostic labels were extracted from the radiology reports via an error-prone automated process”; Bernhardt, Pg. 3, Col. 2, Table 1, “5. j ← arg maxi∈Iavail Φ(xi,l^i; θ) Rank (Eq. (2))” and Bernhardt, Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”, and, in view of Wong are based on a clinical outcome related to the medical imaging procedure, see Wong, Pg. 2, Col. 1, Para. 4, “Planning computed tomography (CT) images from both centers were captured using a GE Healthcare Optima CT580 series scanner with the following parameters depending on disease site: 120kVp, 100–700mAs, 1.25–2.5 mm slice thickness, and 0.683–1.270 mm in-plane pixel size. Using these CT images”, where “disease site” is a clinical outcome related to “Planning computed tomography”, a medical imaging procedure used to generate the medical imagining data “CT images”, which are the basis of the metric);
determining , based on the application dependent clinical evaluation metric, that the medical imaging data (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where the “iterate[ve]” “rank[ing]” and selection of a “batch of samples” based on “correctness and ambiguity (i.e. annotation difficulty)” is within the broadest reasonable interpretation of determining about one or more data sets, which are identified based on the “predicted posteriors”, which as discussed above are the application dependent clinical evaluation metric for each model iteration, see Bernhardt, Pg. 2-3, Col. 2-1, Para. 3-1, “To this end, we propose to rank available samples by the following scoring function Φ: . . . [equation 2]. The first term . . . corresponds to the estimated noisiness [and the second term] . . . penalises ambiguous cases in the ranking”)
comprises a data set that deviates from a defined performance range (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where, as discussed above, the “iterate[ve]” “rank[ing]” and selection of a “batch of samples” based on “correctness and ambiguity” is within the broadest reasonable interpretation of determining data comprises a data set that deviates, which in view of He, is deviation from the defined performance range, see He, Para. [0050], “the subject-matter . . . can optionally include . . . [identifying and performing operation on] a predetermined range of target performance values”); and
prioritizing the medical imaging data for retraining of the machine learned network (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where the data is prioritized in active label cleaning, which is a retraining approach, see Bernhardt, Pg. 3, Col. 2, Table 1, “13: θ ← UPDATE(θ, D) . Fine-tune model”, of a machine learned network, see Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”; see also Bernhardt, Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”, where the data is “medical imaging” data).
The reasons of obviousness are discussed above in regard to the rejection of Claim 1 and remain
applicable here.
Claims 3, 5, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt in view of Wong, He, and CapeStart contributors (hereinafter CapeStart) (“Medical Image Annotation”).
Regarding Claim 3, Bernhardt in view of Wong and He teach the method of claim 1, wherein prioritizing the one or more data sets comprises wherein the one or more data sets are given priority [in label reannotation] . . . during retraining of the machine trained model (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where the identified data is prioritized in active label cleaning, which is a retraining approach, see Bernhardt, Pg. 3, Col. 2, Table 1, “13: θ ← UPDATE(θ, D) . Fine-tune model”, of a machine learned network, see Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”; see generally Bernhardt, Pg. 2, Col. 1, Para. 3, “Prioritising samples for labelling also underpins the paradigm of active learning, whose goal is to select unlabelled samples that would be most beneficial for training in order to improve the performance of a predictive model on a downstream task . . . The key difference here for the proposed approach is that our goal is not only to improve model performance but also to maximise the quality of labels given limited resources, which makes it valuable for both training and evaluation of predictive models. In more detail, we demonstrate how active learning and noiserobust learning (NRL) can play complementary roles in coping with label noise”).
Bernhardt in view of Wong and He do not explicitly disclose . . . in at least one of data transfer, data anonymization, or preprocessing . . . .
However, CapeStart teaches . . . in at least one of data transfer [to annotators], data anonymization, or preprocessing . . . (Pg. 1, “Expert Medical Image Annotation . . . CapeStart gets you to value faster through our expert team and intuitive, powerful SaaS platform”, where a “Saas platform” for “annotation” of user-supplied medical data by an “expert team” requires data transfer).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the method for retraining a machine trained model by prioritization of one or more data sets in label reannotation during training of Bernhardt in view of Wong and He with the transfer of medical data to annotators of CapeStart in order to receive services on transferred data from remote annotators, which may allow for utilization of a faster and more experienced team than locally available (CapeStart, Pg. 1, “Scale Healthcare Machine Learning Through Expert Medical Image Annotation. Machine learning can revolutionize healthcare delivery – but only when models and applications are well trained with expertly labeled and annotated data. CapeStart gets you to value faster through our expert team and intuitive, powerful SaaS platform”).
Regarding Claim 5, Bernhardt in view of Wong and He teach the method of claim 2, wherein prioritizing the one or more data sets comprises assigning the one or more data sets to annotators such that expert annotators are assigned data sets that deviate a most from the defined performance range to annotate (Bernhardt, Pg. 2, Col. 2, Para. 4, “at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritized sample is reviewed sequentially by different annotators”, where the “batches” “ranked” high priority due to “predicted label correctness”, which, in view of He, include data that deviate from the defined performance range, see He, Para. [0050], “the subject-matter . . . can optionally include . . . [identifying and performing operation on] a predetermined range of target performance values”, where, if all data that deviates from “a predetermined range of target performance values” are identified, then the identified data must include data sets that deviate a most; as well as all the data batches, in instances where sufficient resources allow the method to run to completion, are assigned to the annotators, who, in view of CapeStart, are experts, see CapeStart, Pg. 1, “Expert Medical Image Annotation . . . CapeStart gets you to value faster through our expert team and intuitive, powerful SaaS platform”; see generally Bernhardt, Pg. 1. Col. 2, Para. 2, “Due to the practical constraints on the total number of reannotations, samples often need to be prioritised to maximise the benefits of relabelling efforts (see Fig. 1), as the difficulty of reviewing labelling errors can vary across samples. Some cases are easy to assess and correct, others may be inherently ambiguous even for expert annotators (Fig. 2). For such difficult cases, several annotations (i.e. expert opinions) may be needed to form a ground-truth consensus”).
The reasons of obviousness are discussed above in regard to the rejection of Claim 1, for the combination with He, and in regard to the rejection of Claim 3, for the combination with CapeStart, and remain applicable here.
Regarding Claim 19, Bernhardt in view of Wong, He, and CapeStart teach the method of claim 18, wherein prioritizing comprises at least one of data transfer, data anonymization, or preprocessing of the medical imaging data during retraining of the machine learned network (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where the identified data is prioritized in active label cleaning, which is a retraining approach, see Bernhardt, Pg. 3, Col. 2, Table 1, “13: θ ← UPDATE(θ, D) . Fine-tune model”, of a machine learned network, see Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”; see generally Bernhardt, Pg. 2, Col. 1, Para. 3, “Prioritising samples for labelling also underpins the paradigm of active learning, whose goal is to select unlabelled samples that would be most beneficial for training in order to improve the performance of a predictive model on a downstream task . . . The key difference here for the proposed approach is that our goal is not only to improve model performance but also to maximise the quality of labels given limited resources, which makes it valuable for both training and evaluation of predictive models. In more detail, we demonstrate how active learning and noiserobust learning (NRL) can play complementary roles in coping with label noise”, where, in view of CapeStart, the annotation comprises transfer to remote experts, see CapeStart, Pg. 1, “Expert Medical Image Annotation . . . CapeStart gets you to value faster through our expert team and intuitive, powerful SaaS platform”).
The reasons of obviousness are discussed above in regard to the rejection of Claim 3 and remain
applicable here.
Regarding Claim 20, Bernhardt in view of Wong, He, and CapeStart teach the method of claim 18, wherein prioritizing the medical imaging data comprises assigning the medical imaging data to annotators such that an expert annotator gets the medical imaging data to annotate (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where, in view of CapeStart, the annotation comprises transfer to experts, see CapeStart, Pg. 1, “Expert Medical Image Annotation . . . CapeStart gets you to value faster through our expert team and intuitive, powerful SaaS platform”).
The reasons of obviousness are discussed above in regard to the rejection of Claim 3 and remain
applicable here.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt in view of Wong, He, Northcutt et al. (hereinafter Northcutt) (“Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks”) and CPlusPlus.com contributors (hereinafter CPlusPlus.com) (“std::queue::push”).
Regarding Claim 4, Bernhardt in view of Wong and He teach the method of claim 1, wherein identifying the one or more data sets comprises automatically assigning a priority . . . to the one or more data set (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where, as discussed above, the “iterate[ve]” “rank[ing]” and selection of a “batch of samples” based on “correctness” is within the broadest reasonable interpretation of identifying data sets that are challenging for the trained model, where each “batch of samples” dataset are assigned a different degree of priority based on “correctness”, see generally Bernhardt, Pg. 1, Fig. 1, which happens automatically when the method depicted by the pseudocode algorithm is executed, see Bernhardt, Pg. 3, Col. 2, Table 1)
and prioritizing the one or more data sets comprises pushing the one or more data sets to an annotation [environment] . . . according to their priority . . . (Bernhardt, Pg. 2, Col. 2, Para. 4, “at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritized sample is reviewed sequentially”, where “reviewing” the “batch of samples” “sequentially” requires an action by a computer program to display the data to an annotation environment, which is within the broadest reasonable interpretation of pushing).
Bernhardt in view of Wong and He do not explicitly disclose . . . score . . . queue . . . score . . . .
However, Northcutt teaches . . . [assigning a priority] score [to the data sets] . . . score (Northcutt, Pg. 1, Abstract, “We identify label errors in the test sets of 10 of the most commonly-used computer vision, natural language, and audio datasets, and subsequently study the potential for these label errors to affect benchmark results”; Northcutt, Pg. 4, table 1, “We observe that error rates vary across datasets, from 0.15% (MNIST) to 10.12% (QuickDraw)”; where, in the context of the method of Bernhardt, “% error”, which is the rate “label errors” in a “dataset”, expressed as a percent by multiplying by 100, are within the broadest reasonable interpretation of priority scores because the method is premised on prioritizing the greatest return in corrected labels for an investment of resources, see Bernhardt, Pg. 2, Col. 1, Para. 4, “In this work, we introduce a sequential label cleaning procedure that maximises the number of corrected samples under a total resource budget”; Bernhadt, Pg. 1, Col. 2, Fig. 1).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the identifying of data sets for annotation, where identifying data sets automatically assigns a priority to the dataset of Bernhardt in view of Wong and He with the dataset-level determining of a priority score of Northcutt in order to make cleaning determinations at the dataset level (Northcutt, Pg. 4, table 1, “We observe that error rates vary across datasets, from 0.15% (MNIST) to 10.12% (QuickDraw); unsurprisingly, simpler datasets, datasets with more carefully designed labeling methodologies, and datasets with more careful human curation generally had less error than datasets that used more automated data collection procedures”, where knowledge of a dataset as a whole can be used to generalize details about the associated data samples), which will provide additional information that may be useful for maximizing the number of corrections for a total resource budget (Bernhardt, Pg. 2, Col. 1, Para. 4, “In this work, we introduce a sequential label cleaning procedure that maximises the number of corrected samples under a total resource budget”).
Additionally, CPlusPlus.com teaches . . . [pushing data to] queue [to prioritize performance of a task] . . . (Pg. 1, Para. 1, “std::queue::push . . . Inserts a new element at the end of the queue, after its current last element. The content of this new element is initialized to val”; Pg. 2, Fig. “Example”, “while (!myqueue.empty()){ std::cout << ' ' << myqueue.front(); myqueue.pop(); }”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the prioritizing the data sets by pushing of the data sets to an annotation environment based on their priority scores of Bernhardt in view of Wong, He, and Northcutt with the pushing data to queue to prioritize performance of a task of CplusPlus.com in order to use a data structure with well-documented functionality and low computational complexity for push and pop functionality to ensure high priority data sets are annotated before lower priority data sets (CPlusPlus.com, Pg. 2, Sect. “Complexity”, “one call to push_back on the underlying container”, where insertion and removal of elements is less complex than alternative data structures, such as sets; see generally, CPlusPlus.com, Pg. 1-2, where the functionality of the queue data structure is well-documented).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt in view of Wong, He, and Chen et al. (hereinafter Chen) (“FOCUS: Dealing with Label Quality Disparity in Federated Learning”).
Regarding Claim 6, Bernhardt in view of Wong and He teach the method of claim 1, wherein prioritizing comprises identifying . . . the data sets that deviate the most from the defined performance range (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where each “batch of samples”, must be identified to be “ranked”, and where in view of He, are the data that deviate from the defined performance range, see He, Para. [0050], “the subject-matter . . . can optionally include . . . [identifying and performing operation on] a predetermined range of target performance values”, where, if all data that deviates from “a predetermined range of target performance values” are identified, then the identified data must include the ten percent of data sets that deviate the most).
The reasons of obviousness are discussed above in regard to the rejection of Claim 1 and remain
applicable here.
Bernhardt in view of Wong and He do not explicitly disclose . . . sites in a federated learning setup that include . . . .
However, Chen teaches . . . [identifying] sites in a federated learning setup that include [the most challenging data] (Pg. 1, Col.1, Abstract, “Ubiquitous systems with End-Edge-Cloud architecture are increasingly being used in healthcare applications. Federated Learning (FL) is highly
useful for such applications . . . FOCUS has been experimentally evaluated on both synthetic data and real-world data. The results show that it effectively identifies clients with noisy labels”, where “clients” are within the broadest reasonable interpretation of sites).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the prioritizing of data sets during model training, comprising identifying the that deviate the most from the define performance range of Bernhardt in view of Wong and He with the identifying of sites, in a federated learning healthcare setup, that include the most challenging data of Chen in order to maximize the number of corrected labels for a resource amount (Bernhardt, Pg. 2, Col. 1, Para. 4, “In this work, we introduce a sequential label cleaning procedure that maximises the number of corrected samples under a total resource budget”; Bernhadt, Pg. 1, Col. 2, Fig. 1), where label noise may be significantly correlated with originating device site (Chen, Pg. 1, Col. 2, Para. 1-3, “one key challenge that remains open and hinders wide spread adoption of FL in ubiquitous systems, especially in the healthcare domain, is label quality disparity . . . we propose the Federated Opportunistic Computing for Ubiquitous System (FOCUS) approach to address this challenging problem. It is designed to identify clients with noisy labels”).
Claims 11-12 are rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt in view of Northcutt.
Regarding Claim 11, Bernhardt in view of Northcutt teach a system for data selection for training a deep learning network, comprising (Bernhardt, Pg. 1, Abstract, “This work advocates for a data-driven approach to prioritising samples for re-annotation—which we term “active label cleaning” . . . the proposed active label cleaning enables correcting labels up to 4× more effectively than typical random selection”, where “data-driven” and “active” selection of training data sets for cleaning is used for training a deep learning network, Bernhardt, Pg. 3, Col. 2, Table 1, “1: θ ← TRAINROBUSTMODEL(D)”; see also Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”, and implemented as a system, see generally Bernhardt, Pg. 3, Col. 2, Table 1, where the pseudocode algorithm could not be implemented without a system with a processor):
a datastore configured to store a plurality of data sets (Bernhardt, Pg. 3, Col. 1, Para. 2, “To analyse label noise scenarios, we experiment with two imaging datasets, namely CIFAR10H and NoisyCXR, containing multiple annotations per data point, which helps us to model the true label distributions and associated labelling cost”, where the system must have a datastore to use the data as input, see Bernhardt, Pg. 3, Col. 2, Table 1, “Input: D = {(xi, l^i)}Ni=1: Dataset with noisy labels),
wherein each data set of the plurality of data sets is assigned an application dependent clinical evaluation metric that correlates with a clinical outcome associated with a respective data set (Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, where the “predicted posteriors” are an evaluation metric because they are used to evaluate the “label[s]” see also Bernhardt, Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy”, where the “predicted posteriors” are correlated with the clinical outcome of result accuracy, and are therefore an application dependent clinical evaluation metric when used for evaluating “labels” from “radiology reports” for “medical imaging”, see Bernhardt, Pg. 1, Col. 1, Para. 2, “Diagnostic labels were extracted from the radiology reports via an error-prone automated process”; Bernhardt, Pg. 3, Col. 2, Table 1, “5. j ← arg maxi∈Iavail Φ(xi,l^i; θ) Rank (Eq. (2))” and Bernhardt, Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”; “predicted posteriors” are correlated with the clinical outcome of result accuracy for the respective dataset, see Bernhardt, Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy” and see generally Bernhardt, Pg. 1, Col. 1, Para. 2, “Diagnostic labels were extracted from the radiology reports via an error-prone automated process”, and are calculated and assigned for each sample, see Bernhardt, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially, and, in view of Northcutt, each set, see Northcutt, Pg. 1, Abstract, “We identify label errors in the test sets of 10 of the most commonly-used computer vision, natural language, and audio datasets, and subsequently study the potential for these label errors to affect benchmark results”; Northcutt, Pg. 4, table 1, “We observe that error rates vary across datasets, from 0.15% (MNIST) to 10.12% (QuickDraw)”);
the deep learning network configured to perform a task relating to medical imaging (Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model, pθ(ˆy|x), parametrised by θ”; Bernhardt, Pg. 3, Col. 2, Table 1, “1: θ ← TRAINROBUSTMODEL(D)”, where the task is “classification”; Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”, where the model is a deep learning network, for a “medical imaging” task, see Bernhardt, Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”);
and a processor configured to train the deep learning network using the plurality of data sets (Bernhardt, Pg. 3, Col. 2, Table 1, “Input: D = {(xi, l^i)}Ni=1: Dataset with noisy labels . . . 13: θ ← UPDATE(θ, D) . Fine-tune model”, where the data sets are used to iteratively train the model, which is a deep learning network, Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”; see generally Bernhardt, Pg. 4, Col. 2, Para. 3, “The self-supervised models have been trained with BYOL6 using a ResNet-50 encoder. The final model is trained using the NIH dataset9 (80% training, 20% validation), using an effective batch size of 4800 (batches of 600 pairs of images of size 224 × 224 on 8 GPUs) for 1000 epochs. The momentum parameter τ used in BYOL teacher encoder was set to 0.99”),
the processor configured to prioritize one or more data sets of the plurality of data sets for use in the training based on the application dependent clinical evaluation metric (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where the “predicted label correctness and ambiguity”, which is the “predicted posteriors” application dependent clinical evaluation metric, see Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, is used to prioritize data sets of the plurality of data sets in training by “cleaning” during an earlier “iteration” of “fine-tun[ning]”, Bernhardt, Pg. 3, Col. 2, Table 1, “13: θ ← UPDATE(θ, D) . Fine-tune model”, which, as discussed above, requires a processor).
The reasons of obviousness are discussed above in regard to the rejection of Claim 4 and remain
applicable here.
Regarding Claim 12, Bernhardt in view of Northcutt teach the system of claim 11, wherein during training of the deep learning network, the deep learning network is tested using a testing metric that is different than the evaluation metric (Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”; Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model”, where, as discussed above, the “predicted posteriors” are the application dependent clinical evaluation metric, and “accuracy”, a different testing metric, is also computed, see Bernhardt, Pg. 6, Col. 2, Table 3, “We compare classification accuracy on true, noisy, and cleaned labels”; and where both metrics are used during training, see Bernhardt, Pg. 3, Col. 2, Table 1, “5. j ← arg maxi∈Iavail Φ(xi,l^i; θ) Rank (Eq. (2)), where the “predicted posteriors” are calculated at each training iteration, and Bernhardt, Pg. 5, Fig. 3, where “accuracy” is computed for different “Number[s] of collected relabels on the dataset”, where a batch is relabeled at each iteration of training, see Bernhardt, Pg. 3, Col. 2, Table 1, “13: θ ← UPDATE(θ, D) . Fine-tune model”).
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt in view of Northcutt and Wong.
Regarding Claim 13, Bernhardt in view of Northcutt and Wong teach the system of claim 11, wherein the application dependent clinical evaluation metric (Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, where, as discussed above the “predicted posteriors” are an application dependent clinical evaluation metric)
comprises an assessment of how well an output of the deep learning network guided a clinician during a medical procedure (Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”; Wong, Pg. 1, Abstract, “Purpose . . . we evaluate the performance of implemented DC models in the clinical radiotherapy (RT) planning workflow and report on user experience”, where, in view of Wong, the evaluation metric includes an assessment of the “models” in “clinical radiotherapy” medical procedures; Wong, Pg. 2, Col. 1-2, Para. 4-1, “the auto-segmentation software prospectively generated DCs to be reviewed and edited on all patients undergoing RT treatment”, where task outputs by the model are subsequently used to guide the clinician in a planning procedure, Wong, Pg. 2, Col. 2, Para. 3, “Generated DCs underwent manual review and were edited as needed prior to being used for RT treatment planning”, and then assessed by how well they performed, Wong, Pg. 1, Abstract, “Methods and materials: DC models were implemented at two cancer centers and used to generate OAR and CTVs for all patients undergoing RT for a central nervous system (CNS), head and neck (H&N), or prostate cancer. Radiation Therapists/Dosimetrists and Radiation Oncologists completed post-contouring surveys rating the degree of edits required for DCs (1 = minimal, 5 = significant) and overall DC satisfaction (1 = poor, 5 = high)”).
The reasons of obviousness are discussed above in regard to the rejection of Claim 9 and remain
applicable here.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt in view of Northcutt and CapeStart.
Regarding Claim 14, Bernhardt in view of Northcutt teach the system of claim 11, wherein . . . training of the deep learning network (Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”).
The remaining elements are substantially the same as the limitations of Claim 19, therefore it is rejected under the same rationale.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt in view of Northcutt, CapeStart, and He.
Regarding Claim 15, Bernhardt in view of Northcutt teach the system of claim 11, wherein the processor is configured to prioritize the data sets . . . (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where the “predicted label correctness and ambiguity” is used to prioritize data sets in training by “cleaning” during an earlier “iteration” of “fine-tun[ning]”, Bernhardt, Pg. 3, Col. 2, Table 1, “13: θ ← UPDATE(θ, D) . Fine-tune model”, which, as discussed above, requires a processor).
The remaining elements are substantially the same as the limitations of Claim 5, therefore it is rejected under the same rationale.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt in view Northcutt and CPlusPlus.com.
Regarding Claim 16, Bernhardt in view of Northcutt and CPlusPlus.com teach the system of claim 11, wherein the processor is configured to assign the plurality of data sets a priority score based on the application dependent clinical evaluation metric and prioritize the one or more data sets (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where the “predicted label correctness and ambiguity”, which, as discussed above, is the “predicted posteriors” application dependent clinical evaluation metric, see Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, is used to prioritize data sets in training by “cleaning” during an earlier “iteration” of “fine-tun[ning]”, Bernhardt, Pg. 3, Col. 2, Table 1, “13: θ ← UPDATE(θ, D) . Fine-tune model”, which, as discussed above, requires a processor; and where, in view of Northcutt, the evaluation metric, “error rates” is assigned at the level of “datasets” and forms the basis for the “% error” priority score, see Northcutt, Pg. 1, Abstract, “We identify label errors in the test sets of 10 of the most commonly-used computer vision, natural language, and audio datasets, and subsequently study the potential for these label errors to affect benchmark results”; Northcutt, Pg. 4, table 1, “We observe that error rates vary across datasets, from 0.15% (MNIST) to 10.12% (QuickDraw)”; where, in the context of the method of Bernhardt, “% error”, which is the rate “label errors” in a “dataset”, expressed as a percent by multiplying by 100, are within the broadest reasonable interpretation of priority scores because the method is premised on prioritizing the greatest return in corrected labels for an investment of resources, see Bernhardt, Pg. 2, Col. 1, Para. 4, “In this work, we introduce a sequential label cleaning procedure that maximises the number of corrected samples under a total resource budget”; Bernhadt, Pg. 1, Col. 2, Fig. 1)
by pushing the one or more data sets to an annotation queue according to their priority score (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, which, in view of CPlusPlus.com, is effectuated by pushing the data sets to an annotation queue according to their priority score, see CPlusPlus.com, Pg. 1, Para. 1, “std::queue::push . . . Inserts a new element at the end of the queue, after its current last element. The content of this new element is initialized to val”; Pg. 2, Fig. “Example”, “while (!myqueue.empty()){ std::cout << ' ' << myqueue.front(); myqueue.pop(); }”).
The reasons of obviousness are discussed above in regard to the rejection of Claim 4 and remain
applicable here.
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt in view of Northcutt, He, Kohler et al. (hereinafter Kohler) (“Uncertainty Based Detection and Relabeling of Noisy Image Labels”), and Smit et al. (hereinafter Smit) (“CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT”).
Regarding Claim 17, Bernhardt in view of Northcutt and He teach the system of claim 11, wherein the processor is configured to . . . [perform iterative label correcting] of data sets that deviate the most from a predefined performance range based on the application dependent clinical evaluation metric for subsequent training of the deep learning network (Bernhardt, Pg. 3, Col. 2, Table 1, “Input: D = {(xi, l^i)}Ni=1: Dataset with noisy labels . . . 13: θ ← UPDATE(θ, D) . Fine-tune model”, where the data sets, see Northcutt, Pg. 4, table 1, “We observe that error rates vary across datasets, from 0.15% (MNIST) to 10.12% (QuickDraw)”, are updated based on the “predicted posteriors” application dependent clinical evaluation metric, see Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, where in view of He, are the data that deviate from the defined performance range, see He, Para. [0050], “the subject-matter . . . can optionally include . . . [identifying and performing operation on] a predetermined range of target performance values”, where, if all data that deviates from “a predetermined range of target performance values” are identified, then the identified data must include the data sets that deviate the most, which are used to iteratively train the model that is a deep learning network, Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”; Bernhardt, Pg. 3, Col. 2, Table 1, where the pseudocode algorithm could not be implemented without a system with a processor; see generally Bernhardt, Pg. 4, Col. 2, Para. 3, “The self-supervised models have been trained with BYOL6 using a ResNet-50 encoder. The final model is trained using the NIH dataset9 (80% training, 20% validation), using an effective batch size of 4800 (batches of 600 pairs of images of size 224 × 224 on 8 GPUs) for 1000 epochs. The momentum parameter τ used in BYOL teacher encoder was set to 0.99”).
The reasons of obviousness are discussed above in regard to the rejection of Claim 1, for the combination with He, and in regard to the rejection of Claim 4, for the combination with Northcutt, and remain applicable here.
Bernhardt in view of Northcutt and He do not explicitly disclose . . . only use the ten percent . . . (where the first iteration of the pseudocode algorithm uses an unspecified percentage of the data sets and all of the data, instead of only that percentage, are used for the subsequent training at each iteration).
However, Kohler teaches . . . use the ten percent . . . [most challenging for each label correction iteration] (Pg. 1, Abstract, “Data with noisy labels can therefore be cleaned in an iterative process”; Pg. Pg. 3, Col. 1, Para. 4, “The noisy images are identified by taking the top 10% of most uncertain images”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the use of a processor to perform iterative label correction of data sets that deviate the most from a predefined performance range, based on an application dependent clinical evaluation metric, for subsequent training of a deep learned network of Bernhardt in view of Northcutt and He with the use of the ten percent most challenging data sets for each label correction iteration of Kohler in order to conduct priority-based relabeling using an easy and effective implementation approach (Kohler, Pg. 1, Col. 1, Abstract, “Our proposed method can be easily implemented, and shows promising performance on the task of noisy label detection on CIFAR-10 and CIFAR-100”; see generally Bernhardt, Pg. 1, Col. 2, Para. 2, “Due to the practical constraints on the total number of reannotations, samples often need to be prioritised to maximise the benefits of relabelling efforts”).
Additionally, Smit teaches . . . only . . . [using human-generated annotations for subsequent training] (Pg. 1, Col. 2, Fig. 1, “We introduce a method for radiology report labeling, in which a biomedically pretrained BERT model is first trained on annotations of a rule-based labeler, and then fine-tuned on a small set of expert annotations”, where “fine-tun[ing]” is subsequent training).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the use a processor to perform iterative label correction, by human annotators on the ten percent of data sets that deviate the most from a predefined performance range, based on the application dependent clinical evaluation metric, for subsequent training of a deep learned network of Bernhardt in view of Northcutt, He, and Kohler with the use of only human-generated annotations for subsequent training of Smit in order to utilize only labels created by trusted and qualified annotators for model fine-tuning (compare Bernhardt, Pg. 1, Col. 1, Para. 1, “Labelling errors can occur due to automated label extraction . . . At training time, incorrect labels hamper the generalisation of predictive models, as labelling errors may be memorised by the model resulting in undesired biases” with Smit, Pg. 8, Col. 1, Para. 3, “collection of expert labels can produce a small set of high quality labels”), which will include only the ten percent most challenging data sets during the subsequent training in the first iteration of the pseudocode algorithm.
Response to Arguments
Applicant's arguments filed on December 18th, 2025 have been fully considered. Each argument is addressed in detail below.
I. Applicant argues the rejections to the claims, under 35 USC § 112(b), should be withdrawn
(Applicant’s Remarks, 12/18/2025, Pg. 6, Section “II. Claim Rejections under 35 U.S.C. § 112”).
Applicant’s amendments have overcome each and every rejection to the claims, under 35 USC §
112(b), previously set forth in the September 30th, 2025 Office Action. As a result, these rejections have
been withdrawn.
II. Applicant argues the rejections to the claims, under 35 USC § 101, should be withdrawn
(Applicant’s Remarks, 12/18/2025, Pg. 6-9, Section “III. Claim Rejections under 35 U.S.C. § 101”).
1) First, Applicant argues the amended claims do not recite mental processes (Step 2A, Prong 1). Specifically, Applicant argues the claimed subject matter cannot practically be performed in the human mind, as required by MPEP 2106.04(a)(2), because each limitation affirmatively requires machine execution and interaction with technical systems that require computer components and clinical workflows (Pg. 6-7, Para. 6-1). Additionally, Applicant argues the metal processes are based on an evaluation metric, which Applicant asserts is computed from machine performed tasks on clinical outcomes (Pg. 7, Para. 1). In support of these assertions, Applicant cites Ex parte Desjardins (Appeal 2024-000567) to argue training a machine learning model, deploying a machine learning model, and computing an application-dependent clinical evaluation metric correlated with the model are not mental processes because they are fundamentally rooted in computer technology (Pg. 8-9, Para. 3-1).
According to MPEP 2106.04(a)(2)(III), “The courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea . . . the "mental processes" abstract idea grouping is defined as concepts performed in the human mind, and examples of mental processes include observations, evaluations, judgments, and opinions” (internal quotation marks omitted).
Additionally, according to MPEP 2106.04(a), “A Claim That Requires a Computer May Still Recite a Mental Process . . . examiners should review the specification to determine if the claimed invention is described as a concept that is performed in the human mind and applicant is merely claiming that concept performed 1) on a generic computer, or 2) in a computer environment, or 3) is merely using a computer as a tool to perform the concept. In these situations, the claim is considered to recite a mental process”.
Here, as discussed in detail above, the claimed subject matter requires computer components and particular technical environments or fields of use. However, as also discussed in detail above, the claimed subject matter recites mental processes that are merely performed on generic computer components and in generic technical environments. Specifically, the mental processes of computing and identifying are merely applied with generic machine learning components in a generic clinical computing environment. Furthermore, there are no positively recited limitations, in either the original or the amended claims, which would distinguish the evaluation metric from an opinion, which could be generated by human mental processes of evaluation. As a result, while the claimed subject matter includes the recitation of generic computer components, the claimed mental processes are not fundamentally rooted in computer technology. Therefore, the claimed subject matter, as currently formulated, is fundamentally distinct from the subject matter of the claims discussed in Desjardins.
As a result, the arguments are not persuasive.
2) Second, Applicant argues the amended claims, even if abstract, are integrated into a practical application (Step 2A, Prong 2). Specifically, Applicant argues “clinical evaluation metrics to control retraining of a machine learning model by prioritizing specific data sets” and “closed-loop technical workflow in which post-deployment clinical performance directly informs retraining behavior” are a technological improvement to the functioning of a computer under MPEP § 2106.05(a) (Pg. 7, Para. 4). Additionally, Applicant argues the “recitation of a medical imaging task, a clinical environment, a medical procedure, and a clinical outcome evaluation metric confines the claim to a specific medical-technical context and prevents preemption of abstract data analysis” (Pg. 7, Para. 5). As such, Applicant argues the claims “recites a sequence of technical operations that alter how a machine learning system is trained and retrained based on real-world clinical performance”, which “satisfies the practical application requirement under MPEP § 2106.05(c)” (Pg. 7-8, Para. 5-1). In support of these assertions, Applicant cites Ex parte Desjardins (Appeal 2024-000567) to argue the claims should be found subject matter eligible because they constitute a technological improvement (Pg. 9, Para. 2-3).
According to MPEP 2106.05(a), “An important consideration in determining whether a claim improves technology is the extent to which the claim covers a particular solution to a problem or a particular way to achieve a desired outcome, as opposed to merely claiming the idea of a solution or outcome . . . It is important to note, the judicial exception alone cannot provide the improvement. The improvement can be provided by one or more additional elements”.
Additionally, according to MPEP 2106.05(c), “when making a determination of whether a claim satisfies the particular transformation consideration . . . the following factors are relevant to the analysis . . . Whether the transformation is extra-solution activity or a field-of-use (i.e., the extent to which (or how) the transformation imposes meaningful limits on the execution of the claimed method steps)”.
Furthermore, according to MPEP 2106.05(f), “Another consideration when determining whether a claim integrates a judicial exception into a practical application in Step 2A Prong Two . . . [is that] A claim having broad applicability across many fields of endeavor may not provide meaningful limitations that integrate a judicial exception into a practical application”.
Here, merely stating the data sets “are prioritized for use” based on an “evaluation metric that correlates with . . . model perform[ance]” (Claim 1) does not constitute a technological improvement. Neither the prioritization for use nor the determination of an evaluation metric are described with sufficient particularity to qualify as a particular way of achieving a solution or a particular solution itself. Instead, the claims, as currently formulated, amount to merely claiming the idea of data prioritization to improve model performance. As a result, the recitation of generic machine learning components amounts to merely claiming the performance of abstract ideas on generic computer components.
Additionally, merely stating the task is “a medical imaging task”, the deploying is to a “clinical environment”, and the evaluation metric is an “application dependent clinical evaluation metric” (Claim 1), does not satisfy the particular transformation consideration. Specifically, the recitation of clinical and medical imaging does not impose any particularity to the mental processes of computing an evaluation metric or identifying information. Instead, it merely links the use of the recited abstract ideas to a medical field-of-use. This is evidenced by the fact that, even when linked to the recited field of use, the claimed subject matter has broad applicability.
As a result, the claimed subject matter does not constitute a technological improvement and the recitation of medical and clinical subject matter fails to sufficiently limit the recited abstract ideas. Therefore, the claimed subject matter, as currently formulated, is fundamentally distinct from the subject matter of the claims discussed in Desjardins.
As a result, the arguments are not persuasive.
3) Third, Applicant argues the amended claims, even if abstract, are significantly more (Step 2B). Specifically, Applicant argues “[t]he claims describe a specific technological solution to the technical problem of improving machine learning performance in clinical medical imaging systems” (Pg. 8, Para. 2). In support of these assertions, Applicant cites Ex parte Desjardins (Appeal 2024-000567) to argue the claims should be found subject matter eligible because they constitute a technological improvement (Pg. 9, Para. 2-3).
Here, as discussed in regard to Step 2A, Prong 2, the claims, as currently recited, do not constitute a technological solution to a technological problem and are fundamentally distinct from the subject matter of the claims discussed in Desjardins. Applicant has not provided any additional arguments in favor of subject matter eligibility at Step 2B, which were not already addressed at Step 2A, Prong 2. As a result, the arguments that the claims are significantly more than the claimed abstract ideas are unpersuasive for substantially the same reasoning.
As a result, the arguments are not persuasive.
III. Applicant argues the rejections to the claims, under 35 USC § 103, should be withdrawn
(Applicant’s Remarks, 12/18/2025, Pg. 6-9, Section “III. Claim Rejections under 35 U.S.C. § 103”).
In response to Applicant’s amendments, the previously communicated rejections under 35 U.S.C. § 103, have been withdrawn. However, Applicants arguments are not persuasive in light of the new grounds for rejection, under 35 U.S.C. § 103, discussed in detail above. The new grounds of rejection rely on new prior art of record to teach the new combinations of elements in the amended claims, which were not presented in these arrangements in any of the previously presented claims. As a result, Applicant arguments against the previously communicated rejections under 35 U.S.C. § 103 are rendered moot.
However, for clarity of the record and in the interest of compact prosecution, arguments still relevant to the new grounds of rejection are discussed below. Relevant MPEP excerpts are reproduced here:
According to MPEP 2111, “During patent examination, the pending claims must be given their broadest reasonable interpretation consistent with the specification” (internal quotation marks omitted) (see also Phillips v. AWH Corp., 415 F.3d 1303, 1316, 75 USPQ2d 1321, 1329 (Fed. Cir. 2005)).
Additionally, according to MPEP 2145, “One cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references” (see also In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981)).
Next, according to MPEP 2143, “a Prima Facie Case of Obviousness” can be shown through “Some Teaching, Suggestion, or Motivation in the Prior Art That Would Have Led One of Ordinary Skill To Modify the Prior Art Reference or To Combine Prior Art Reference Teachings To Arrive at the Claimed Invention”, which “may be implicit and may be found in the knowledge of one of ordinary skill in the art, or, in some cases, from the nature of the problem to be solved”.
Finally, according to 37 C.F.R. 1.111(b), “In order to be entitled to reconsideration or further examination, . . . The reply by the applicant or patent owner must . . . specifically points out the supposed errors in the examiner’s action . . . A general allegation that the claims define a patentable invention without specifically pointing out how the language of the claims patentably distinguishes them from the references does not comply with the requirements of this section”.
1) First, Applicant argues that Bernhardt, Wong, and Northcutt, alone or in combination, fail to anticipate or make obvious each and every limitation of the amended independent claims. In support of this argument, Applicant further argues i) each of Bernhardt, Wong, and Northcutt, fail to teach or suggest each and every limitation of the amended independent claims, ii) the relied upon motivations to combine Bernhardt with Wong are insufficient, and iii) the relied upon motivations to combine Bernhardt with Northcutt are insufficient.
Regarding i), Applicant argues each of Bernhardt, Wong, and Northcutt, individually, fail to teach or suggest "computing an application dependent clinical evaluation metric that correlates with a clinical outcome for each instance of the machine trained model performing the medical imaging task for a medical procedure," "identifying, based on the application dependent clinical evaluation metric, one or more data sets for which performance of the machine trained model deviates from a defined performance range," and "retraining the machine learning model where the one or more data sets are prioritized for use over data sets that do not deviate from the defined performance range" (Claim 1).
However, Bernhardt in combination with either Wong and He (amended Claim 1 and amended Claim 18) or Northcutt (amended Claim 11) are relied upon to teach these limitations. Additionally, none of the arguments against Wong (Pg. 12-13, Para. 2-3) or Northcutt (Pg. 13-14, Para. 4-1) specifically pointing out any failures of these references to teach claim elements that they are relied upon to teach (see 37 C.F.R. 1.111(b)). Instead, these arguments amount to attacking Wong and Northcutt, individually, for failure to teach elements that Bernhardt is relied upon to teach (MPEP 2145). Therefore, the arguments in regard to Wong and Northcutt are not persuasive.
Similarly, arguments asserting that Bernhardt individually fails to teach or suggest deployment in a clinical environment or preforming operations post-clinical deployment (Pg. 11, Para. 2) are not persuasive because Wong is relied upon to teach this element (MPEP 2145). However, unlike the arguments against Wong and Northcutt, the Applicant does assert a shortcoming with Bernhardt that is relevant to what the reference is relied upon to teach. Specifically, Applicant argues Bernhardt fails to disclose identifying data sets where model performance deviates because the data sets are identified based on difficulty of relabeling instead of difficulty for the model (Pg. 11-12, Para. 2-1). However, as discussed in detail above, the Bernhardt discloses identifying data sets based on “estimated noisiness” (Pg. 2-3, Col. 2-1, Para. 3-1, “To this end, we propose to rank available samples by the following scoring function Φ: . . . [equation 2]. The first term . . . corresponds to the estimated noisiness”), which directly corresponds with model performance (Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy”). This disclosure, in combination with the teachings of He (Para. [0050], “the subject-matter . . . can optionally include . . . [identifying and performing operation on] a predetermined range of target performance values”), is withing the broadest reasonable interpretation of identifying data sets where model performance deviates from a performance range (MPEP 2111). Therefore, the arguments in regard to Bernhardt are not persuasive.
As a result, the arguments are not persuasive.
As to ii), Applicant argues the relied upon motivations to combine Bernhardt with Wong are insufficient because it would require impermissible hindsight to arrive at the subject matter of the claims and the stated motivation to combine fails to meet the standard of an articulated rationale with a rational underpinning (Pg. 14, Para. 2; Pg. 15, Para. 2). In support of this argument, Applicant argues the combination of Bernhardt with Wong is insufficient for multiple reasons, labeled below as a) through f). While each argument is address below, an overview of the combination of Bernhard with Wong and the motivation for this combination are discussed first to facilitate an evaluation of Applicant’s arguments.
As discussed in detail above, Bernhardt discloses the use of medical data, “Diagnostic labels were extracted from the radiology reports”, for a “medical imaging” task (Pg. 1, Col. 1, Para. 2, “Diagnostic labels were extracted from the radiology reports via an error-prone automated process”; Pg. 3, Col. 2, Table 1, “5. j ← arg maxi∈Iavail Φ(xi,l^i; θ) Rank (Eq. (2))” and Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”). Additionally, Bernhardt suggests that its methodology could be deployed into an environmental “setting”, which, in the context of medical imaging, would be a clinical environment (Pg. 7, Col. 1, Para. 1, “extensions to active cleaning will significantly broaden its application scope, enabling more reliable deployment of machine learning systems in resource-constrained settings”). It would be entirely reasonable to conclude that these disclosures are sufficient to show that Bernhardt teaches deploying the machine trained model to a clinical environment to conduct a task for a medical procedure. However, even if concluded that Bernhardt’s “active cleaning” method for “medical imaging” that “enable[s] more reliable deployment of machine learning systems in resource-constrained settings” does not actually teach deploying the machine trained model to a clinical environment to conduct a task for a medical procedure, it would nevertheless be clear that the differences between the teachings of Bernhardt and the claimed subject matter are well-within the expected use cases for Bernhardt’s method. Out of an abundance of caution, the Office Action relies on Wong to teach deploying the machine trained model to a clinical environment to conduct a task for a medical procedure (Pg. 1, Abstract, “DC models were implemented at two cancer centers and used to generate OAR and CTVs for all patients undergoing RT for a central nervous system (CNS), head and neck (H&N), or prostate cancer”).
As discussed in detail above, one of ordinary skill in the art would have been motivated to combine the prior art of record to arrive at the claimed invention in order to apply the method to a clinical setting, where its benefits are of significant importance and can be utilized by actual patients (Bernhardt, Pg. 1, Abstract, “employing experts to remove label noise by fully re-annotating large datasets is infeasible in resource-constrained settings, such as healthcare”; Bernhardt, Pg. 2, Col. 2, Para. 2, “This is imperative in safety-critical domains such as healthcare, as model robustness must be validated on clean labels”; Bernhardt, Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy”; Wong, Pg. 1, Abstract, “DC models were implemented at two cancer centers and used to generate OAR and CTVs for all patients undergoing RT for a central nervous system (CNS), head and neck (H&N), or prostate cancer”) by allowing resource efficient and accurate fine-tunning of models on actual client data, which will offer an improvement over generic models (compare Wong, Pg. 2, Col. 1, Para. 4, “These models were trained using publicly available data; no local institutional data was used”, where fine-tunning on client data is not performed, with Wong, Pg. 3, Col. 1, Para. 2, “objective comparisons do assist in the identification of consistent DC contouring errors that can then be the target of model training and improvement”, where fine-tunning on client data would be useful).
Regarding a), Applicant argues “the motivation is framed at an impermissibly high level of generality” because “[t]he rationale is essentially that applying machine learning methods in healthcare is beneficial and that improving models would help patients”, which is “conclusory reasoning” and insufficient under MPEP 2143 (Pg. 15, Para. 3). This characterization is inaccurate. While the motivation is in part to improve patient outcomes by deploying a model with improved accuracy in a clinical setting, the motivation is also to improve model accuracy for a particular patient population by fine-tuning with actual patient data. Additionally, given the differences between Bernhardt and the claimed subject matter being represented in a suggested use case within Bernhardt, as well as the generality of the deployment in the claims, a general motivation to benefit patients would be sufficient to motivate a person of ordinary skill in the art to modify Bernhardt to arrive at the claimed invention (see MPEP 2143). To put another way, any appearances that the motivation is discussed at a high level of generality are attributable to the fact that the claim itself discusses these elements without any significant detail (e.g. the motivation to deploy a model in a clinical environment needs to be less specific than the motivation to deploy a model in a clinical environment in a specific way or to do a specific action). Therefore, the motivation is commensurate with the level of specificity of the proposed modification.
As a result, this argument is not persuasive.
As to b), Applicant argues “the motivation does not explain why Bernhardt's method would be modified to operate on deployed clinical performance rather than training-time signals. The asserted motivation does not address why a skilled artisan would move from Bernhardt's internal, training-loop prioritization to a post-deployment, clinical feedback loop. The reasoning skips over the core architectural change required by the claim” (Pg. 15-16. Para. 4-1). However, as discussed in detail above, Bernhardt already disclosures training using actual clinical feedback in the form of medical imaging data from radiology reports and a person of ordinary skill in the art would be motivated to use the post-deployment data to improve model accuracy by fine-tuning with population-specific data (see MPEP 2143). Furthermore, Claim 1 notably recites “computing an application dependent clinical evaluation metric . . . for a medical procedure” (Claim 1, ln. 6-7) (emphasis added). This is a recitation of the intended purpose of the computing, not a post-deployment clinical feedback loop (see MPEP 2111).
As a result, this argument is not persuasive.
Regarding c), Applicant argues “the motivation does not explain why a skilled artisan would combine Bernhardt's data-selection mechanism with Wong's deployment in a way that feeds clinical evaluation metrics back into training data prioritization” (Pg. 16, Para. 2). However, as discussed in detail above, Bernhardt discloses this element and, as such, the motivation does not need to explain this point (see MPEP 2145).
As a result, this argument is not persuasive.
As to d), Applicant argues “the motivation improperly relies on results-oriented reasoning” because the motivation “describe benefits of the claimed invention, not reasons found in the prior art that would have prompted the modification” (Pg. 16, Para. 3). However, as discussed in detail above, a person of ordinary skill in the art, would be motivated to modify Bernhardt’s disclosure to arrive at the subject matter of the claims in order to improve patient outcomes by deploying a model with improved accuracy in a clinical setting, and to improve model accuracy for a particular patient population by fine-tuning with actual patient data. Bernhardt explicitly discuss the deployment benefits (Pg. 7, Col. 1, Para. 1, “extensions to active cleaning will significantly broaden its application scope, enabling more reliable deployment of machine learning systems in resource-constrained settings”) and the benefits of fine-tuning on population specific data is a well-founded conclusion that would be drawn by one of ordinary skill in the art (see MPEP 2143).
As a result, this argument is not persuasive.
Regarding e), Applicant argues “the motivation lacks proportionality to the claim scope” because “[t]he motivation does not scale to that complexity. It does not explain why a POSITA would have had a reasonable expectation of success in integrating those steps, nor why the combination would have been routine or predictable” (Pg. 16, Para. 4). However, as discussed in detail above, the complexity of the motivation is proportional to the complexity of the claims. To put another way, any appearances that the motivation lacks complexity are attributable to the fact that the claim itself, discusses these elements without significant detail (e.g. the motivation to deploy a model in a clinical environment needs to be less specific than the motivation to deploy a model in a clinical environment in a specific way or to do a specific action). Therefore, the motivation is commensurate with the level of specificity of the proposed modification (see MPEP 2143).
As to f), Applicant argues Wong does not meaningfully support the stated combination of “dataset-level cleaning, nor annotation prioritization, nor resource-budget optimization”, which Applicant believes demonstrates its inclusion is “an after-the-fact aggregation rather than a coherent technical role” (Pg. 19, Para. 1). However, as discussed in detail above, Bernhardt discloses these elements and, as such, the motivation does not need to explain this point (see MPEP 2145). Additionally, given the degree of differences between Bernhardt and the claimed subject matter, as well as the general description of the deployment in the claims, a general motivation to combine the teachings of Wong, as opposed to a specific technical motivation, would be sufficient to motivate a person of ordinary skill in the art to modify Bernhardt to arrive at the claimed invention (see MPEP 2143). To put another way, any appearances that the motivation is an after-the-fact aggregation are attributable to the fact that the claim itself, discusses the deployment of the model in a clinical environment without significant connection to the training, computing, identifying, and retraining (e.g. the motivation to deploy a model in a clinical environment needs to be less specific than the motivation to deploy a model in a clinical environment in a specific way or to do a specific action). Therefore, the motivation is commensurate with the level of specificity of the proposed modification (see MPEP 2143).
As a result, this argument is not persuasive.
Regarding iii), Applicant argues the relied upon motivations to combine Bernhardt with Northcutt are not plausible (Pg. 17, Para. 1). In support of this argument, Applicant argues the combination of Bernhardt with Northcutt is insufficient for multiple reasons, labeled below as a) through f). While each argument is address below, an overview of the combination of Bernhard with Northcutt and the motivation for this combination are discussed first to facilitate an evaluation of Applicant’s arguments.
As discussed above, Bernhardt teaches assigning an application dependent clinical evaluation metric to samples (Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, where the “predicted posteriors” are an evaluation metric because they are used to evaluate the “label[s]” see also Bernhardt, Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy”, where the “predicted posteriors” are correlated with the clinical outcome of result accuracy, and are therefore an application dependent clinical evaluation metric when used for evaluating “labels” from “radiology reports” for “medical imaging”, see Bernhardt, Pg. 1, Col. 1, Para. 2, “Diagnostic labels were extracted from the radiology reports via an error-prone automated process”; Bernhardt, Pg. 3, Col. 2, Table 1, “5. j ← arg maxi∈Iavail Φ(xi,l^i; θ) Rank (Eq. (2))” and Bernhardt, Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”; “predicted posteriors” are correlated with the clinical outcome of result accuracy for the respective dataset, see Bernhardt, Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy” and see generally Bernhardt, Pg. 1, Col. 1, Para. 2, “Diagnostic labels were extracted from the radiology reports via an error-prone automated process”, and are calculated and assigned for each sample, see Bernhardt, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially). It would be entirely reasonable to conclude that these disclosures are sufficient to show that Bernhardt teaches dataset level assigning because each sample can reasonably be considered a dataset. However, even if concluded that Bernhardt’s “samples” are not datasets, it would nevertheless be clear that the differences between the teachings of Bernhardt and the claimed subject matter are operations being performed on samples, which could consistently be applied at the dataset level. Out of an abundance of caution, the Office Action relies on Northcutt to teach dataset-level priority scores (Northcutt, Pg. 1, Abstract, “We identify label errors in the test sets of 10 of the most commonly-used computer vision, natural language, and audio datasets, and subsequently study the potential for these label errors to affect benchmark results”; Northcutt, Pg. 4, table 1, “We observe that error rates vary across datasets, from 0.15% (MNIST) to 10.12% (QuickDraw)”; where, in the context of the method of Bernhardt, “% error”, which is the rate “label errors” in a “dataset”, expressed as a percent by multiplying by 100, are within the broadest reasonable interpretation of priority scores because the method is premised on prioritizing the greatest return in corrected labels for an investment of resources, see Bernhardt, Pg. 2, Col. 1, Para. 4, “In this work, we introduce a sequential label cleaning procedure that maximises the number of corrected samples under a total resource budget”; Bernhadt, Pg. 1, Col. 2, Fig. 1).
As discussed in detail above, one of ordinary skill in the art would have been motivated to combine the prior art of record, to modify the data sample level assigning of Bernhardt to be the data-set level assigning of Northcutt, to arrive at the claimed invention in order to make cleaning determinations at the dataset level (Northcutt, Pg. 4, table 1, “We observe that error rates vary across datasets, from 0.15% (MNIST) to 10.12% (QuickDraw); unsurprisingly, simpler datasets, datasets with more carefully designed labeling methodologies, and datasets with more careful human curation generally had less error than datasets that used more automated data collection procedures”, where knowledge of a dataset as a whole can be used to generalize details about the associated data samples), which will provide additional information that may be useful for maximizing the number of corrections for a total resource budget (Bernhardt, Pg. 2, Col. 1, Para. 4, “In this work, we introduce a sequential label cleaning procedure that maximises the number of corrected samples under a total resource budget”).
Regarding a), Applicant argues “the motivation mischaracterizes the nature of what is being combined” because “Bernhardt teaches sample-level prioritization for re-annotation during training using model- derived uncertainty scores, with the express goal of maximizing corrected labels under a resource budget” and “Northcutt teaches dataset-level descriptive statistics about benchmark label error rates for the purpose of analyzing benchmark reliability and test-set contamination” (Pg. 17, Para. 3). However, as discussed above, the dataset-level descriptive statistics about benchmark label error rates of Northcutt are comparable and compatible with the model-derived scores of Bernhardt and Applicant has not specifically pointed out any supposed errors with this combination (see 37 C.F.R. 1.111(b)).
As a result, this argument is not persuasive.
As to b), Applicant argues “Wong is not meaningfully integrated into the stated motivation at all, because the motivation is framed entirely around annotation and dataset cleaning, not clinical deployment or medical procedures. A motivation that purports to combine three references but substantively relies on only two already indicates analytical weakness under MPEP 2143” (Pg. 17, Para. 3). However, none of the independent claims, which Applicant purports to be discussing (Pg. 19, 3), rely on a combination including both Wong and Northcutt. Regardless, the above discussed motivation for combining Bernhardt with Northcutt does not change if Bernhardt was previously combined with Wong (see MPEP 2143).
As a result, this argument is not persuasive.
Regarding c), Applicant argues “the leap from Northcutt's dataset-level observations to "dataset-level determining of a priority score" is unsupported” because “Northcutt reports empirical error rates across well-known benchmarks to make a methodological point about benchmark instability.
Northcutt does not teach assigning priority scores to datasets for cleaning, does not teach operational decision-making based on those statistics, and does not teach using dataset-level characteristics to drive annotation workflows” (Pg. 17-18, Para. 4-1). However, as discussed above, Northcutt is only relied upon to teach data-set level scoring. While the motivation to combine includes a discussion of how the scores of Northcutt can reasonably be considered priority scores within the context of Bernhardt’s disclosure, Northcutt is not relied upon to teach or discuss any of cleaning, decision-making, or annotation workflows. As a result, arguments that simply state that Northcutt fails to disclose these elements, instead of arguing why Northcutt is incompatible with these elements, amounts to attacking references within a combination individually (see MPEP 2145).
As a result, this argument is not persuasive.
Regarding d), Applicant argues “the motivation does not explain why a skilled artisan would combine dataset- level statistics with Bernhardt's sample-level active label cleaning framework” and “Introducing dataset-level generalizations would undermine, rather than complement, Bernhardt's approach, because Bernhardt already assumes heterogeneous difficulty within a dataset and explicitly operates at the sample level” (Pg. 18, Para. 2). However, as discussed in detail above, a person of ordinary skill in the art would be motivated to make dataset-level determinations because identification of dataset-level trends in label noise can lead to efficient generalizations of which data to correct within a resource budget. While it is correct that Bernhardt operates at the sample level to individually determine noisiness, this is not evidence of incompatibility. If the mere fact that a modification was necessary to overcome a difference between a reference and the claimed subject matter could, itself, be used as evidence against the proposed combination then all combinations would be insufficient. Therefore, this argument cannot stand (see MPEP 2143).
As a result, this argument is not persuasive.
Regarding e), Applicant argues the benefit of “provid[ing] additional information that may be useful for maximizing the number of corrections for a total resource budget” is “hindsight reconstruction” because “Neither Bernhardt nor Northcutt suggests that dataset-level error statistics should be combined with sample-level uncertainty metrics to optimize annotation budgets” (Pg. 19, Para. 3). However, a valid motivation to combine references does not require the motivation to combine be explicitly present in any single reference (see MPEP 2143). Instead, the motivation can be implicit within the combined teachings (see MPEP 2143). Instead , as discussed in detail above, a person of ordinary skill with knowledge of Bernhardt’s method of prioritizing data samples for relabeling with a resource constrained budget and Northcutt’s research on dataset-level trends in label noise the art would be motivated to make dataset-level determinations because identification of dataset-level trends in label noise can lead to make efficient generalizations of which data to correct within a resource budget (see MPEP 2143).
As a result, the arguments are not persuasive.
Regarding f), Applicant argues “the motivation lacks proportionality to the modification required. The proposed combination requires redefining Bernhardt's per-sample prioritization mechanism to incorporate dataset-level characteristics inferred from Northcutt's benchmark analysis”, which “provides no explanation why a skilled artisan would reasonably expect success in doing so, nor why such a modification would have been predictable. Under KSR, predictability and reasoned explanation matter, and both are absent here” (Pg. 19, Para. 3). However, as discussed above, a person of ordinary skill with knowledge of Northcutt’s research on dataset-level trends in label noise would be predictably motivated to modify Bernhardt’s method of prioritizing data samples for relabeling with a resource constrained budget so as to successfully make efficient prioritization decisioned based on dataset-level trends (see MPEP 2143). Additionally, this motivation is proportional to the modification from assigning sample-level scores to dataset-level scores, which merely requires a motivation to prioritize data at the dataset level instead of the sample level, and which is fully consistent with Bernhardt’s method, which already operates on batches of samples (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled”).
As a result, the arguments are not persuasive.
2) Second, based on the asserted failures of Bernhardt, Wong, and Northcutt to anticipate or render obvious the independent claims, Applicant argues the dependent claims are allowable. However, as discussed in detail above, arguments in favor of the allowability of the independent claims are not persuasive.
As a result, the arguments are not persuasive.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW BRYCE GOLAN whose telephone number is (571)272-5159. The examiner can normally be reached Monday through Friday, 8:00 AM to 5:00 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571) 270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW BRYCE GOLAN/Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights