DETAILED ACTION
This Office action is in response to a request for continued examination and associated fee payment filed on July 20th, 2026, for Application No. 18/066,079. The amendments filed on June 30th, 2026, have been entered, where claims 1 and 3-20 are presented for examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/30/2026 has been entered.
Claim Objections
Claims 18-20 are objected to because of the following informalities:
“range, , wherein” (Claim 18, Ln. 8) should be “range, wherein” (objection applies equally to dependent claims 19-20).
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 15 and 17-20 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Regarding Claim 15, the claim recites the term “predefined” (ln. 3), which is a relative term that renders the claim indefinite. The term “predefined” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. As a result, a person of ordinary skill in the art would not be reasonably apprised as to whether a given “performance range” qualifies as “a predefined performance range” (ln. 3-4) because it is not clear how much in advance the range must be defined. Therefore, the scope of the claim is indefinite. As a result, the claim is rejected. The claim should be amended to provide a standard for ascertaining the requisite degree of “predefined”.
Regarding Claim 17, the claim recites the term “predefined” (ln. 2), which is indefinite for substantially the same reasoning as discussed in regard to the rejection of claim 15. As a result, the claim is similarly rejected and should be amended in a similar manner.
Regarding Claim 18, the claim recites the term “predefined” (ln. 9), which is indefinite for substantially the same reasoning as discussed in regard to the rejection of claim 15. As a result, the claim is similarly rejected and should be amended in a similar manner.
Regarding Claims 19-20, the claims are rejected because they depend on an indefinite claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 and 3-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract ideas without significantly more.
Regarding Claim 1:
Step 1: Claim 1 is a process claim. Therefore, claims 1 and 3-10 are directed to a statutory category of eligible subject matter.
Step 2A Prong 1: If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the "Mental Processes" grouping of abstract ideas. Here, steps of the claimed method are mental processes. Specifically, the claim recites
“A method for selection of training data sets, the method comprising . . . computing an . . . evaluation metric that correlates with a clinical outcome for each instance of the machine trained model . . . wherein computing the . . . evaluation metric comprises calculating a geometric metric . . . ” (mental process – apart from “the machine trained model” itself, amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a metric comprising a geometric metric, which may be aided by pen and paper) and
“identifying, based on the . . . evaluation metric, one or more data sets for which performance . . . deviates from a defined performance range; and . . . where the one or more data sets are prioritized for use over data sets that do not deviate from the defined performance range” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on identified subset to be prioritized, with reference to a known range, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional elements:
“machine training a model to perform a . . . task . . . the machine trained model performing . . . of the machine trained model . . . retraining the machine learning model” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea) and
“medical imaging . . . wherein the medical imaging task comprises generating a segmentation surface of an anatomy . . . deploying the machine trained model to a clinical environment . . . application dependent clinical . . . the medical imaging task for a medical procedure . . . application dependent clinical . . . between the generated segmentation surface and one or more physical clinical markers associated with the medical procedure . . . application dependent clinical” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional elements:
“machine training a model to perform a . . . task . . . the machine trained model performing . . . of the machine trained model . . . retraining the machine learning model” (mere instructions to apply the exception using generic computer components does not provide an inventive concept) and
“medical imaging . . . wherein the medical imaging task comprises generating a segmentation surface of an anatomy . . . deploying the machine trained model to a clinical environment . . . application dependent clinical . . . the medical imaging task for a medical procedure . . . application dependent clinical . . . between the generated segmentation surface and one or more physical clinical markers associated with the medical procedure . . . application dependent clinical” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
For the reasons above, Claim 1 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 3-10. The additional limitations of the dependent claims are addressed below.
Regarding Claim 3:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 3 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“wherein prioritizing the one or more data sets comprises wherein the one or more data sets are given priority” (mental process – amounts to exercising judgment to evaluate observed or known information in order to prioritize previously identified samples of information, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“in at least one of data transfer, data anonymization, or preprocessing” (transmitting data, anonymizing data, and preprocessing data all amount to insignificant extra-solution activity that is incidental to the claimed subject matter) and
“during retraining of the machine trained model” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“in at least one of data transfer, data anonymization, or preprocessing” (the recited element is well‐understood, routine, and conventional; whether in the form of transmitting data, see generally Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362; see also buySAFE, Inc. v. Google Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014), or preprocessing health data, Ferrão et al., Preprocessing structured clinical data for predictive modeling and decision support, Pg. 1145, Para. 5-6, “the target audience of this article is not limited to researchers, but also extended to those involved in system implementation, training and use in routine practice . . . Considering the strong trends in clinical informatics towards structured data entry, we believe that the preprocessing tasks addressed in this article increasingly play a major role”, or anonymizing health data, Ghinita et al., Anonymous Publication of Sensitive Transactional Data, Pg. 161, Col. 1, Para. 1, “Organizations, such as hospitals, publish detailed data (microdata) about individuals (e.g., medical records) . . . Existing privacy-preserving techniques focus on anonymizing personal data”, Pg. 165, Col. 1, Para. 3, “It is well understood [24] that publishing privacy-sensitive data is caught between the conflicting requirements of privacy and utility”; therefore, the limitation, which is recited with a high level of generality, remains insignificant extra-solution activity even upon reconsideration) and
“during retraining of the machine trained model” (mere instructions to apply the exception using generic computer components does not provide an inventive concept).
Accordingly, Claim 3 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 4:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 4 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“wherein identifying the one or more data sets comprises automatically assigning a priority score to the one or more data sets and prioritizing the one or more data sets . . . according to their priority score” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a priority score, which may happen automatically for simple scoring methods like as binary, and forming a priority opinion based on these scores, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“comprises pushing the one or more data sets to an annotation queue” (pushing data to a queue amounts to insignificant extra-solution activity that is incidental to the claimed subject matter).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“comprises pushing the one or more data sets to an annotation queue” (pushing data to a queue is well‐understood, routine, and conventional; whether in the form of transmitting data, see generally Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362; see also buySAFE, Inc. v. Google Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014), or storing data in memory, see Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); therefore, the limitation, which is recited with a high level of generality, remains insignificant extra-solution activity even upon reconsideration).
Accordingly, Claim 4 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 5:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 5 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“wherein prioritizing the one or more data sets comprises assigning the one or more data sets to annotators such that expert annotators are assigned data sets that deviate a most from the defined performance range to annotate” (mental process – apart from the “assigning” itself, which may require a particular technological environment, amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a specific subset to be evaluated by annotators, with reference to a known range, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“assigning” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“assigning” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
Accordingly, Claim 5 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 6:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 6 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“wherein prioritizing comprises identifying sites in a . . . setup that include data sets that deviate the most from the defined performance range” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a specific subset to be prioritized, with reference to a known performance range, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“federated learning” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“federated learning” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
Accordingly, Claim 6 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 7:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 7 depends on.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“wherein during machine training of the model, the model is tested using a testing metric that is different than the . . . evaluation metric” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea) and
“application dependent clinical” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“wherein during machine training of the model, the model is tested using a testing metric that is different than the . . . evaluation metric” (mere instructions to apply the exception using generic computer components does not provide an inventive concept) and
“application dependent clinical” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
Accordingly, Claim 7 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 8:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 8 depends on.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“wherein the medical imaging data is acquired using one of MRI, CT, X-ray, or Ultrasound” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“wherein the medical imaging data is acquired using one of MRI, CT, X-ray, or Ultrasound” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
Accordingly, Claim 8 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 9:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 9 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“wherein the . . . evaluation metric further provides an assessment of how well an output” (mental process – amounts to exercising judgment to evaluate observed or known information to form an opinion, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“of the machine trained model” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea) and
“application dependent clinical . . . guided a clinician during the medical procedure subsequent to the performance of the task” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“of the machine trained model” (mere instructions to apply the exception using generic computer components does not provide an inventive concept) and
“application dependent clinical . . . guided a clinician during the medical procedure subsequent to the performance of the task” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
Accordingly, Claim 9 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 10:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 10 depends on. Here, the claim recites additional elements that are mental processes. Specifically, the claim recites
“wherein identifying comprises identifying one or more data sets that are the ten percent of data sets that deviate the most from the defined performance range” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a specific subset percentage of data to be prioritized, with reference to a known performance range, which may be aided by pen and paper).
Step 2A Prong 2 & Step 2B: There are no elements left for consideration of implementation within a practical application or for consideration of significantly more.
Accordingly, Claim 10 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 11:
Step 1: Claim 11 is a machine claim. Therefore, claims 11-20 are directed to a statutory category of eligible subject matter.
Step 2A Prong 1: If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the "Mental Processes" grouping of abstract ideas. Here, steps performed by the claimed system are mental processes. Specifically, the claim recites
“wherein each data set of the plurality of data sets is assigned an . . . evaluation metric . . . and wherein the . . . evaluation metric comprises a geometric metric calculated . . . prioritize one or more data sets of the plurality of data sets for use in the training based on the . . . evaluation metric” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a metric, comprising a geometric metric, which is used to form opinions on priority that may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional elements:
“A system for data selection for training a deep learning network, comprising: a datastore configured to . . . the deep learning network configured to perform a task; and a processor configured to train the deep learning network using the plurality of data sets, the processor configured to . . . ” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea);
“store a plurality of data sets” (storing data amount to insignificant extra-solution activity that is incidental to the claimed subject matter); and
“application dependent clinical . . . that correlates with a clinical outcome associated with a respective data set . . . relating to medical imaging . . . wherein the task comprises generating a segmentation surface of an anatomy . . . application dependent clinical . . . between the generated segmentation surface and one or more physical clinical markers associated with a medical procedure . . . application dependent clinical” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional elements:
“A system for data selection for training a deep learning network, comprising: a datastore configured to . . . the deep learning network configured to perform a task; and a processor configured to train the deep learning network using the plurality of data sets, the processor configured to . . . ” (mere instructions to apply the exception using generic computer components does not provide an inventive concept);
“store a plurality of data sets” (storing data is well‐understood, routine, and conventional, see Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); therefore, the limitation, which is recited with a high level of generality, remains insignificant extra-solution activity even upon reconsideration); and
“application dependent clinical . . . that correlates with a clinical outcome associated with a respective data set . . . relating to medical imaging . . . wherein the task comprises generating a segmentation surface of an anatomy . . . application dependent clinical . . . between the generated segmentation surface and one or more physical clinical markers associated with a medical procedure . . . application dependent clinical” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
For the reasons above, Claim 11 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 12-17. The additional limitations of the dependent claims are addressed below.
Regarding Claim 12:
Step 2A Prong 1: See the rejection of Claim 11 above, which Claim 12 depends on.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“wherein during training of the deep learning network, the deep learning network is tested using a testing metric that is different than the . . . evaluation metric” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea) and
“application dependent clinical” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“wherein during training of the deep learning network, the deep learning network is tested using a testing metric that is different than the . . . evaluation metric” (mere instructions to apply the exception using generic computer components does not provide an inventive concept) and
“application dependent clinical” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
Accordingly, Claim 12 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 13:
Step 2A Prong 1: See the rejection of Claim 11 above, which Claim 13 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“wherein the . . . evaluation metric further provides an assessment of how well an output” (mental process – amounts to exercising judgment to evaluate observed or known information to form an opinion, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“of the deep learning network” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea) and
“application dependent clinical . . . guided a clinician during a medical procedure” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“of the deep learning network” (mere instructions to apply the exception using generic computer components does not provide an inventive concept) and
“application dependent clinical . . . guided a clinician during a medical procedure” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
Accordingly, Claim 13 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 14:
Step 2A Prong 1: See the rejection of Claim 11 above, which Claim 14 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“prioritize the one or more data sets” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on prioritization, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“in at least one of data transfer, data anonymization, or preprocessing” (transmitting data, anonymizing data, and preprocessing data all amount to insignificant extra-solution activity that is incidental to the claimed subject matter) and
“wherein the processor is configured to . . . during training of the deep learning network” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“in at least one of data transfer, data anonymization, or preprocessing” (the recited element is well‐understood, routine, and conventional; whether in the form of transmitting data, see generally Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362; see also buySAFE, Inc. v. Google Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014), or preprocessing health data, Ferrão et al., Preprocessing structured clinical data for predictive modeling and decision support, Pg. 1145, Para. 5-6, “the target audience of this article is not limited to researchers, but also extended to those involved in system implementation, training and use in routine practice . . . Considering the strong trends in clinical informatics towards structured data entry, we believe that the preprocessing tasks addressed in this article increasingly play a major role”, or anonymizing health data, Ghinita et al., Anonymous Publication of Sensitive Transactional Data, Pg. 161, Col. 1, Para. 1, “Organizations, such as hospitals, publish detailed data (microdata) about individuals (e.g., medical records) . . . Existing privacy-preserving techniques focus on anonymizing personal data”, Pg. 165, Col. 1, Para. 3, “It is well understood [24] that publishing privacy-sensitive data is caught between the conflicting requirements of privacy and utility”; therefore, the limitation, which is recited with a high level of generality, remains insignificant extra-solution activity even upon reconsideration) and
“wherein the processor is configured to . . . during training of the deep learning network” (mere instructions to apply the exception using generic computer components does not provide an inventive concept).
Accordingly, Claim 14 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 15:
Step 2A Prong 1: See the rejection of Claim 11 above, which Claim 15 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“prioritize the one or more data sets by assigning the one or more data sets to annotators such that expert annotators are assigned data sets that deviate a most from a predefined performance range to annotate” (mental process – apart from the “assigning” itself, which may require the system processor, amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a specific subset to be evaluated by annotators, with reference to a range-based criteria, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“wherein the processor is configured to . . . assigning” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“wherein the processor is configured to . . . assigning” (mere instructions to apply the exception using generic computer components does not provide an inventive concept).
Accordingly, Claim 15 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 16:
Step 2A Prong 1: See the rejection of Claim 11 above, which Claim 16 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“assign the plurality of data sets a priority score based on the . . . evaluation metric and prioritize the one or more data sets . . . according to their priority score” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a priority score based on a metric, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“wherein the processor is configured to” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea);
“application dependent clinical” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use); and
“by pushing the one or more data sets to an annotation queue” (pushing data to a queue amounts to insignificant extra-solution activity that is incidental to the claimed subject matter).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“wherein the processor is configured to” (mere instructions to apply the exception using generic computer components does not provide an inventive concept);
“application dependent clinical” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept); and
“by pushing the one or more data sets to an annotation queue” (pushing data to a queue is well‐understood, routine, and conventional; whether in the form of transmitting data, see generally Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362; see also buySAFE, Inc. v. Google Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014), or storing data in memory, see Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); therefore, the limitation, which is recited with a high level of generality, remains insignificant extra-solution activity even upon reconsideration).
Accordingly, Claim 16 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 17:
Step 2A Prong 1: See the rejection of Claim 11 above, which Claim 17 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“only use the ten percent of data sets that deviate the most from a predefined performance range based on the . . . evaluation metric for subsequent training” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a specific subset percentage of data to be included in future use, based in turn on a previously determined range, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“wherein the processor is configured to . . . of the deep learning network” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea) and
“application dependent clinical” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“wherein the processor is configured to . . . of the deep learning network” (mere instructions to apply the exception using generic computer components does not provide an inventive concept) and
“application dependent clinical” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
Accordingly, Claim 17 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 18:
Step 1: Claim 18 is a process claim. Therefore, claims 18-20 are directed to a statutory category of eligible subject matter.
Step 2A Prong 1: If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the "Mental Processes" grouping of abstract ideas. Here, steps of the claimed method are mental processes. Specifically, the claim recites
“A method for data collection, the method comprising . . . computing an . . . evaluation metric for the processed medical imaging data . . . determining, based on the . . . evaluation metric, that the medical imaging data comprises a data set that deviates from a defined performance range; and prioritizing the medical imaging data” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a metric, which is based a previously known range, which is then used to form opinions on the associated information and what should be prioritized which may be aided by pen and paper);
“wherein determining comprises sorting the medical imaging data in an order based on the . . . evaluation metric and selecting a predefined top percentage of the sorted medical imaging data to define a challenging data subset” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a sorted order and a subset that falls within a known range, which may be aided by pen and paper); and
“by adding the challenging data subset to an original training data set . . . to form a new training data set” (mental process – amounts to exercising judgment to form an opinion that two known or observed training datasets should be considered as components of a new set, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional elements:
“processing the medical imaging data using a machine learned network . . . for retraining of the machine learned network . . . and executing the retraining” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea);
“performing a medical imaging procedure . . . application dependent clinical . . . based on a clinical outcome related to the medical imaging procedure . . . application dependent clinical . . . application dependent clinical . . . used to initially train the machine learned network . . . using the new training data set” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use); and
“to generate medical imaging data” (mere data gathering amounts to insignificant extra-solution activity that is incidental to the claimed subject matter).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional elements:
“processing the medical imaging data using a machine learned network . . . for retraining of the machine learned network . . . and executing the retraining” (mere instructions to apply the exception using generic computer components does not provide an inventive concept);
“performing a medical imaging procedure . . . application dependent clinical . . . based on a clinical outcome related to the medical imaging procedure . . . application dependent clinical . . . application dependent clinical . . . used to initially train the machine learned network . . . using the new training data set” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept); and
“to generate medical imaging data” (generating medical data through clinical procedures is well‐understood, routine, and conventional, see Genetic Techs. Ltd., 818 F.3d at 1377; 118 USPQ2d at 1546; Sequenom, 788 F.3d at 1377-78, 115 USPQ2d at 1157); Cleveland Clinic Foundation 859 F.3d at 1362, 123 USPQ2d at 1088 (Fed. Cir. 2017); therefore, the limitation, which is recited with a high level of generality, remains insignificant extra-solution activity even upon reconsideration).
For the reasons above, Claim 18 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 19-20. The additional limitations of the dependent claims are addressed below.
Regarding Claim 19:
Step 2A Prong 1: See the rejection of Claim 18 above, which Claim 19 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“wherein prioritizing comprises” (mental process – amounts to exercising judgment to evaluate observed or known information in order to form an opinion on prioritization, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“at least one of data transfer, data anonymization, or preprocessing of the medical imaging data” (transmitting data, anonymizing data, and preprocessing data all amount to insignificant extra-solution activity that is incidental to the claimed subject matter) and
“during retraining of the machine learned network” (amounts to mere instructions to apply the judicial exception on generic and unspecialized computer components, which do not impose any meaningful limits on practicing the abstract idea).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“at least one of data transfer, data anonymization, or preprocessing of the medical imaging data” (the recited element is well‐understood, routine, and conventional; whether in the form of transmitting data, see generally Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362; see also buySAFE, Inc. v. Google Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014), or preprocessing health data, Ferrão et al., Preprocessing structured clinical data for predictive modeling and decision support, Pg. 1145, Para. 5-6, “the target audience of this article is not limited to researchers, but also extended to those involved in system implementation, training and use in routine practice . . . Considering the strong trends in clinical informatics towards structured data entry, we believe that the preprocessing tasks addressed in this article increasingly play a major role”, or anonymizing health data, Ghinita et al., Anonymous Publication of Sensitive Transactional Data, Pg. 161, Col. 1, Para. 1, “Organizations, such as hospitals, publish detailed data (microdata) about individuals (e.g., medical records) . . . Existing privacy-preserving techniques focus on anonymizing personal data”, Pg. 165, Col. 1, Para. 3, “It is well understood [24] that publishing privacy-sensitive data is caught between the conflicting requirements of privacy and utility”; therefore, the limitation, which is recited with a high level of generality, remains insignificant extra-solution activity even upon reconsideration) and
“during retraining of the machine learned network” (mere instructions to apply the exception using generic computer components does not provide an inventive concept).
Accordingly, Claim 19 is rejected as being directed to an abstract idea without significantly more.
Regarding Claim 20:
Step 2A Prong 1: See the rejection of Claim 18 above, which Claim 20 depends on. Here, the claim recites additionally elements that are mental processes. Specifically, the claim recites:
“wherein prioritizing the medical imaging data comprises assigning the medical imaging data to annotators such that an expert annotator gets the medical imaging data to annotate” (mental process – apart from the “assigning” itself, which may require a particular technological environment, amounts to exercising judgment to evaluate observed or known information in order to form an opinion on a specific subset to be evaluated by annotators, which may be aided by pen and paper).
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The claim recites the additional element:
“assigning” (amounts to generally linking the use of the judicial exception to a particular technological environment or field of use).
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional element:
“assigning” (merely generally linking the use of the judicial exception to a particular technological environment or field of use does not provide an inventive concept).
Accordingly, Claim 20 is rejected as being directed to an abstract idea without significantly more.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 7-10, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt et al. (hereinafter Bernhardt) (“Active label cleaning: Improving dataset quality under resource constraints”) in view of Wong et al. (hereinafter Wong) (“Implementation of deep learning-based auto-segmentation for radiotherapy planning structures: a workfow study at two cancer centers”) and He et al. (hereinafter He) (Pat. Pub. No. US 2022/0347583 A1).
Regarding Claim 1, Bernhardt teaches a method for selection of training data sets, the method comprising (Pg. 1, Abstract, “Imperfections in data annotation, known as label noise, are detrimental to the training of machine learning models . . . This work advocates for a data-driven approach to prioritising samples for re-annotation—which we term “active label cleaning” . . . the proposed active label cleaning enables correcting labels up to 4× more effectively than typical random selection”, where the “data-driven approach” is a method for “selecti[ng]” data sets, “samples”, for “re-annotation” and where the data sets are for “training of machine learning models”):
machine training a model to perform a medical imaging task, wherein the medical imaging task comprises generating a . . . [classification](Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model, pθ(ˆy|x), parametrised by θ”; Pg. 3, Col. 2, Table 1, “1: θ ← TRAINROBUSTMODEL(D)”, where the task is “classification” for “medical imaging”, see Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”; see also Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”);
. . . ;
computing an application dependent clinical evaluation metric that correlates with a clinical outcome (Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, where the “predicted posteriors” are an evaluation metric because they are used to evaluate the “label[s]”; see also Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy”, where the “predicted posteriors” are correlated with the clinical outcome of result accuracy, and are therefore an application dependent clinical evaluation metric when used for evaluating “labels” from “radiology reports” for “medical imaging”, see Pg. 1, Col. 1, Para. 2, “Diagnostic labels were extracted from the radiology reports via an error-prone automated process”; Pg. 3, Col. 2, Table 1, “5. j ← arg maxi∈Iavail Φ(xi,l^i; θ) Rank (Eq. (2))” and Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”)
for each instance of the machine trained model performing the medical imaging task . . . (Pg. 3, Col. 2, Table 1, “5. j ← arg maxi∈Iavail Φ(xi,l^i; θ) Rank (Eq. (2))” and Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”, where the predicted postures, “Φ(xi,l^i; θ) ”, which as discussed above are the application dependent clinical evaluation metric, are calculated at each iteration, which are within the broadest reasonable interpretation of an instance, of the model performing the “medical imaging” task),
wherein computing the application dependent clinical evaluation metric comprises calculating a . . . metric . . . (Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, where the “predicted posteriors” are an evaluation metric because they are used to evaluate the “label[s]”; see also Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy”, where the “predicted posteriors” are correlated with the clinical outcome of result accuracy, and are therefore an application dependent clinical evaluation metric when used for evaluating “labels” from “radiology reports” for “medical imaging”, see Pg. 1, Col. 1, Para. 2, “Diagnostic labels were extracted from the radiology reports via an error-prone automated process”; Pg. 3, Col. 2, Table 1, “5. j ← arg maxi∈Iavail Φ(xi,l^i; θ) Rank (Eq. (2))” and Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”);
identifying, based on the application dependent clinical evaluation metric, one or more data sets (Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where the “iterate[ve]” “rank[ing]” and selection of a “batch of samples” based on “correctness and ambiguity (i.e. annotation difficulty)” is within the broadest reasonable interpretation of identifying one or more data sets, which are identified based on the “predicted posteriors”, which as discussed above are the application dependent clinical evaluation metric for each model iteration, see Pg. 2-3, Col. 2-1, Para. 3-1, “To this end, we propose to rank available samples by the following scoring function Φ: . . . [equation 2]. The first term . . . corresponds to the estimated noisiness [and the second term] . . . penalises ambiguous cases in the ranking”)
for which performance of the machine trained model deviates . . . (Pg. 2-3, Col. 2-1, Para. 3-1, “To this end, we propose to rank available samples by the following scoring function Φ: . . . [equation 2]. The first term . . . corresponds to the estimated noisiness”, where the “first term” of the application dependent clinical evaluation metrics “corresponds to the estimated noisiness”, which results in reduced performance of the machined trained model, see and Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy”, thus the above discussed identified data sets deviate in model performance); and
retraining the machine learning model where the one or more data sets are prioritized for use over data sets that do not [sufficiently] deviate . . . (Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where the identified data, which as discussed above deviates in model performance, are prioritized for use in active label cleaning over data over data that is not identified because they do not sufficiently deviate; and where active label cleaning is a retraining approach, see Pg. 3, Col. 2, Table 1, “13: θ ← UPDATE(θ, D) . Fine-tune model”, of a machine learned network, see Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”; see Pg. 2, Col. 1, Para. 3, “Prioritising samples for labelling also underpins the paradigm of active learning, whose goal is to select unlabelled samples that would be most beneficial for training in order to improve the performance of a predictive model on a downstream task . . . The key difference here for the proposed approach is that our goal is not only to improve model performance but also to maximise the quality of labels given limited resources, which makes it valuable for both training and evaluation of predictive models. In more detail, we demonstrate how active learning and noiserobust learning (NRL) can play complementary roles in coping with label noise”).
Bernhardt does not explicitly disclose . . . segmentation surface of an anatomy . . . deploying the machine trained model to a clinical environment . . . for a medical procedure . . . geometric . . . between the generated segmentation surface and one or more physical clinical markers associated with the medical procedure (where segmentation, geometric metrics relating to segmentation, and model deployment in a clinical environment for a medical procedure are not specifically discussed, but see Pg. 7, Col. 1, Para. 1, “extensions to active cleaning will significantly broaden its application scope, enabling more reliable deployment of machine learning systems in resource-constrained settings”, where “deployment” is suggested)
. . . from a defined performance range . . . (where the identifying of data for prioritization is not specifically described as within a defined performance range; subsequent recitations of performance range omitted).
However, Wong teaches . . . [a medical task comprising] segmentation surface of an anatomy (Pg. 2, Col. 2, Para. 2, “The OARs to be auto-segmented for each eligible disease site were selected by each center; the OARs available to be selected included brainstem, globe, optic chiasm, optic nerve, parotid, submandibular, mandible, spinal cord, bladder, femoral head, and rectum. The CTV contours available were neck CTV, prostate, and seminal vesicles (SV). DCs for neck CTV included
lymph node levels Ib, II, III, IV, V, and the retropharyngeal and retrostyloid nodes” and Pg. 3, Col. 1, Para. 3, “DSC is defined as D(A,B) = 2|A ∩ B|/(|A| +|B|) and describes the relative overlap of segmentation volumes A and B. DSC values range from 0 for no overlap to 1 for complete overlap. HD is a bi-directional measure to quantify the distance between two contour surfaces. The 95% HD is the distance that represents the largest surface-to-surface separation among the closest 95% of surface
points”, where a medical task comprising segmentation, “The OARs to be auto-segmented for each eligible disease site”, “surface” of an anatomy, “brainstem, globe, optic chiasm, optic nerve, parotid, submandibular, mandible, spinal cord, bladder, femoral head, and rectum . . . neck CTV, prostate, and seminal vesicles (SV) . . . lymph node levels Ib, II, III, IV, V, and the retropharyngeal and retrostyloid nodes”, is disclosed)
. . . deploying the machine trained model to a clinical environment . . . [to conduct a task] for a medical procedure . . . (Pg. 1, Abstract, “DC models were implemented at two cancer centers and used to generate OAR and CTVs for all patients undergoing RT for a central nervous system (CNS), head and neck (H&N), or prostate cancer”)
[and use of] geometric [metrics] . . . between the generated segmentation surface and one or more physical clinical markers associated with the medical procedure (Pg. 3, Col. 1, Para. 2-3, “unedited DCs were compared to the final treatment approved contours using Dice similarity coefficient (DSC) and 95% Hausdorff distance (HD) to provide an objective assessment of DC editing . . . DSC is defined as D(A,B) = 2|A ∩ B|/(|A| +|B|) and describes the relative overlap of segmentation volumes A and B. DSC values range from 0 for no overlap to 1 for complete overlap. HD is a bi-directional measure to quantify the distance between two contour surfaces. The 95% HD is the distance that represents the largest surface- to-surface separation among the closest 95% of surface points. For example, if a 95% HD is 2 mm, then 95% of contour A points are within 2 mm of contour B points. No comparisons were performed on RT plans containing”, where use of geometric metrics, “DSC” and “HD”, is between the generated segmentation surface, “segmentation volumes A and B” and “unedited DCs”, and “compared to the final treatment approved contours”, which are within the broadest reasonable interpretation of and one or more physical, “manual”, clinical markers, “contouring of organs at risk”, associated with the medical procedure, “radio therapy (RT) planning”, see Pg. 2, Col. 2, Para. 3, “Generated DCs underwent manual review and were edited as needed prior to being used for RT treatment planning”; Pg. 1, Col. 2, Para. 1, “Manual contouring of organs at risk (OAR) and clinical target volumes (CTV) is an essential task in radio therapy (RT) planning”, and Pg. 1, Abstract, “Unedited DCs were compared to the edited treatment approved contours using Dice similarity coefficient (DSC) and 95% Hausdorff distance (HD)”; see also Pg. 1, Abstract, “DC models were implemented at two cancer centers and used to generate OAR and CTVs for all patients undergoing RT for a central nervous system (CNS), head and neck (H&N), or prostate cancer”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the method of training data selection comprising: training a model, computing an application dependent evaluation metric correlated with a clinical outcome for each instance of the model, wherein computing the application dependent clinical evaluation metric comprises calculating a metric and using the metric to identify datasets with performance deviation of Bernhardt with the deployment of a trained machine learning model in a clinical setting to conduct a task for a medical procedure, wherein the medical task comprises segmentation of the surface of an anatomy and use of geometric metrics between the generated segmentation surface and one or more physical clinical markers associated with the medical procedure of Wong in order to apply the method disclosed by Bernhardt to a segmentation task in a clinical setting, where its benefits can be utilized to alleviate workload pressures, improve results (Wong, Pg. 2, Col. 1, Para. 1, “Auto-segmentation solutions are frequently explored to alleviate workload pressures [1], and deep learning based auto-segmentation is thought to provide improved results over atlas-based methods”), and reduce time costs (Wong, Pg. 2, Col. 1, Para. 2, “DCs were also noted to take substantially less time to produce compared to manual contours, although this did not take into account the amount of potential editing that may be done in clinical practice”), which will have significant importance when utilized in clinical practice by actual patients (compare Bernhardt, Pg. 1, Abstract, “employing experts to remove label noise by fully re-annotating large datasets is infeasible in resource-constrained settings, such as healthcare”; Bernhardt, Pg. 2, Col. 2, Para. 2, “This is imperative in safety-critical domains such as healthcare, as model robustness must be validated on clean labels”; and Bernhardt, Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy” with Wong, Pg. 1, Abstract, “DC models were implemented at two cancer centers and used to generate OAR and CTVs for all patients undergoing RT for a central nervous system (CNS), head and neck (H&N), or prostate cancer” ). Specifically, this modification could benefit model training for a wide variety of clinical tasks (Wong, Pg. 2, Col. 1, Para. 3, “we hope to increase interest in adopting machine learning auto-segmentation applications in other Radiation Oncology clinical practices”) by allowing resource efficient and accurate fine-tunning of models on actual client data, which will offer an improvement over generic models (compare Wong, Pg. 2, Col. 1, Para. 4, “These models were trained using publicly available data; no local institutional data was used”, where fine-tunning on client data is not performed, with Wong, Pg. 3, Col. 1, Para. 2, “objective comparisons do assist in the identification of consistent DC contouring errors that can then be the target of model training and improvement”, where fine-tunning on client data would be useful), using geometric metrics associated with increased efficiency and user satisfaction (Wong, Pg. 4, Col. 2, Para. 2, “We implemented previously validated DC models for CNS, H&N, and prostate RT planning and found that OAR DCs required minimal subjective editing and were associated with high user satisfaction” and Wong, Pg. 2, Col. 1, Para. 2, “DCs were also noted to take substantially less time to produce compared to manual contours, although this did not take into account the amount of potential editing that may be done in clinical practice”).
Additionally, He teaches . . . [identifying and processing one or more data sets for which performance of the data deviates] from a defined performance range . . . (Para. [0050], “the subject-matter . . . can optionally include that filtering the input data further includes: removing data in the consolidated data that correspond to performance values that fall outside a predetermined range of target performance values”, where one or more data sets, “input data”, are identified and processed, “removing data in the consolidated data”, when performance deviates from a defined performance range, “performance values that fall outside a predetermined range of target performance values”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the method for selection of training data sets, wherein the machine trained model and application dependent clinical evaluation metric are used to identify one or more data sets for which performance of the model deviates and wherein the machine learning model is retained in a manner where the one or more data sets are prioritized for use over data sets that do not sufficiently deviate of Bernhardt in view of Wong with the identifying and processing of one or more data sets for which performance of the data deviates from a defined performance range of He in order utilize a simplified and less computationally intensive approach to identifying data for annotation, which improves model accuracy by focusing specifically on data detrimental to model performance (compare Bernhardt, Pg. 2-3, Col. 2-1, Para. 3-1, “To this end, we propose to rank available samples by the following scoring function Φ: . . . [equation 2]. The first term . . . corresponds to the estimated noisiness . . . On the other hand . . . entropy term . . . penalises ambiguous cases in the ranking”, where the ranking requires a computationally intensive calculation of conflicting “term[s]”, which must be further analyzed to balance the interest in identifying “nois[y]” data that is detrimental to model performance, see Bernhardt, Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy”, against the interest of avoiding “ambiguous cases”; with He, Para. [0050], “the subject-matter . . . can optionally include . . . [identifying and performing operation on] a predetermined range of target performance values”, where noisy data detrimental to model performance can be directly identified using a simplified computational operation).
Regarding Claim 7, Bernhardt in view of Wong and He teach the method of claim 1, wherein during machine training of the model, the model is tested using a testing metric that is different than the application dependent clinical evaluation metric (Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model”, where, as discussed above, the “predicted posteriors” are the application dependent clinical evaluation metric, and “accuracy”, a different testing metric, is also computed, see Bernhardt, Pg. 6, Col. 2, Table 3, “We compare classification accuracy on true, noisy, and cleaned labels”; and where both metrics are used during training, see Bernhardt, Pg. 3, Col. 2, Table 1, “5. j ← arg maxi∈Iavail Φ(xi,l^i; θ) Rank (Eq. (2)), where the “predicted posteriors” are calculated at each training iteration, and Bernhardt, Pg. 5, Fig. 3, where “accuracy” is computed for different “Number[s] of collected relabels on the dataset”, where a batch is relabeled at each iteration of training, see Bernhardt, Pg. 3, Col. 2, Table 1, “13: θ ← UPDATE(θ, D) . Fine-tune model”).
Regarding Claim 8, Bernhardt in view of Wong and He teach the method of claim 1, wherein the medical imaging data is acquired using one of MRI, CT, X-ray, or Ultrasound (Bernhardt, Pg. 2, Col. 2, Para. 3, “a trained classification model”, where it is trained on medical imaging data using an X-ray, see Bernhardt, Pg. 9, Col. 1, Para. 7, “For further information on the acquisition and anonymisation of chest X-ray scans”, and where, in view of Wong, the task is segmentation and the medical image data also includes data acquired by CT, see Wong, Pg. 2, Col. 1-2, Para. 3-1, “By sharing our experience implementing machine learning auto-segmentation into the workflow, we hope to increase interest in adopting machine learning auto-segmentation applications in other Radiation Oncology clinical practices . . . Using these CT images, the auto-segmentation software prospectively generated DCs to be reviewed and edited on all patients undergoing RT treatment planning for CNS, H&N, and prostate malignancies”).
The reasons for obviousness are discussed above in regard to the rejection of Claim 1 and remain applicable here.
Regarding Claim 9, Bernhardt in view of Wong and He teach the method of claim 1, wherein the application dependent clinical evaluation metric (Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, where, as discussed above the “predicted posteriors” are the application dependent clinical evaluation metric)
further provides an assessment of how well an output of the machine trained model guided a clinician during the medical procedure subsequent to the performance of the task (Wong, Pg. 1, Abstract, “Purpose . . . we evaluate the performance of implemented DC models in the clinical radiotherapy (RT) planning workflow and report on user experience”, where, in view of Wong, the evaluation metric includes an assessment of the “models” in “clinical radiotherapy” medical procedures; Wong, Pg. 2, Col. 1-2, Para. 4-1, “the auto-segmentation software prospectively generated DCs to be reviewed and edited on all patients undergoing RT treatment”, where task outputs by the model are subsequently used to guide the clinician in a planning procedure, Wong, Pg. 2, Col. 2, Para. 3, “Generated DCs underwent manual review and were edited as needed prior to being used for RT treatment planning”, and then assessed by how well they performed, Wong, Pg. 1, Abstract, “Methods and materials: DC models were implemented at two cancer centers and used to generate OAR and CTVs for all patients undergoing RT for a central nervous system (CNS), head and neck (H&N), or prostate cancer. Radiation Therapists/Dosimetrists and Radiation Oncologists completed post-contouring surveys rating the degree of edits required for DCs (1 = minimal, 5 = significant) and overall DC satisfaction (1 = poor, 5 = high)”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the use of an evaluation metric that correlates with a clinical outcome to identify challenging machine learning datasets of Bernhardt in view of Wong and He with the use of an evaluation metric based on how well the output of the machine learning model guided a clinician in further view of Wong in order to determine difficult training data based on practical effectiveness instead of just metrics correlated with a clinical outcome (Wong, Pg. 5, Col. 1, Para. 2-3, “than any auto-segmentation time savings. We therefore relied on post-contouring survey feedback as a quantifiable indicator as to whether DCs impeded, rather than streamlined, existing workflow with the presumption that any unusable DCs would result in poor editing scores and overall satisfaction results”; Wong, Pg. 5, Col. 2, Para. 3, “These favourable survey results suggest that the OAR DCs were associated with a clinical workflow benefit”, where “favourable” evaluation metrics are an indication that the data labels do not need to be reviewed).
Regarding Claim 10, Bernhardt in view of Wong and He teach the method of claim 1, wherein identifying comprises identifying the one or more data sets that are the ten percent of data sets that deviate the most from the defined performance range (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where, as discussed above, the “iterate[ve]” “rank[ing]” and selection of a “batch of samples” based on “correctness and ambiguity” is within the broadest reasonable interpretation of identifying data sets, which in view of He, are the data that deviate from the defined performance range, see He, Para. [0050], “the subject-matter . . . can optionally include . . . [identifying and performing operation on] a predetermined range of target performance values”, where, if all data that deviates from “a predetermined range of target performance values” are identified, then the identified data must include the ten percent of data sets that deviate the most).
The reasons for obviousness were discussed in regard to the rejection of claim 1 and remain applicable here.
Regarding Claim 18, Bernhardt in view of Wong and He teach a method for data collection, the method comprising: performing a medical imaging procedure to generate medical imaging data (Bernhardt, Pg. 1, Abstract, “This work advocates for a data-driven approach to prioritising samples for re-annotation—which we term “active label cleaning” . . . the proposed active label cleaning enables correcting labels up to 4× more effectively than typical random selection”, where “data-driven” and “active” selection of training data sets is a method for identifying data for cleaning, which, in view of Wong, includes data collection, see Wong, Pg. 2, Col. 1, Para. 4, “Planning computed tomography (CT) images from both centers were captured using a GE Healthcare Optima CT580 series scanner with the following parameters depending on disease site: 120kVp, 100–700mAs, 1.25–2.5 mm slice thickness, and 0.683–1.270 mm in-plane pixel size. Using these CT images”, where “Planning computed tomography” is a medical imaging procedure that is performed to generate data of “CT images”);
processing the medical imaging data using a machine learned network (Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model, pθ(ˆy|x), parametrised by θ”; Bernhardt, Pg. 3, Col. 2, Table 1, “1: θ ← TRAINROBUSTMODEL(D)”, where the “trained . . .model” is a machine learned network, see Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”, which processes medical imaging data, see Bernhardt, Pg. 3, Col. 1, Para. 2, “we experiment with two imaging datasets, namely CIFAR10H and NoisyCXR . . . We also set up a new benchmark for label cleaning on medical images, NoisyCXR, comprising 26.6k chest radiographs and multiple labels from clinical datasets”, which in view of Wong, is the medical imaging data, see Wong, Pg. 1, Abstract, “DC models were implemented at two cancer centers and used to generate OAR and CTVs for all patients undergoing RT for a central nervous system (CNS), head and neck (H&N), or prostate cancer”);
computing an application dependent clinical evaluation metric for the processed medical imaging data based on a clinical outcome related to the medical imaging procedure (Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, where the “predicted posteriors” are an evaluation metric because they are used to evaluate the “label[s]” of the imaging data; see also Bernhardt, Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy”, where the “predicted posteriors” are correlated with the clinical outcome of result accuracy, and are therefore an application dependent clinical evaluation metric when used for evaluating “labels” from “radiology reports” for “medical imaging”, see Bernhardt, Pg. 1, Col. 1, Para. 2, “Diagnostic labels were extracted from the radiology reports via an error-prone automated process”; Bernhardt, Pg. 3, Col. 2, Table 1, “5. j ← arg maxi∈Iavail Φ(xi,l^i; θ) Rank (Eq. (2))” and Bernhardt, Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”, and, in view of Wong are based on a clinical outcome related to the medical imaging procedure, see Wong, Pg. 2, Col. 1, Para. 4, “Planning computed tomography (CT) images from both centers were captured using a GE Healthcare Optima CT580 series scanner with the following parameters depending on disease site: 120kVp, 100–700mAs, 1.25–2.5 mm slice thickness, and 0.683–1.270 mm in-plane pixel size. Using these CT images”, where “disease site” is a clinical outcome related to “Planning computed tomography”, a medical imaging procedure used to generate the medical imagining data “CT images”, which are the basis of the metric);
determining, based on the application dependent clinical evaluation metric, that the medical imaging data (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where the “iterate[ve]” “rank[ing]” and selection of a “batch of samples” based on “correctness and ambiguity (i.e. annotation difficulty)” is within the broadest reasonable interpretation of determining about one or more data sets, which are identified based on the “predicted posteriors”, which as discussed above are the application dependent clinical evaluation metric for each model iteration, see Bernhardt, Pg. 2-3, Col. 2-1, Para. 3-1, “To this end, we propose to rank available samples by the following scoring function Φ: . . . [equation 2]. The first term . . . corresponds to the estimated noisiness [and the second term] . . . penalises ambiguous cases in the ranking”)
comprises a data set that deviates from a defined performance range (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where, as discussed above, the “iterate[ve]” “rank[ing]” and selection of a “batch of samples” based on “correctness and ambiguity” is within the broadest reasonable interpretation of determining data comprises a data set that deviates, which in view of He, is deviation from the defined performance range, see He, Para. [0050], “the subject-matter . . . can optionally include . . . [identifying and performing operation on] a predetermined range of target performance values”),
wherein determining comprises sorting the medical imaging data in an order based on the application dependent clinical evaluation metric (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where, as discussed above, the “iterate[ve]” “rank[ing]” and selection of a “batch of samples” based on “correctness and ambiguity (i.e. annotation difficulty)” is the determining, which comprises sorting data in an order, “rank[ing]”, based on the application dependent clinical evaluation metric for each model iteration, “predicted posteriors”, see Bernhardt, Pg. 2-3, Col. 2-1, Para. 3-1, “To this end, we propose to rank available samples by the following scoring function Φ: . . . [equation 2]. The first term . . . corresponds to the estimated noisiness [and the second term] . . . penalises ambiguous cases in the ranking”; see also Bernhardt, Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”, where the data is “medical imaging” data)
and selecting a predefined top percentage of the sorted medical imaging data to define a challenging data subset (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where, as discussed above, the “iterate[ve]” “rank[ing]” and selection of a “batch of samples” based on “correctness and ambiguity (i.e. annotation difficulty)” is the determining, which comprises selecting the sorted data, “ranked”, as “a batch of samples” “prioritised” [to be] “relabelled”, which, over the course of “multiple iterations” defines a challenging data subset, “ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, which corresponds to some top percentage of “noisy labels is sorted to prioritise clearly mislabelled samples”, which is predefined by the “fixed relabelling budget”, see Bernhardt, Pg. 1, Col. 2, Fig. 1, “A dataset with noisy labels is sorted to prioritise clearly mislabelled samples, maximizing the number of corrected samples given a fixed relabelling budget”; or, alternatively, in view of He, is predefined by deviation from the predefined performance range, see He, Para. [0050], “the subject-matter . . . can optionally include . . . [identifying and performing operation on] a predetermined range of target performance values”; see also Bernhardt, Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”, where the data is “medical imaging” data); and
prioritizing the medical imaging data for retraining of the machine learned network (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where the data is prioritized in active label cleaning, which is a retraining approach, see Bernhardt, Pg. 3, Col. 2, Table 1, “13: θ ← UPDATE(θ, D) . Fine-tune model”, of a machine learned network, see Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”; see also Bernhardt, Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”, where the data is “medical imaging” data),
by adding the challenging data subset to an original training data set used to initially train the machine learned network to form a new training data set, and executing the retraining using the new training data set (Bernhardt, Pg. 3, Col. 2, Table 1, where, after cleaning at line “10”, the challenging data subset, which is the subset of “D . . . Dataset with noisy labels” that is collectively added to “j” while the “budget remains”, is added back to the original data set “D” to form a new training data set at line “11”, where “D” was used to initially train the machine learned network at line “1”, and then the retraining is executed using the new training data set at line “13”).
The reasons for obviousness are discussed above in regard to the rejection of Claim 1 and remain applicable here.
Claims 3, 5, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt in view of Wong, He, and CapeStart contributors (hereinafter CapeStart) (“Medical Image Annotation”).
Regarding Claim 3, Bernhardt in view of Wong and He teach the method of claim 1, wherein prioritizing the one or more data sets comprises wherein the one or more data sets are given priority [in label reannotation] . . . during retraining of the machine trained model (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where the identified data is prioritized in active label cleaning, which is a retraining approach, see Bernhardt, Pg. 3, Col. 2, Table 1, “13: θ ← UPDATE(θ, D) . Fine-tune model”, of a machine learned network, see Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”; see generally Bernhardt, Pg. 2, Col. 1, Para. 3, “Prioritising samples for labelling also underpins the paradigm of active learning, whose goal is to select unlabelled samples that would be most beneficial for training in order to improve the performance of a predictive model on a downstream task . . . The key difference here for the proposed approach is that our goal is not only to improve model performance but also to maximise the quality of labels given limited resources, which makes it valuable for both training and evaluation of predictive models. In more detail, we demonstrate how active learning and noiserobust learning (NRL) can play complementary roles in coping with label noise”).
Bernhardt in view of Wong and He do not explicitly disclose . . . in at least one of data transfer, data anonymization, or preprocessing . . . .
However, CapeStart teaches . . . in at least one of data transfer [to annotators], data anonymization, or preprocessing . . . (Pg. 1, “Expert Medical Image Annotation . . . CapeStart gets you to value faster through our expert team and intuitive, powerful SaaS platform”, where a “Saas platform” for “annotation” of user-supplied medical data by an “expert team” requires data transfer).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the method for retraining a machine trained model by prioritization of one or more data sets in label reannotation during training of Bernhardt in view of Wong and He with the transfer of medical data to annotators of CapeStart in order to receive services on transferred data from remote annotators, which may allow for utilization of a faster and more experienced team than locally available (CapeStart, Pg. 1, “Scale Healthcare Machine Learning Through Expert Medical Image Annotation. Machine learning can revolutionize healthcare delivery – but only when models and applications are well trained with expertly labeled and annotated data. CapeStart gets you to value faster through our expert team and intuitive, powerful SaaS platform”).
Regarding Claim 5, Bernhardt in view of Wong and He teach the method of claim 2, wherein prioritizing the one or more data sets comprises assigning the one or more data sets to annotators such that expert annotators are assigned data sets that deviate a most from the defined performance range to annotate (Bernhardt, Pg. 2, Col. 2, Para. 4, “at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritized sample is reviewed sequentially by different annotators”, where the “batches” “ranked” high priority due to “predicted label correctness”, which, in view of He, include data that deviate from the defined performance range, see He, Para. [0050], “the subject-matter . . . can optionally include . . . [identifying and performing operation on] a predetermined range of target performance values”, where, if all data that deviates from “a predetermined range of target performance values” are identified, then the identified data must include data sets that deviate a most; as well as all the data batches, in instances where sufficient resources allow the method to run to completion, are assigned to the annotators, who, in view of CapeStart, are experts, see CapeStart, Pg. 1, “Expert Medical Image Annotation . . . CapeStart gets you to value faster through our expert team and intuitive, powerful SaaS platform”; see generally Bernhardt, Pg. 1. Col. 2, Para. 2, “Due to the practical constraints on the total number of reannotations, samples often need to be prioritised to maximise the benefits of relabelling efforts (see Fig. 1), as the difficulty of reviewing labelling errors can vary across samples. Some cases are easy to assess and correct, others may be inherently ambiguous even for expert annotators (Fig. 2). For such difficult cases, several annotations (i.e. expert opinions) may be needed to form a ground-truth consensus”).
The reasons for obviousness are discussed above in regard to the rejection of Claim 1, for the combination with He, and in regard to the rejection of Claim 3, for the combination with CapeStart, and remain applicable here.
Regarding Claim 19, Bernhardt in view of Wong, He, and CapeStart teach the method of claim 18, wherein prioritizing comprises at least one of data transfer, data anonymization, or preprocessing of the medical imaging data during retraining of the machine learned network (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where the identified data is prioritized in active label cleaning, which is a retraining approach, see Bernhardt, Pg. 3, Col. 2, Table 1, “13: θ ← UPDATE(θ, D) . Fine-tune model”, of a machine learned network, see Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”; see generally Bernhardt, Pg. 2, Col. 1, Para. 3, “Prioritising samples for labelling also underpins the paradigm of active learning, whose goal is to select unlabelled samples that would be most beneficial for training in order to improve the performance of a predictive model on a downstream task . . . The key difference here for the proposed approach is that our goal is not only to improve model performance but also to maximise the quality of labels given limited resources, which makes it valuable for both training and evaluation of predictive models. In more detail, we demonstrate how active learning and noiserobust learning (NRL) can play complementary roles in coping with label noise”, where, in view of CapeStart, the annotation comprises transfer to remote experts, see CapeStart, Pg. 1, “Expert Medical Image Annotation . . . CapeStart gets you to value faster through our expert team and intuitive, powerful SaaS platform”).
The reasons for obviousness are discussed above in regard to the rejection of Claim 3 and remain applicable here.
Regarding Claim 20, Bernhardt in view of Wong, He, and CapeStart teach the method of claim 18, wherein prioritizing the medical imaging data comprises assigning the medical imaging data to annotators such that an expert annotator gets the medical imaging data to annotate (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where, in view of CapeStart, the annotation comprises transfer to experts, see CapeStart, Pg. 1, “Expert Medical Image Annotation . . . CapeStart gets you to value faster through our expert team and intuitive, powerful SaaS platform”).
The reasons for obviousness are discussed above in regard to the rejection of Claim 3 and remain applicable here.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt in view of Wong, He, Northcutt et al. (hereinafter Northcutt) (“Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks”) and CPlusPlus.com contributors (hereinafter CPlusPlus.com) (“std::queue::push”).
Regarding Claim 4, Bernhardt in view of Wong and He teach the method of claim 1, wherein identifying the one or more data sets comprises automatically assigning a priority . . . to the one or more data set (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where, as discussed above, the “iterate[ve]” “rank[ing]” and selection of a “batch of samples” based on “correctness” is within the broadest reasonable interpretation of identifying data sets that are challenging for the trained model, where each “batch of samples” dataset are assigned a different degree of priority based on “correctness”, see generally Bernhardt, Pg. 1, Fig. 1, which happens automatically when the method depicted by the pseudocode algorithm is executed, see Bernhardt, Pg. 3, Col. 2, Table 1)
and prioritizing the one or more data sets comprises pushing the one or more data sets to an annotation [environment] . . . according to their priority . . . (Bernhardt, Pg. 2, Col. 2, Para. 4, “at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritized sample is reviewed sequentially”, where “reviewing” the “batch of samples” “sequentially” requires an action by a computer program to display the data to an annotation environment, which is within the broadest reasonable interpretation of pushing).
Bernhardt in view of Wong and He do not explicitly disclose . . . score . . . queue . . . score . . . .
However, Northcutt teaches . . . [assigning a priority] score [to the data sets] . . . score (Northcutt, Pg. 1, Abstract, “We identify label errors in the test sets of 10 of the most commonly-used computer vision, natural language, and audio datasets, and subsequently study the potential for these label errors to affect benchmark results”; Northcutt, Pg. 4, table 1, “We observe that error rates vary across datasets, from 0.15% (MNIST) to 10.12% (QuickDraw)”; where, in the context of the method of Bernhardt, “% error”, which is the rate “label errors” in a “dataset”, expressed as a percent by multiplying by 100, are within the broadest reasonable interpretation of priority scores because the method is premised on prioritizing the greatest return in corrected labels for an investment of resources, see Bernhardt, Pg. 2, Col. 1, Para. 4, “In this work, we introduce a sequential label cleaning procedure that maximises the number of corrected samples under a total resource budget”; Bernhadt, Pg. 1, Col. 2, Fig. 1).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the identifying of data sets for annotation, where identifying data sets automatically assigns a priority to the dataset of Bernhardt in view of Wong and He with the dataset-level determining of a priority score of Northcutt in order to make cleaning determinations at the dataset level (Northcutt, Pg. 4, table 1, “We observe that error rates vary across datasets, from 0.15% (MNIST) to 10.12% (QuickDraw); unsurprisingly, simpler datasets, datasets with more carefully designed labeling methodologies, and datasets with more careful human curation generally had less error than datasets that used more automated data collection procedures”, where knowledge of a dataset as a whole can be used to generalize details about the associated data samples), which will provide additional information that may be useful for maximizing the number of corrections for a total resource budget (Bernhardt, Pg. 2, Col. 1, Para. 4, “In this work, we introduce a sequential label cleaning procedure that maximises the number of corrected samples under a total resource budget”).
Additionally, CPlusPlus.com teaches . . . [pushing data to] queue [to prioritize performance of a task] . . . (Pg. 1, Para. 1, “std::queue::push . . . Inserts a new element at the end of the queue, after its current last element. The content of this new element is initialized to val”; Pg. 2, Fig. “Example”, “while (!myqueue.empty()){ std::cout << ' ' << myqueue.front(); myqueue.pop(); }”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the prioritizing the data sets by pushing of the data sets to an annotation environment based on their priority scores of Bernhardt in view of Wong, He, and Northcutt with the pushing data to queue to prioritize performance of a task of CplusPlus.com in order to use a data structure with well-documented functionality and low computational complexity for push and pop functionality to ensure high priority data sets are annotated before lower priority data sets (CPlusPlus.com, Pg. 2, Sect. “Complexity”, “one call to push_back on the underlying container”, where insertion and removal of elements is less complex than alternative data structures, such as sets; see generally, CPlusPlus.com, Pg. 1-2, where the functionality of the queue data structure is well-documented).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt in view of Wong, He, and Chen et al. (hereinafter Chen) (“FOCUS: Dealing with Label Quality Disparity in Federated Learning”).
Regarding Claim 6, Bernhardt in view of Wong and He teach the method of claim 1, wherein prioritizing comprises identifying . . . the data sets that deviate the most from the defined performance range (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where each “batch of samples”, must be identified to be “ranked”, and where in view of He, are the data that deviate from the defined performance range, see He, Para. [0050], “the subject-matter . . . can optionally include . . . [identifying and performing operation on] a predetermined range of target performance values”, where, if all data that deviates from “a predetermined range of target performance values” are identified, then the identified data must include the ten percent of data sets that deviate the most).
The reasons for obviousness are discussed above in regard to the rejection of Claim 1 and remain applicable here.
Bernhardt in view of Wong and He do not explicitly disclose . . . sites in a federated learning setup that include . . . .
However, Chen teaches . . . [identifying] sites in a federated learning setup that include [the most challenging data] (Pg. 1, Col.1, Abstract, “Ubiquitous systems with End-Edge-Cloud architecture are increasingly being used in healthcare applications. Federated Learning (FL) is highly
useful for such applications . . . FOCUS has been experimentally evaluated on both synthetic data and real-world data. The results show that it effectively identifies clients with noisy labels”, where “clients” are within the broadest reasonable interpretation of sites).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the prioritizing of data sets during model training, comprising identifying the that deviate the most from the define performance range of Bernhardt in view of Wong and He with the identifying of sites, in a federated learning healthcare setup, that include the most challenging data of Chen in order to maximize the number of corrected labels for a resource amount (Bernhardt, Pg. 2, Col. 1, Para. 4, “In this work, we introduce a sequential label cleaning procedure that maximises the number of corrected samples under a total resource budget”; Bernhadt, Pg. 1, Col. 2, Fig. 1), where label noise may be significantly correlated with originating device site (Chen, Pg. 1, Col. 2, Para. 1-3, “one key challenge that remains open and hinders wide spread adoption of FL in ubiquitous systems, especially in the healthcare domain, is label quality disparity . . . we propose the Federated Opportunistic Computing for Ubiquitous System (FOCUS) approach to address this challenging problem. It is designed to identify clients with noisy labels”).
Claims 11-13 are rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt in view of Wong and Northcutt.
Regarding Claim 11, Bernhardt in view of Wong and Northcutt teach a system for data selection for training a deep learning network, comprising (Bernhardt, Pg. 1, Abstract, “This work advocates for a data-driven approach to prioritising samples for re-annotation—which we term “active label cleaning” . . . the proposed active label cleaning enables correcting labels up to 4× more effectively than typical random selection”, where “data-driven” and “active” selection of training data sets for cleaning is used for training a deep learning network, Bernhardt, Pg. 3, Col. 2, Table 1, “1: θ ← TRAINROBUSTMODEL(D)”; see also Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”, and implemented as a system, see generally Bernhardt, Pg. 3, Col. 2, Table 1, where the pseudocode algorithm could not be implemented without a system with a processor):
a datastore configured to store a plurality of data sets (Bernhardt, Pg. 3, Col. 1, Para. 2, “To analyse label noise scenarios, we experiment with two imaging datasets, namely CIFAR10H and NoisyCXR, containing multiple annotations per data point, which helps us to model the true label distributions and associated labelling cost”, where the system must have a datastore to use the data as input, see Bernhardt, Pg. 3, Col. 2, Table 1, “Input: D = {(xi, l^i)}Ni=1: Dataset with noisy labels),
wherein each data set of the plurality of data sets is assigned an application dependent clinical evaluation metric that correlates with a clinical outcome associated with a respective data set (Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, where the “predicted posteriors” are an evaluation metric because they are used to evaluate the “label[s]” see also Bernhardt, Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy”, where the “predicted posteriors” are correlated with the clinical outcome of result accuracy, and are therefore an application dependent clinical evaluation metric when used for evaluating “labels” from “radiology reports” for “medical imaging”, see Bernhardt, Pg. 1, Col. 1, Para. 2, “Diagnostic labels were extracted from the radiology reports via an error-prone automated process”; Bernhardt, Pg. 3, Col. 2, Table 1, “5. j ← arg maxi∈Iavail Φ(xi,l^i; θ) Rank (Eq. (2))” and Bernhardt, Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”; “predicted posteriors” are correlated with the clinical outcome of result accuracy for the respective dataset, see Bernhardt, Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy” and see generally Bernhardt, Pg. 1, Col. 1, Para. 2, “Diagnostic labels were extracted from the radiology reports via an error-prone automated process”, and are calculated and assigned for each sample, see Bernhardt, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially, and, in view of Northcutt, each set, see Northcutt, Pg. 1, Abstract, “We identify label errors in the test sets of 10 of the most commonly-used computer vision, natural language, and audio datasets, and subsequently study the potential for these label errors to affect benchmark results”; Northcutt, Pg. 4, table 1, “We observe that error rates vary across datasets, from 0.15% (MNIST) to 10.12% (QuickDraw)”);
the deep learning network configured to perform a task relating to medical imaging, wherein the task comprises (Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model, pθ(ˆy|x), parametrised by θ”; Bernhardt, Pg. 3, Col. 2, Table 1, “1: θ ← TRAINROBUSTMODEL(D)”, where the task is “classification”; Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”, where the model is a deep learning network, for a “medical imaging” task, see Bernhardt, Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”)
generating a segmentation surface of an anatomy (Wong, Pg. 2, Col. 2, Para. 2, “The OARs to be auto-segmented for each eligible disease site were selected by each center; the OARs available to be selected included brainstem, globe, optic chiasm, optic nerve, parotid, submandibular, mandible, spinal cord, bladder, femoral head, and rectum. The CTV contours available were neck CTV, prostate, and seminal vesicles (SV). DCs for neck CTV included lymph node levels Ib, II, III, IV, V, and the retropharyngeal and retrostyloid nodes” and Wong, Pg. 3, Col. 1, Para. 3, “DSC is defined as D(A,B) = 2|A ∩ B|/(|A| +|B|) and describes the relative overlap of segmentation volumes A and B. DSC values range from 0 for no overlap to 1 for complete overlap. HD is a bi-directional measure to quantify the distance between two contour surfaces. The 95% HD is the distance that represents the largest surface-to-surface separation among the closest 95% of surface points”, where a medical task comprising segmentation, “The OARs to be auto-segmented for each eligible disease site”, “surface” of an anatomy, “brainstem, globe, optic chiasm, optic nerve, parotid, submandibular, mandible, spinal cord, bladder, femoral head, and rectum . . . neck CTV, prostate, and seminal vesicles (SV) . . . lymph node levels Ib, II, III, IV, V, and the retropharyngeal and retrostyloid nodes”, is disclosed),
and wherein the application dependent clinical evaluation metric comprises (Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, where the “predicted posteriors” are an evaluation metric because they are used to evaluate the “label[s]” see also Bernhardt, Pg. 4, Col. 1, Para. 3, “Cleaning noisy training labels improves predictive accuracy”, where the “predicted posteriors” are correlated with the clinical outcome of result accuracy, and are therefore an application dependent clinical evaluation metric when used for evaluating “labels” from “radiology reports” for “medical imaging”, see Bernhardt, Pg. 1, Col. 1, Para. 2, “Diagnostic labels were extracted from the radiology reports via an error-prone automated process”; Bernhardt, Pg. 3, Col. 2, Table 1, “5. j ← arg maxi∈Iavail Φ(xi,l^i; θ) Rank (Eq. (2))” and Bernhardt, Pg. 1, Abstract, “Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection”)
a geometric metric calculated between the generated segmentation surface and one or more physical clinical markers associated with a medical procedure (Wong, Pg. 3, Col. 1, Para. 2-3, “unedited DCs were compared to the final treatment approved contours using Dice similarity coefficient (DSC) and 95% Hausdorff distance (HD) to provide an objective assessment of DC editing . . . DSC is defined as D(A,B) = 2|A ∩ B|/(|A| +|B|) and describes the relative overlap of segmentation volumes A and B. DSC values range from 0 for no overlap to 1 for complete overlap. HD is a bi-directional measure to quantify the distance between two contour surfaces. The 95% HD is the distance that represents the largest surface- to-surface separation among the closest 95% of surface points. For example, if a 95% HD is 2 mm, then 95% of contour A points are within 2 mm of contour B points. No comparisons were performed on RT plans containing”, where use of geometric metrics, “DSC” and “HD”, is between the generated segmentation surface, “segmentation volumes A and B” and “unedited DCs”, and “compared to the final treatment approved contours”, which are within the broadest reasonable interpretation of and one or more physical, “manual”, clinical markers, “contouring of organs at risk”, associated with the medical procedure, “radio therapy (RT) planning”, see Wong, Pg. 2, Col. 2, Para. 3, “Generated DCs underwent manual review and were edited as needed prior to being used for RT treatment planning”; Wong, Pg. 1, Col. 2, Para. 1, “Manual contouring of organs at risk (OAR) and clinical target volumes (CTV) is an essential task in radio therapy (RT) planning”, and Wong, Pg. 1, Abstract, “Unedited DCs were compared to the edited treatment approved contours using Dice similarity coefficient (DSC) and 95% Hausdorff distance (HD)”; see also Wong, Pg. 1, Abstract, “DC models were implemented at two cancer centers and used to generate OAR and CTVs for all patients undergoing RT for a central nervous system (CNS), head and neck (H&N), or prostate cancer”);
and a processor configured to train the deep learning network using the plurality of data sets (Bernhardt, Pg. 3, Col. 2, Table 1, “Input: D = {(xi, l^i)}Ni=1: Dataset with noisy labels . . . 13: θ ← UPDATE(θ, D) . Fine-tune model”, where the data sets are used to iteratively train the model, which is a deep learning network, Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”; see generally Bernhardt, Pg. 4, Col. 2, Para. 3, “The self-supervised models have been trained with BYOL6 using a ResNet-50 encoder. The final model is trained using the NIH dataset9 (80% training, 20% validation), using an effective batch size of 4800 (batches of 600 pairs of images of size 224 × 224 on 8 GPUs) for 1000 epochs. The momentum parameter τ used in BYOL teacher encoder was set to 0.99”),
the processor configured to prioritize one or more data sets of the plurality of data sets for use in the training based on the application dependent clinical evaluation metric (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where the “predicted label correctness and ambiguity”, which is the “predicted posteriors” application dependent clinical evaluation metric, see Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, is used to prioritize data sets of the plurality of data sets in training by “cleaning” during an earlier “iteration” of “fine-tun[ning]”, Bernhardt, Pg. 3, Col. 2, Table 1, “13: θ ← UPDATE(θ, D) . Fine-tune model”, which, as discussed above, requires a processor).
The reasons for obviousness are discussed above in regard to the rejection of Claim 1 and the rejection of Claim 4 and remain applicable here.
Regarding Claim 12, Bernhardt in view of Wong and Northcutt teach the system of claim 11, wherein during training of the deep learning network, the deep learning network is tested using a testing metric that is different than the evaluation metric (Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”; Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model”, where, as discussed above, the “predicted posteriors” are the application dependent clinical evaluation metric, and “accuracy”, a different testing metric, is also computed, see Bernhardt, Pg. 6, Col. 2, Table 3, “We compare classification accuracy on true, noisy, and cleaned labels”; and where both metrics are used during training, see Bernhardt, Pg. 3, Col. 2, Table 1, “5. j ← arg maxi∈Iavail Φ(xi,l^i; θ) Rank (Eq. (2)), where the “predicted posteriors” are calculated at each training iteration, and Bernhardt, Pg. 5, Fig. 3, where “accuracy” is computed for different “Number[s] of collected relabels on the dataset”, where a batch is relabeled at each iteration of training, see Bernhardt, Pg. 3, Col. 2, Table 1, “13: θ ← UPDATE(θ, D) . Fine-tune model”).
Regarding Claim 13, Bernhardt in view of Wong and Northcutt teach the system of claim 11, wherein the application dependent clinical evaluation metric (Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, where, as discussed above the “predicted posteriors” are an application dependent clinical evaluation metric)
further provides an assessment of how well an output of the deep learning network guided a clinician during a medical procedure (Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”; Wong, Pg. 1, Abstract, “Purpose . . . we evaluate the performance of implemented DC models in the clinical radiotherapy (RT) planning workflow and report on user experience”, where, in view of Wong, the evaluation metric includes an assessment of the “models” in “clinical radiotherapy” medical procedures; Wong, Pg. 2, Col. 1-2, Para. 4-1, “the auto-segmentation software prospectively generated DCs to be reviewed and edited on all patients undergoing RT treatment”, where task outputs by the model are subsequently used to guide the clinician in a planning procedure, Wong, Pg. 2, Col. 2, Para. 3, “Generated DCs underwent manual review and were edited as needed prior to being used for RT treatment planning”, and then assessed by how well they performed, Wong, Pg. 1, Abstract, “Methods and materials: DC models were implemented at two cancer centers and used to generate OAR and CTVs for all patients undergoing RT for a central nervous system (CNS), head and neck (H&N), or prostate cancer. Radiation Therapists/Dosimetrists and Radiation Oncologists completed post-contouring surveys rating the degree of edits required for DCs (1 = minimal, 5 = significant) and overall DC satisfaction (1 = poor, 5 = high)”).
The reasons for obviousness are discussed above in regard to the rejection of Claim 9 and remain applicable here.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt in view of Wong and Northcutt and CapeStart.
Regarding Claim 14, Bernhardt in view of Wong and Northcutt teach the system of claim 11, wherein . . . training of the deep learning network (Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”).
The remaining elements are substantially the same as the limitations of Claim 19, therefore it is rejected under the same rationale.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt in view of Wong, Northcutt, CapeStart, and He.
Regarding Claim 15, Bernhardt in view of Wong and Northcutt teach the system of claim 11, wherein the processor is configured to prioritize the data sets . . . (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where the “predicted label correctness and ambiguity” is used to prioritize data sets in training by “cleaning” during an earlier “iteration” of “fine-tun[ning]”, Bernhardt, Pg. 3, Col. 2, Table 1, “13: θ ← UPDATE(θ, D) . Fine-tune model”, which, as discussed above, requires a processor).
The remaining elements are substantially the same as the limitations of Claim 5, therefore it is rejected under the same rationale.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt in view Wong, Northcutt, and CPlusPlus.com.
Regarding Claim 16, Bernhardt in view of Wong, Northcutt, and CPlusPlus.com teach the system of claim 11, wherein the processor is configured to assign the plurality of data sets a priority score based on the application dependent clinical evaluation metric and prioritize the one or more data sets (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, where the “predicted label correctness and ambiguity”, which, as discussed above, is the “predicted posteriors” application dependent clinical evaluation metric, see Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, is used to prioritize data sets in training by “cleaning” during an earlier “iteration” of “fine-tun[ning]”, Bernhardt, Pg. 3, Col. 2, Table 1, “13: θ ← UPDATE(θ, D) . Fine-tune model”, which, as discussed above, requires a processor; and where, in view of Northcutt, the evaluation metric, “error rates” is assigned at the level of “datasets” and forms the basis for the “% error” priority score, see Northcutt, Pg. 1, Abstract, “We identify label errors in the test sets of 10 of the most commonly-used computer vision, natural language, and audio datasets, and subsequently study the potential for these label errors to affect benchmark results”; Northcutt, Pg. 4, table 1, “We observe that error rates vary across datasets, from 0.15% (MNIST) to 10.12% (QuickDraw)”; where, in the context of the method of Bernhardt, “% error”, which is the rate “label errors” in a “dataset”, expressed as a percent by multiplying by 100, are within the broadest reasonable interpretation of priority scores because the method is premised on prioritizing the greatest return in corrected labels for an investment of resources, see Bernhardt, Pg. 2, Col. 1, Para. 4, “In this work, we introduce a sequential label cleaning procedure that maximises the number of corrected samples under a total resource budget”; Bernhadt, Pg. 1, Col. 2, Fig. 1)
by pushing the one or more data sets to an annotation queue according to their priority score (Bernhardt, Pg. 2, Col. 2, Para. 2, “Label cleaning is performed over multiple iterations and at each iteration either a single or a batch of samples are relabelled. Within an iteration, samples are first ranked according to predicted label correctness and ambiguity (i.e. annotation difficulty). Then, each prioritised sample is reviewed sequentially”, which, in view of CPlusPlus.com, is effectuated by pushing the data sets to an annotation queue according to their priority score, see CPlusPlus.com, Pg. 1, Para. 1, “std::queue::push . . . Inserts a new element at the end of the queue, after its current last element. The content of this new element is initialized to val”; Pg. 2, Fig. “Example”, “while (!myqueue.empty()){ std::cout << ' ' << myqueue.front(); myqueue.pop(); }”).
The reasons for obviousness are discussed above in regard to the rejection of Claim 4 and remain applicable here.
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Bernhardt in view of Wong, Northcutt, He, Kohler et al. (hereinafter Kohler) (“Uncertainty Based Detection and Relabeling of Noisy Image Labels”), and Smit et al. (hereinafter Smit) (“CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT”).
Regarding Claim 17, Bernhardt in view of Wong, Northcutt, and He teach the system of claim 11, wherein the processor is configured to . . . [perform iterative label correcting] of data sets that deviate the most from a predefined performance range based on the application dependent clinical evaluation metric for subsequent training of the deep learning network (Bernhardt, Pg. 3, Col. 2, Table 1, “Input: D = {(xi, l^i)}Ni=1: Dataset with noisy labels . . . 13: θ ← UPDATE(θ, D) . Fine-tune model”, where the data sets, see Northcutt, Pg. 4, table 1, “We observe that error rates vary across datasets, from 0.15% (MNIST) to 10.12% (QuickDraw)”, are updated based on the “predicted posteriors” application dependent clinical evaluation metric, see Bernhardt, Pg. 2, Col. 2, Para. 2, “The proposed framework (see Table 1) determines relabelling priority based on predicted posteriors from a trained classification model . . according to predicted label correctness and ambiguity (i.e. annotation difficulty)”, where in view of He, are the data that deviate from the defined performance range, see He, Para. [0050], “the subject-matter . . . can optionally include . . . [identifying and performing operation on] a predetermined range of target performance values”, where, if all data that deviates from “a predetermined range of target performance values” are identified, then the identified data must include the data sets that deviate the most, which are used to iteratively train the model that is a deep learning network, Bernhardt, Pg. 8, Col. 1, Para. 4, “we first train deep neural networks”; Bernhardt, Pg. 3, Col. 2, Table 1, where the pseudocode algorithm could not be implemented without a system with a processor; see generally Bernhardt, Pg. 4, Col. 2, Para. 3, “The self-supervised models have been trained with BYOL6 using a ResNet-50 encoder. The final model is trained using the NIH dataset9 (80% training, 20% validation), using an effective batch size of 4800 (batches of 600 pairs of images of size 224 × 224 on 8 GPUs) for 1000 epochs. The momentum parameter τ used in BYOL teacher encoder was set to 0.99”).
The reasons for obviousness are discussed above in regard to the rejection of Claim 1, for the combination with He, and in regard to the rejection of Claim 4, for the combination with Northcutt, and remain applicable here.
Bernhardt in view of Wong, Northcutt, and He do not explicitly disclose . . . only use the ten percent . . . (where the first iteration of the pseudocode algorithm uses an unspecified percentage of the data sets and all of the data, instead of only that percentage, are used for the subsequent training at each iteration).
However, Kohler teaches . . . use the ten percent . . . [most challenging for each label correction iteration] (Pg. 1, Abstract, “Data with noisy labels can therefore be cleaned in an iterative process”; Pg. Pg. 3, Col. 1, Para. 4, “The noisy images are identified by taking the top 10% of most uncertain images”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the use of a processor to perform iterative label correction of data sets that deviate the most from a predefined performance range, based on an application dependent clinical evaluation metric, for subsequent training of a deep learned network of Bernhardt in view of Wong, Northcutt, and He with the use of the ten percent most challenging data sets for each label correction iteration of Kohler in order to conduct priority-based relabeling using an easy and effective implementation approach (Kohler, Pg. 1, Col. 1, Abstract, “Our proposed method can be easily implemented, and shows promising performance on the task of noisy label detection on CIFAR-10 and CIFAR-100”; see generally Bernhardt, Pg. 1, Col. 2, Para. 2, “Due to the practical constraints on the total number of reannotations, samples often need to be prioritised to maximise the benefits of relabelling efforts”).
Additionally, Smit teaches . . . only . . . [using human-generated annotations for subsequent training] (Pg. 1, Col. 2, Fig. 1, “We introduce a method for radiology report labeling, in which a biomedically pretrained BERT model is first trained on annotations of a rule-based labeler, and then fine-tuned on a small set of expert annotations”, where “fine-tun[ing]” is subsequent training).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the use a processor to perform iterative label correction, by human annotators on the ten percent of data sets that deviate the most from a predefined performance range, based on the application dependent clinical evaluation metric, for subsequent training of a deep learned network of Bernhardt in view of Wong, Northcutt, He, and Kohler with the use of only human-generated annotations for subsequent training of Smit in order to utilize only labels created by trusted and qualified annotators for model fine-tuning (compare Bernhardt, Pg. 1, Col. 1, Para. 1, “Labelling errors can occur due to automated label extraction . . . At training time, incorrect labels hamper the generalisation of predictive models, as labelling errors may be memorised by the model resulting in undesired biases” with Smit, Pg. 8, Col. 1, Para. 3, “collection of expert labels can produce a small set of high quality labels”), which will include only the ten percent most challenging data sets during the subsequent training in the first iteration of the pseudocode algorithm.
Response to Arguments
Applicant's arguments filed on June 30th, 2026, have been fully considered. Each argument is addressed in detail below.
I. Applicant argues the objections to the claims should be withdrawn (Applicant’s Remarks, 06/30/2026, Pg. 7, Section “II. Claim Objections”).
Applicant’s amendments have overcome each and every objection to claims as
previously set forth in the June 30th, 2026 Office action. As a result, these objections have
been withdrawn. However, as discussed in detail above, Applicant’s amendments introduce new informalities, which necessitate new grounds for objection.
II. Applicant argues the rejections to the claims, under 35 USC § 101, should be withdrawn
(Applicant’s Remarks, 06/30/2026, Pg. 7-12, Section “III. Claim Rejections under 35 U.S.C. § 101”).
1) First, Applicant argues the amended claims do not recite mental processes (Step 2A, Prong 1). Specifically, Applicant asserts “Claims 1 and 11 have been amended to recite specific, technological operations that remove the claims from the arena of abstract ideas. Specifically, the claims now require that the medical imaging task comprises generating a segmentation surface of an anatomy, and that the computing of the application dependent clinical evaluation metric comprises calculating a geometric metric between the generated segmentation surface and one or more physical clinical markers associated with the medical procedure” (Applicant’s Remarks, Pg. 7, Para. 6) (internal quotation marks omitted).
Additionally, Applicant asserts “Claim 18 has been amended to recite a specific, computational workflow for restructuring the data architecture used to train a machine learned network. Specifically, the claim now requires that determining the deviating data set comprises sorting the medical imaging data in an order based on the application dependent clinical evaluation metric and selecting a predefined top percentage of the sorted medical imaging data to define a challenging data subset. Furthermore, the claim now requires prioritizing the data by adding the challenging data subset to an original training data set used to initially train the machine learned network to form a new training data set, and executing the retraining using the new training data set” (Applicant’s Remarks, Pg. 9, Para. 4) (internal quotation marks omitted).
As a result, Applicant argues the independent claims recite “complex computational processes”, are “firmly rooted in computer-implemented image processing and geometric calculation”, “define specific algorithmic data transformations”, recite “a specific, programmatic restructuring of a machine learning data pipeline”, and contain “automated computational operations” like “[g]enerating distinct data structures, merging them with baseline datasets, and executing algorithmic retraining on the newly formed data architecture”, such that the claims cannot “practically be performed in the human mind” (Applicant’s Remarks, Pg. 7-8, Para. 7-2 and Pg. 9-10, Para. 5-1).
However, geometric calculation, complex computational processing, programmatic restructuring, computational operations, and specific algorithmic data transformations are well-within the definition of evaluations, judgments, and opinions that can be performed by a human using pen and paper (see MPEP 2106.04(a)(2)(III), “The courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea . . . the "mental processes" abstract idea grouping is defined as concepts performed in the human mind, and examples of mental processes include observations, evaluations, judgments, and opinions”).
Additionally, while Applicant is correct that the claim recites computer-implemented image processing and automated computational operations relating to machine learning pipelines, a claim that requires a computer may still recite a mental process (see MPEP 2106.04(a)), “A Claim That Requires a Computer May Still Recite a Mental Process . . . examiners should review the specification to determine if the claimed invention is described as a concept that is performed in the human mind and applicant is merely claiming that concept performed 1) on a generic computer, or 2) in a computer environment, or 3) is merely using a computer as a tool to perform the concept. In these situations, the claim is considered to recite a mental process”). Here, as discussed in detail above, the additional elements relating to computer-implemented image processing and automated computational operations relating to machine learning pipelines amount to merely claiming that concept performed on a generic computer, in a computer environment, and merely using a computer as a tool to perform the concept (id.)
As a result, the arguments are not persuasive.
2) Second, Applicant argues the amended claims, even if abstract, are integrated into a practical application (Step 2A, Prong 2). Specifically, Applicant argues amended claims 1 and 11 “provide a specific, technical solution to . . . [a] problem by establishing a closed-loop retraining architecture. Instead of relying on generic statistical label noise or accuracy percentages, the claimed system executes a specific physical evaluation: it calculates a geometric metric between the model's generated segmentation surface and actual physical clinical markers”, which is described in the specification and which “solves a recognized technological problem in machine learning: models trained on benchmark data often fail in clinical environments due to edge-case variations not present in the original training data” by “applying a physical, geometric evaluation metric” to “physically alter” model training, such that it “identify[s] data sets where the model's generated surface geometrically deviates from physical clinical markers”, “prioritize[es] those specific data sets for retraining”, and “selectively route[s] clinically deficient data back into the training pipeline” (Applicant’s Remarks, Pg. 8-9, Para. 3-3)
Additionally, Applicant argues amended claim 18 “defin[es] a precise computational mechanism to optimize the retraining pipeline” because it “dictates exactly how the computer must filter the data (by sorting and extracting a predefined top percentage to form a challenging subset) and exactly how the computer must execute the retraining (by appending only that specific subset to the original training data set to form a new, enriched data structure)”, which “is a specific set of rules that alters the data architecture and the retraining behavior of the machine learning system” that “addresses a specific technological problem in artificial intelligence: retraining models on entirely new datasets is computationally expensive, while retraining on redundant data fails to improve the model” through a “particular solution” that improves the “efficiency and efficacy of the machine learning training process itself” (Applicant’s Remarks, Pg. 10-11, Para. 2-2).
However, the specification sets forth the alleged improvements to machine learning model training, through identification of clinical data using an evaluation metric and model retraining, in a conclusory manner (MPEP 2106.04(d)(1), “Conversely, if the specification explicitly sets forth an improvement but in a conclusory manner (i.e., a bare assertion of an improvement without the detail necessary to be apparent to a person of ordinary skill in the art), the examiner should not determine the claim improves technology”; see also Applicant’s Spec. Para. [0056] – [0058]).
Additionally, claims 1 and 11 recite limitations, such as calculating a geometric evaluation metric, identifying datasets, prioritizing datasets, and routing datasets, that have broad applicability across many fields of endeavor (see MPEP 2106.05(f), “A claim having broad applicability across many fields of endeavor may not provide meaningful limitations that integrate a judicial exception into a practical application or amount to significantly more”), such as to fail to reflect the alleged improvements (see MPEP 2106.04(d)(1)), “the claim must be evaluated to ensure that the claim itself reflects the disclosed improvement”).
Similarly, claim 18 also recites limitations, such as selecting sorted data based on a percentage threshold and generating data by adding a subset to a dataset, that have broad applicability across many fields of endeavor (see MPEP 2106.05(f)), such as to fail to reflect the alleged improvements (see MPEP 2106.04(d)(1))).
As a result, the arguments are not persuasive.
3) Third, Applicant argues the claims are subject matter eligible because amending claims 1 and 11 to recite “calculating a geometric metric between a computationally generated segmentation surface and physical clinical markers” and amending claim 18 to recite “computationally sorting the data, extracting a predefined top percentage subset, and merging it with an original baseline dataset to build a new training data structure” transform the claimed subject matter to a “specific, technical sequence of operations that alter how the machine learning system is trained”, such that “the claims are now fundamentally aligned with the subject matter found eligible in Desjardins, as they provide a specific, technical solution to the technological problem of improving machine learning performance and training efficiency and post- deployment model deployment accuracy” (Applicant’s Remarks, Pg. 11, Para. 3-4).
However, as discussed in detail above, the specification sets forth the alleged improvements to machine learning model training, through identification of clinical data using an evaluation metric and model retraining, in only a conclusory manner (see MPEP 2106.04(d)(1); see also Applicant’s Spec. Para. [0056] – [0058]). Therefore, the claimed subject matter, as currently formulated, is fundamentally distinct from the subject matter of the claims discussed in Desjardins.
As a result, the arguments are not persuasive.
4) Fourth, Applicant argues that the dependent claims are subject matter eligible because they each depend on one of the independent claims, which Applicant argues are all subject matter eligible (Applicant’s Remarks, Pg. 11-12, Para. 5-2). However, as discussed in detail above, arguments in favor of the subject matter eligibility of the independent claims are not persuasive.
As a result, the argument is not persuasive.
III. Applicant argues the rejections to the claims, under 35 USC § 103, should be withdrawn
(Applicant’s Remarks, 06/30/2026, Pg. 12, Section “IV. Claim Rejections under 35 U.S.C. § 103”).
In response to Applicant’s amendments, the previously communicated rejections under 35 U.S.C. § 103, have been withdrawn. However, Applicants arguments are not persuasive in light of the new grounds for rejection, under 35 U.S.C. § 103, discussed in detail above. The new grounds of rejection rely on new combinations of the existing prior art of record to teach the new combinations of elements in the amended claims, which were not presented in these arrangements in any of the previously presented claims. As a result, Applicant arguments against the previously communicated rejections under 35 U.S.C. § 103 are rendered moot.
However, for clarity of the record and in the interest of compact prosecution, arguments still relevant to the new grounds of rejection are discussed below.
1) First, Applicant argues “The References Fail to Teach or Suggest the Claimed Evaluation Metric” (Applicant’s Remarks, Pg. 13, Para. 2-5). Specifically, Applicant argues “[n]one of the cited references disclose this specific, physical-geometric evaluation mechanism” because “Bernhardt does not teach generating a segmentation surface, nor does it teach calculating a geometric metric against physical clinical markers” and “He contains no disclosure regarding medical imaging, segmentation surfaces, or physical clinical markers” (Applicant’s Remarks, Pg. 13, Para. 2-5). However, these arguments are insufficient because they amount to attacking references individually where the rejection is based on a combination of references (see MPEP 2145, “One cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references”).
Additionally, Applicant argues “Wong fails to teach the claimed clinical evaluation metric” because “Wong does not teach or suggest evaluating the generated segmentation surface against actual physical clinical markers associated with a medical procedure (such as physical ablation tags or surgical markers utilized during a procedure)” (Applicant’s Remarks, Pg. 13, Para. 2-5). However, as discussed in detail above, Wong discloses elements that are within the broadest reasonable interpretation of evaluating the generated segmentation surface against physical clinical markers associated with a medical procedure (MPEP 2111, “During patent examination, the pending claims must be given their broadest reasonable interpretation consistent with the specification”). Notably, the interpretation of what constitutes physical clinical markers associated with a medical procedure is not limited to physical ablation tags or surgical markers utilized during a procedure because these limitations are not positively recited in the claims and, even if positively recited in the specification, are not read into the claims (see MPEP 2145, “Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims”). Furthermore, Applicant’s assertions that Wong fails to disclose these elements are insufficient because they amount to a general allegation that the claim defines a patentable invention without specifically pointing out how the language of the claim patentably distinguishes it from the references (see 37 C.F.R. 1.111(b), “In order to be entitled to reconsideration or further examination, . . . The reply by the applicant or patent owner must . . . specifically points out the supposed errors in the examiner’s action . . . A general allegation that the claims define a patentable invention without specifically pointing out how the language of the claims patentably distinguishes them from the references does not comply with the requirements of this section”).
As a result, the arguments are not persuasive.
2) Second, Applicant argues “The proposed Combination Fails to Arrive at the Claimed Invention” (Applicant’s remarks, Pg. 14, Para. 1-4). Specifically, Applicant argues “[b]ecause none of Bernhardt, Wong, or He teaches calculating a geometric metric between a segmentation surface and physical clinical markers, no combination of these references can yield the claimed invention” and “[t]he cited references remain entirely silent on a technical solution of the amended claims: evaluating the model post-deployment by measuring the physical, geometric deviation between the generated anatomy surface and the actual physical clinical markers encountered during a medical procedure” such that “there is no teaching or suggestion in the art that would lead a POSITA to modify the references to arrive at this feature” (Applicant’s remarks, Pg. 14, Para. 1-4). However, as discussed in detail above, Wong discloses elements that are within the broadest reasonable interpretation of evaluating the generated segmentation surface against physical clinical markers associated with a medical procedure (MPEP 2111). Furthermore, Applicant’s assertions that Wong fails to disclose these elements are insufficient because they amount to a general allegation that the claim defines a patentable invention without specifically pointing out how the language of the claim patentably distinguishes it from the references (see 37 C.F.R. 1.111(b)).
As a result, the arguments are not persuasive.
3) Third, Applicant argues “The Cited Art Fails to Teach the Claimed Data Restructuring and Retraining Mechanism” (Applicant’s Remarks, Pg. 15, Para. 2-5). Specifically, Applicant argues “[t]he amended claim requires a specific sequence for assembling a new training data set: (1) sorting the data, (2) applying a threshold to extract a predefined top percentage to define a challenging data subset, and (3) adding this subset to the original training data set used to initially train the model to form a new training data set”, which “[n]one of the cited references teach or suggest” because “Bernhardt does not teach extracting a subset of data and appending it to an original baseline training data set to form a new training data set. Bernhardt merely cleans a dataset in place” (Applicant’s Remarks, Pg. 15, Para. 2-3). However, as discussed in detail above, Bernhardt discloses elements that are within the broadest reasonable interpretation of adding the challenging data subset to an original training data set used to initially train the machine learned network to form a new training data set, and executing the retraining using the new training data set (MPEP 2111). Notably, the interpretation of what constitutes adding is not limited to extracting or appending because these limitations are not positively recited in the claims and, even if positively recited in the specification, are not read into the claims (see MPEP 2145). Furthermore, Applicant’s assertions that Bernhardt fails to disclose these elements are insufficient because they amount to a general allegation that the claim defines a patentable invention without specifically pointing out how the language of the claim patentably distinguishes it from the references (see 37 C.F.R. 1.111(b)).
Additionally, Applicant argues “He does not teach sorting data, selecting a "predefined top percentage" to define a challenging subset, and appending that subset to an original training data set to form a new training data set. He simply discards bad data” (Applicant’s Remarks, Pg. 15, Para. 4). However, this argument is insufficient because it amounts to attacking references individually where the rejection is based on a combination of references (see MPEP 2145).
Next, Applicant argues “Wong is cited generally for deploying a model in a clinical environment and does not cure the deficiencies of Bernhardt and He regarding the algorithmic construction of the retraining data set” (Applicant’s Remarks, Pg. 15, Para. 5). However, this amounts to a general allegation that the claim defines a patentable invention without specifically pointing out how the language of the claim patentably distinguishes it from the references (see 37 C.F.R. 1.111(b)).
As a result, the arguments are not persuasive.
4) Fourth, Applicant argues “The Proposed Combination Fails to Arrive at the Claimed Invention” (Applicant’s Remarks, Pg. 15-16, Para. 6-4). Specifically, Applicant argues “[b]ecause the cited references lack the specific data processing steps recited in the amended claim, the proposed combination cannot render the claim obvious” and “[t]he references are entirely silent on the claimed technical solution: extracting a specific predefined top percentage of sorted data to form a challenging subset, and systematically merging that challenging subset back into the original training data set to construct a new enriched training data set for retraining”, which “prevents "catastrophic forgetting" during retraining by ensuring the model retains its baseline knowledge (from the original data set) while being forced to learn from the hardest new edge-cases”, such that “[t]he cited art provides no teaching, suggestion, or motivation to construct training data in this specific manner” (Applicant’s Remarks, Pg. 16, Para. 1-4).
However, as discussed in detail above, Bernhardt discloses elements that are within the broadest reasonable interpretation of adding the challenging data subset to an original training data set used to initially train the machine learned network to form a new training data set, and executing the retraining using the new training data set (MPEP 2111). Notably, the interpretation of what constitutes the steps of adding and executing are not limited by the Applicant-suggested potential technical effects of this approach, because these potential technical effects are not positively recited in the claims and, even if positively recited in the specification, are not read into the claims (see MPEP 2145). Furthermore, Applicant’s assertions that Bernhardt fails to disclose these elements are insufficient because they amount to a general allegation that the claim defines a patentable invention without specifically pointing out how the language of the claim patentably distinguishes it from the references (see 37 C.F.R. 1.111(b)).
As a result, the arguments are not persuasive.
5) Fifth, Applicant argues “[b]ecause the cited prior art fails to teach the limitations of the independent claims, and the secondary references do not supply the missing limitations, the combinations fail to render the dependent claims obvious” (Applicant’s Remarks, Pg. 16-17, Para. 6-2). However, as discussed in detail above, arguments in favor of the allowability of the independent claims were not persuasive.
As a result, the argument is not persuasive.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW BRYCE GOLAN whose telephone number is (571)272-5159. The examiner can normally be reached Monday through Friday, 8:00 AM to 5:00 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571) 270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW BRYCE GOLAN/Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights