Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 4, 10, and 13 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 4 and 13 recites the limitation "the training dataset.” However, claim 1 recites “the training dataset level,” which identifies a level of feedback, but does not positively introduce a training dataset. Therefore, it is unclear what previously recited training dataset is being referenced by “the training dataset.” There is insufficient antecedent basis for this limitation in the claim.
Claim 10 recites the limitation “the quantitative feedback.” However, claim 1 recites “feedback,” but does not recite or otherwise define “quantitative feedback.” Therefore, it is unclear what previously recited quantitative feedback is being referred. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-2, 4, 8-16, and 19-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Itu et al., (Pub. No.: EP3786972A1 (Published: 2021)).
Regarding claim 1, Itu discloses:
A method of providing feedback to a machine learning model, the method comprising: allowing a user to observe an output of a trained machine learning model (Itu, paragraph [0018] “At step 104, measures of interest for a primary task and one or more secondary tasks are predicted from the one or more input medical images (and optionally the patient data, if any) using a trained machine learning model.” [0022] “At step 106, the predicted measures of interest for the primary task and the one or more secondary tasks are output…outputting the predicted measures of interest for the primary task and the one or more secondary tasks includes visually displaying the predicted measures of interest, e.g., on a display device of a computer system. For example, the predicted measures of interest for the primary task and the one or more secondary tasks may be displayed along with the one or more medical images to facilitate user evaluation of the predicted measures of interest (e.g., for the one or more secondary tasks).”);
allowing the user to input feedback to the machine learning model based on the output, wherein the feedback is on at least one of a model level or on a training dataset level (Itu, paragraph [0023] “At step 108, user feedback on the predicted measures of interest for the one or more secondary tasks is received…The user feedback may be in any suitable form. In one embodiment, the user feedback is an acceptance or rejection by the user of the predicted measures of interest for the one or more secondary tasks. In another embodiment, the user feedback is user input correcting or modifying the predicted measures of interest for the one or more secondary tasks. For example, the user may interact with a user interface to select a common image point that is incorrectly predicted and move the common image point to a correct or desired location (e.g., in 2D or 3D space).” [0024] “In one embodiment, additional training data is formed comprising the one or more input medical images and the user feedback (as the ground truth value), and such additional training data is used to retrain the trained machine learning model” [0042] “Interactive machine learning enables a user to provide feedback to the machine learning model, enabling online retraining.” – Itu teaches receiving user feedback on the predicted measures output by the trained machine learning model, where the feedback may be an acceptance, rejection, correction, or modification of the predicted measures. Itu further teaches forming additional training data using the user feedback as ground truth value. Under the broadest reasonable interpretation, feedback used as ground truth in additional training data corresponds to feedback on a training dataset level. Accordingly, Itu teaches the claimed limitation.); and
incorporating the feedback into the machine learning model to improve the machine learning model, wherein the method is performed using one or more processors (Itu, paragraph [0024] “At step 110, the trained machine learning model is retrained for predicting the measures of interest for the primary task and the one or more secondary tasks based on the received user feedback on the predicted measures of interest for the one or more secondary tasks. In one embodiment, additional training data is formed comprising the one or more input medical images and the user feedback (as the ground truth value), and such additional training data is used to retrain the trained machine learning model…Because of the shared layers of the machine learning model, such retraining based on user feedback on the predicted measures of interest for the one or more secondary tasks implicitly leads to updating the machine learning model for predicting the measures of interest for all tasks” [0061] “Computer 602 includes a processor 604 operatively coupled to a data storage device 612 and a memory 610…Accordingly, by executing the computer program instructions, the processor 604 executes the method and workflow steps or functions of Figures 1-2.”).
Regarding claim 2, Itu discloses:
The method of claim 1, wherein the feedback on the model level comprises qualitative feedback on the machine learning model (Itu, paragraph [0041] “The performance of the one or more secondary tasks may be evaluated (e.g., visually) by a user to thereby evaluate the performance of the machine learning model.” [0042] “User feedback on the performance of the one or more secondary tasks may be received for on line retraining of the machine learning model using interactive machine learning. Interactive machine learning enables a user to provide feedback to the machine learning model, enabling online retraining.” [0023] “The user feedback may be in any suitable form.” – It teaches that the user visually evaluates the performance of the machine learning model and provides feedback on that performance to the machine learning model. Under BRI, feedback based on a user’s evaluation of model performance corresponds to qualitative feedback on the machine learning model.).
Regarding claim 3, Itu discloses:
The method of claim 1, wherein the feedback on the dataset level comprises quantitative feedback on the output of the machine learning model and wherein incorporating the feedback into the machine learning model comprises: writing back the feedback in a write-back dataset; and merging the write-back dataset with the training dataset; the method further comprising retraining the machine learning model with the training dataset; and looping back to the step of allowing a user to observe an output of a trained machine learning model to iteratively improve the machine learning model (Itu, paragraph [0024] “At step 110, the trained machine learning model is retrained for predicting the measures of interest for the primary task and the one or more secondary tasks based on the received user feedback on the predicted measures of interest for the one or more secondary tasks. In one embodiment, additional training data is formed comprising the one or more input medical images and the user feedback (as the ground truth value), and such additional training data is used to retrain the trained machine learning model, e.g., according to method 200 of Figure 2. For example, if the user corrects a location of a common image point in one of the input medical images, the trained machine learning model is retrained using the one or more input medical images with the corrected location of the common image point as the ground truth value… Because of the shared layers of the machine learning model, such retraining based on user feedback on the predicted measures of interest for the one or more secondary tasks implicitly leads to updating the machine learning model for predicting the measures of interest for all tasks (i.e., the primary task and the one or more secondary tasks). The steps of method 100 may be performed any number of times for newly received one or more input medical images.” – Itu teaches forming additional training data using the input medical images and user feedback as the ground truth value, and using that additional data to retrain and update the trained machine learning model. Under BRI, forming additional data from the user feedback corresponds to writing the feedback into a write-back dataset, and using the feedback based training data for retraining corresponds to merging the write-back dataset with the training dataset. Itu further teaches performing method 100 any number of times, which corresponds to looping back to allow the users to observe additional outputs and iteratively improve the machine learning model.).
Regarding claim 8, Itu discloses:
The method of claim 1, wherein the incorporating of the feedback is performed as write-backs (Itu, paragraph [0024] “In one embodiment, additional training data is formed comprising the one or more input medical images and the user feedback (as the ground truth value), and such additional training data is used to retrain the trained machine learning model… For example, if the user corrects a location of a common image point in one of the input medical images, the trained machine learning model is retrained using the one or more input medical images with the corrected location of the common image point as the ground truth value.” – Itu teaches incorporating user feedback by forming additional training data that includes the user feedback as a ground truth value and using that feedback based training data to retrain the machine learning model. Under BRI, writing or recording the user feedback back into training data for retraining corresponds to incorporating the feedback as write-backs.).
Regarding claim 9, Itu discloses:
The method of claim 2, wherein the qualitative feedback is reviewed by a user and manually incorporated into the machine learning model (Itu, paragraph [0023] “At step 108, user feedback on the predicted measures of interest for the one or more secondary tasks is received... The user feedback may be in any suitable form. In one embodiment, the user feedback is an acceptance or rejection by the user of the predicted measures of interest for the one or more secondary tasks. In another embodiment, the user feedback is user input correcting or modifying the predicted measures of interest for the one or more secondary tasks. For example, the user may interact with a user interface to select a common image point that is incorrectly predicted and move the common image point to a correct or desired location” [0024] “At step 110, the trained machine learning model is retrained for predicting the measures of interest for the primary task and the one or more secondary tasks based on the received user feedback” [0041] “The performance of the one or more secondary tasks may be evaluated (e.g., visually) by a user to thereby evaluate the performance of the machine learning model.” [0042] “User feedback on the performance of the one or more secondary tasks may be received for on line retraining of the machine learning model using interactive machine learning.” – Itu teaches receiving user feedback in any suitable form, including acceptance or rejection of the predicted measures, and also teaches user input correcting or modifying the predicted measures through interaction with a user interface. Itu further teaches retraining the trained machine learning model based on the received user feedback, and teaches that the user visually evaluates the performance of the machine learning model and provides feedback on that performance. Under BRI, user evaluation of model performance and acceptance/rejection of the model output correspond to qualitative feedback reviewed by the user, and user input provided through interaction with a user interface and used for retraining corresponds to manually incorporating the feedback into the machine learning model.).
Regarding claim 10, Itu discloses:
The method of claim 1, wherein the quantitative feedback is automatically incorporated into the machine learning model (Itu, paragraph [0023] “In another embodiment, the user feedback is user input correcting or modifying the predicted measures of interest for the one or more secondary tasks. For example, the user may interact with a user interface to select a common image point that is incorrectly predicted and move the common image point to a correct or desired location” [0024] “At step 110, the trained machine learning model is retrained for predicting the measures of interest for the primary task and the one or more secondary tasks based on the received user feedback…In one embodiment, additional training data is formed comprising the one or more input medical images and the user feedback (as the ground truth value), and such additional training data is used to retrain the trained machine learning model… For example, if the user corrects a location of a common image point in one of the input medical images, the trained machine learning model is retrained using the one or more input medical images with the corrected location of the common image point as the ground truth value.” [0061] “Computer 602 includes a processor… Accordingly, by executing the computer program instructions, the processor 604 executes the method and workflow steps or functions of Figures 1-2.” – Itu teaches user feedback that corrects or modifies a predicted measure, including moving an incorrectly predicted image point to a correct location. Under BRI, correcting a predicted measure or location corresponds to quantitative feedback. Itu further teaches that the trained machine learning model is retrained based on the received user feedback, where additional training data is formed using the user feedback as the ground truth value. Because Itu teaches that processor 604 executes the method and workflow steps, the system/processor automatically forms the feedback based training data and retrains the machine learning model using the quantitative feedback. Accordingly, Itu teaches automatically incorporating the quantitative feedback into the machine learning model.).
Regarding claim 11, Itu discloses:
The method of claim 1, wherein the feedback captures accumulated experience and know-how of subject-matter experts over time (Itu, paragraph [0022] “outputting the predicted measures of interest for the primary task and the one or more secondary tasks includes visually displaying the predicted measures of interest, e.g., on a display device of a computer system. For example, the predicted measures of interest for the primary task and the one or more secondary tasks may be displayed along with the one or more medical images to facilitate user evaluation of the predicted measures of interest (e.g., for the one or more secondary tasks).” [0023] “At step 108, user feedback on the predicted measures of interest for the one or more secondary tasks is received… The user feedback may be in any suitable form… In one embodiment, the user feedback is an acceptance or rejection by the user of the predicted measures of interest for the one or more secondary tasks” [0024] “In one embodiment, additional training data is formed comprising the one or more input medical images and the user feedback (as the ground truth value), and such additional training data is used to retrain the trained machine learning model…The steps of method 100 may be performed any number of times for newly received one or more input medical images.” - Itu teaches displaying predicted measures with medical images to facilitate user evaluation, receiving user feedback on predicted measures, and retraining the machine learning model based on the received user feedback. Under BRI, feedback provided by the user evaluating predicted measures in a medical image context corresponds to feedback from subject-matter expert. Itu further teaches that the feedback based method may be performed any number of times for newly received input medical images, which corresponds to accumulating and incorporating the users’ experience and know-how over time.)
Regarding claim 12, Itu discloses:
The method of claim 1, wherein the feedback is bias monitored (Itu, paragraph [0022] “the predicted measures of interest for the primary task and the one or more secondary tasks may be displayed along with the one or more medical images to facilitate user evaluation of the predicted measures of interest” [0023] “At step 108, user feedback on the predicted measures of interest for the one or more secondary tasks is received… In another embodiment, the user feedback is user input correcting or modifying the predicted measures of interest for the one or more secondary tasks.” [0024] “At step 110, the trained machine learning model is retrained for predicting the measures of interest for the primary task and the one or more secondary tasks based on the received user feedback… In one embodiment, additional training data is formed comprising the one or more input medical images and the user feedback (as the ground truth value), and such additional training data is used to retrain the trained machine learning model” [0041] “The performance of the one or more secondary tasks may be evaluated (e.g., visually) by a user to thereby evaluate the performance of the machine learning model.” [0042] “User feedback on the performance of the one or more secondary tasks may be received for on line retraining of the machine learning model using interactive machine learning.” – Itu teaches that predicted measures are displayed for user evaluation, user feedback is received by correcting or modifying the predicted measures, and the feedback is used as ground truth in additional training data for retraining the machine learning model. Itu further teaches evaluating the performance of the machine learning model and receiving feedback on that performance. Under BRI, monitoring feedback for bias includes evaluating feedback based corrections and their effect on model performance when the feedback is used for retraining. Accordingly, Itu teaches that the feedback is bias monitored.).
Regarding claim 13, Itu discloses:
The method of claim 1, wherein the training dataset is bias monitored (Itu, paragraph [0023] “At step 108, user feedback on the predicted measures of interest for the one or more secondary tasks is received… The user feedback may be in any suitable form.” [0024] “At step 110, the trained machine learning model is retrained for predicting the measures of interest for the primary task and the one or more secondary tasks based on the received user feedback… In one embodiment, additional training data is formed comprising the one or more input medical images and the user feedback (as the ground truth value), and such additional training data is used to retrain the trained machine learning model” [0041] “The performance of the one or more secondary tasks may be evaluated (e.g., visually) by a user to thereby evaluate the performance of the machine learning model.” [0042] “User feedback on the performance of the one or more secondary tasks may be received for on line retraining of the machine learning model using interactive machine learning.” – Itu teaches forming additional training data from the input medical images and the user feedback as the ground truth value, and using that feedback based training data to retrain the machine learning model. Itu further teaches evaluating the performance of the machine learning model and receiving feedback on that performance. Under BRI, monitoring the training dataset for bias includes monitoring feedback based training data by evaluating its effect on model performance during the retraining feedback loop. Accordingly, Itu teaches that the training dataset is biased monitored.).
Regarding claim 14, Itu discloses:
The method of claim 1, wherein the feedback on the model level and the feedback on the training dataset level are input to the machine learning model via one common graphical user interface (Itu, paragraph [0022] “In one embodiment, outputting the predicted measures of interest for the primary task and the one or more secondary tasks includes visually displaying the predicted measures of interest, e.g., on a display device of a computer system… the predicted measures of interest for the primary task and the one or more secondary tasks may be displayed along with the one or more medical images to facilitate user evaluation of the predicted measures of interest” [0023] “At step 108, user feedback on the predicted measures of interest for the one or more secondary tasks is received… The user feedback may be in any suitable form. In one embodiment, the user feedback is an acceptance or rejection by the user of the predicted measures of interest for the one or more secondary tasks. In another embodiment, the user feedback is user input correcting or modifying the predicted measures of interest for the one or more secondary tasks. For example, the user may interact with a user interface to select a common image point that is incorrectly predicted and move the common image point to a correct or desired location” [0024] “In one embodiment, additional training data is formed comprising the one or more input medical images and the user feedback (as the ground truth value), and such additional training data is used to retrain the trained machine learning model” – It teaches visually displaying predicted measures with medical images on a display device and receiving user feedback through user interaction with displayed medical image information. Under BRI, visually displaying images and predicted measures on a display device and allowing the user to select and move displayed image points through a user interface, corresponds to a graphical user interface. Itu further teaches feedback in a form of acceptance or rejection of the predicted measures, which corresponds to feedback on the model level, and feedback in the form of correcting or modifying predicted measures, where the feedback is used as ground truth in additional training data, which corresponds to feedback on a training dataset level. Accordingly, Itu teaches inputting feedback on both the model level and the training dataset level through one common graphical user interface.).
Regarding claim 15, Itu discloses:
The method of claim 1, wherein the method is used in a system to predict a variable based on medical images, the user is a clinician and allowing the user to input feedback comprises selecting an area of the image by the clinician, passing the selected area of the image to the machine learning model for a preliminary prognosis, evaluating the output of the model by the clinician and inputting the evaluation as feedback (Itu, paragraph [0015] “At step 102, one or more input medical images are received.” [0018] “At step 104, measures of interest for a primary task and one or more secondary tasks are predicted from the one or more input medical images (and optionally the patient data, if any) using a trained machine learning model.” [0020] “In another embodiment, the primary task is a clinical decision (e.g., taken during or after an intervention), such as, e.g., whether or not to perform percutaneous coronary intervention (PCI) or coronary artery bypass grafting (CABG), an optimal medical therapy, a date of a next examination, etc.” [0021] “The measures of interest for the one or more secondary tasks can be directly verified by the human user from the one or more input medical images. Examples of the one or more secondary tasks include predicting standard measurement locations in the input medical images, predicting a location of one or more common image points (e.g., common anatomical landmarks) in the input medical images, predicting a location of stenosis markers in the input medical images” [0022] “the predicted measures of interest for the primary task and the one or more secondary tasks may be displayed along with the one or more medical images to facilitate user evaluation of the predicted measures of interest” [0023] “At step 108, user feedback on the predicted measures of interest for the one or more secondary tasks is received… For example, the user may interact with a user interface to select a common image point that is incorrectly predicted and move the common image point to a correct or desired location” – Itu teaches a system that predicts measures of interest from medical images using a trained machine learning model, including clinical decisions such as whether to perform PCI or CABG, optimal medical therapy, or a date of the next examination. Under BRI, predicting a medical measure or clinical decision based on medical images corresponds to predicting a variable and providing a preliminary prognosis, and the human user evaluating such medical-image clinical outputs includes a clinician. Itu further teaches predicted medial-image locations, including standard measurement locations, common image points, and stenosis markers, and teaches the user interacting with the interface to select and correct a predicted image point. Under BRI, selecting a common image point or predicted medical image location corresponds to selecting an area of the image. Itu also teaches displaying predicted measures with the medical images for user evaluation and receiving user feedback on the predicted measures. Accordingly, Itu teaches the claim limitation.).
Regarding claim 16, Itu discloses:
The method of claim 15, wherein the feedback is quantitative feedback and comprises medical images annotated by the clinician (Itu, paragraph [0021] “Examples of the one or more secondary tasks include predicting standard measurement locations in the input medical images, predicting a location of one or more common image points (e.g., common anatomical landmarks) in the input medical images, predicting a location of stenosis markers” [0022] “the predicted measures of interest for the primary task and the one or more secondary tasks may be displayed along with the one or more medical images to facilitate user evaluation of the predicted measures of interest” [0023] “In another embodiment, the user feedback is user input correcting or modifying the predicted measures of interest for the one or more secondary tasks. For example, the user may interact with a user interface to select a common image point that is incorrectly predicted and move the common image point to a correct or desired location (e.g., in 2D or 3D space).” [0024] “For example, if the user corrects a location of a common image point in one of the input medical images, the trained machine learning model is retrained using the one or more input medical images with the corrected location of the common image point as the ground truth value.” – Itu teaches predicted locations in medical images, including standard measurement locations, common image points, and stenosis markers. Itu further teaches displaying the predicted measures with the medical images for use evaluation and receiving user input correcting or modifying the predicted measure by selecting an incorrectly predicted image point and moving it to a correct or desired location. Under BRI, a corrected image location used as ground truth value corresponds to quantitative feedback, and marking/correcting a location on a medical image corresponds to annotating the medical image. As discussed with respect to claim 15, the human user evaluating clinical medical image outputs includes a clinician under BRI. Accordingly, Itu teaches that the feedback is quantitative feedback comprising medical images annotated by a clinician.).
Regarding claim 19, Itu discloses:
One or more computer-readable storage media comprising computer executable instructions which, when executed by one or more processors, cause the one or more processors to perform the method of claim 1 (Itu, paragraph [0061] “Computer 602 includes a processor 604 operatively coupled to a data storage device 612 and a memory 610. Processor 604 controls the overall operation of computer 602 by executing computer program instructions that define such operations. The computer program instructions may be stored in data storage device 612, or other computer readable medium, and loaded into memory 610 when execution of the computer program instructions is desired.”).
Regarding claim 20, Itu discloses:
A computer system comprising: one or more processors; and one or more computer-readable storage media comprising computer executable instructions which when executed by the one or more processors cause the one or more processors to perform the method of claim 1 (Itu, paragraph [0061] “A high-level block diagram of an example computer 602 that may be used to implement systems, apparatus, and methods described herein is depicted in Figure 6. Computer 602 includes a processor 604 operatively coupled to a data storage device 612 and a memory 610. Processor 604 controls the overall operation of computer 602 by executing computer program instructions that define such operations. The computer program instructions may be stored in data storage device 612, or other computer readable medium, and loaded into memory 610 when execution of the computer program instructions is desired.”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3 and 5 are rejected under the 35 U.S.C. 103 as being unpatentable over Itu et al., (Pub. No.: EP3786972A1 (Published: 2021)) in view of Prabhakara et al., ( Pat. No.: US 12468960 B1 (Filed: 2019)) ).
Regarding claim 3, Itu teaches all the elements of claim 2, therefore is rejected for the same reasons as those presented for claim 2. Itu does not teach but Itu in view of Prabhakara teaches the following limitations:
wherein the feedback on the model relates to features and/ or hyperparameters of the machine learning model, the method further comprising retraining the machine learning model; and looping back to the step of allowing a user to observe an output of a trained machine learning model to iteratively improve the machine learning model (Itu, paragraph [0024] “At step 110, the trained machine learning model is retrained for predicting the measures of interest for the primary task and the one or more secondary tasks based on the received user feedback… The steps of method 100 may be performed any number of times for newly received one or more input medical images.” [0041] “The performance of the one or more secondary tasks may be evaluated (e.g., visually) by a user to thereby evaluate the performance of the machine learning model.” [0042] “User feedback on the performance of the one or more secondary tasks may be received for on line retraining of the machine learning model using interactive machine learning. Interactive machine learning enables a user to provide feedback to the machine learning model, enabling online retraining.” Prabhakara, [col. 2, lines 18-22] “In various embodiments, a hyperparameter in the set of hyperparameters comprises a number of epochs, an adaptive learning rate, a deep learning layer, a number of neurons in a layer, or any other appropriate hyperparameter.” [col. 4, lines 47-53] “In 322, the prediction model is retrained using hyperparameters and based on user feedback, and control passes to 312. For example, the prediction model is retrained using the best set of hyperparameters and the user feedback of whether detected anomalies are valid and whether there are any undetected anomalies.” – Itu teaches user feedback on the performance of the machine learning model, online retraining based on the feedback, and repeating/retraining process. Prabhakara teaches hyperparameters, including epochs, adaptive learning rate, deep learning layers, and number of neurons, and teaches retraining a prediction model using hyperparameters based on user feedback. In the combination, the model performance feedback taught by Itu is applied in a feedback based retraining workflow that uses selected hyperparameters, as taught by Prabhakara. Accordingly, Itu in view of Prabhakara teaches the claimed limitation.).
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having a combination of Itu and Prabhakara before them, to incorporate the use of selected hyperparameters during feedback based retraining, as taught by Prabhakara, into the feedback based online retraining system of Itu. One would have been motivated to make such a combination in order to improve retraining of the machine learning model using hyperparameters, such as epochs, adaptive learning rate, deep learning layers, and number of neurons, in a retraining workflow that also accounts for user feedback on model performance. This would allow the machine learning model to be more effectively updated based on user feedback while using selected hyperparameters to improve the resulting retrained prediction model.
Regarding claim 5, Itu teaches all the elements of claim 4, therefore is rejected for the same reasons as those presented for claim 4. Itu does not teach but Itu in view of Prabhakara teaches the following limitations:
wherein the user is allowed to flag the prediction output of the machine learning model as a false positive for a binary classification problem (Itu, paragraph [0023] “At step 108, user feedback on the predicted measures of interest for the one or more secondary tasks is received… In one embodiment, the user feedback is an acceptance or rejection by the user of the predicted measures of interest for the one or more secondary tasks. In another embodiment, the user feedback is user input correcting or modifying the predicted measures of interest for the one or more secondary tasks.” Prabhakara, [col. 4, lines 24-29] “In 314, detected anomalies are determined based on a difference between the prediction model output and a forecast. For example, the output of the prediction model and the forecast are used to identify anomalous areas or zones that are indicated as detected anomalies.” [col. 4, lines 31-35] “ In 314, detected anomalies are determined based on a difference between the prediction model output and a forecast. For example, the output of the prediction model and the forecast are used to identify anomalous areas or zones that are indicated as detected anomalies.” [col. 4, lines 38-44] “In 320, user feedback is received on detected anomalies and undetected anomalies. For example, user feedback is received via a user interface. In some embodiments, the user feedback comprises a false detected anomaly indication indicating that a detected anomaly of the detected anomalies is not an anomaly (e.g., a false positive).” – Itu teaches allowing a user to provide feedback on the prediction output of the machine learning model, including accepting, rejecting, correcting, or modifying the predicted output. Prabhakara teaches providing detected anomalies output by a prediction model to a user and receiving user feedback indicating that a detected anomaly is not an anomaly, i.e., false positive. Under BRI, classifying a prediction output as an anomaly or not an anomaly corresponds to a binary classification problem. Accordingly, Itu in view of Prabhakara teaches the claimed limitation.).
Claims 6 and 7 are rejected under the 35 U.S.C. 103 as being unpatentable over Itu et al., (Pub. No.: EP3786972A1 (Published: 2021)) in view of Schriver et al., (Pub. No.: US 20200118675 A1 (Filed: 2018)).
Regarding claim 6, Itu teaches all the elements of claim 4, therefore is rejected for the same reasons as those presented for claim 4. Itu does not teach but Itu in view of Schriver teaches the following limitations:
wherein when the prediction output is a predicted time period for an event the user is allowed to indicate how long the event took in reality (Itu, paragraph [0023] “At step 108, user feedback on the predicted measures of interest for the one or more secondary tasks is received… The user feedback may be in any suitable form… In another embodiment, the user feedback is user input correcting or modifying the predicted measures of interest for the one or more secondary tasks.” Schriver, paragraph [0082] “operation data can include one or more operation parameters associated with one or more operations of injection system 104.” [0083] “an operation parameter can include…a duration of time of one or more injections (e.g., a maximum, a minimum, an average, a total, etc.)” [0088] “the one or more predictive models are designed to receive, as an input, operation data associated with injection system 104 and, provide, as an output a prediction … as to one or more operation failures or misuses of injection system 104.” & “maintenance prediction system 102 can generate the one or more predictive models to determine one or more prediction scores that include a prediction of whether one or more operation failures or misuses of one or more injection systems (e.g., of one or more components or devices of one or more injection systems, etc.) will occur within a time period and/or a number of uses of the one or more injection systems.” – Itu teaches allowing a user to provide feedback by correcting or modifying a prediction output of a trained machine learning model. Schriver teaches a predictive maintenance system that outputs a prediction of whether an operation failure or misuse will occur within a time period and teaches operation data including a duration of time of one or more operations Under BRI, an operation failure or misuse corresponds to an event, and receiving a duration of time operation parameter corresponds to indicating how long the event took in reality. Accordingly, Itu in view of Schriver teaches the claimed limitation.).
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having a combination of Itu and Schriver before them, to modify the user feedback based machine learning system of Itu such that, when the prediction output is a predicted time period for an event, the user is allowed to indicate how long the event actually took, as suggested by Schriver’s teachings of time period based prediction outputs and duration of time operation data. One would have been motivated to make such a combination in order to allow the feedback and correction of a predicted time period to reflect the actual duration of the corresponding event. This would allow the machine learning model to use the actual duration feedback during retraining to improve future predictions of how long similar events will take.
Regarding claim 7, Itu in view of Schriver teaches all the elements of claim 6, therefore is rejected for the same reasons as those presented for claim 6. Itu in view of Schriver further teaches:
wherein the event is a maintenance task (Schriver, paragraph [0108] “ maintenance data includes operation data (e.g., one or more operation parameters associated with one or more operations of injection system 104, etc.) and/or data associated with one or more maintenance actions (e.g., a prompt to a user or operator to perform one or more maintenance actions, an instruction that causes injection system 104 to perform one or more maintenance actions, an indication that one or more maintenance actions have been scheduled to be performed for and/or with injection system 104” – Schriver teaches maintenance actions, including prompting a user to perform maintenance actions and indicating that maintenance actions have been scheduled or performed. Under BRI, a maintenance action corresponds to the claimed maintenance task.).
Claim 17 is rejected under the 35 U.S.C. 103 as being unpatentable over Itu et al., (Pub. No.: EP3786972A1 (Published: 2021)) in view of Baron et al., (Pub. No.: US 20210350930 A1 (Filed: 2020)).
Regarding claim 17, Itu teaches all the elements of claim 15, therefore is rejected for the same reasons as those presented for claim 15. Itu does not teach but Itu in view Baron teaches:
wherein the variable is a survival rate of lung cancer patients and the medical images are images showing human lungs or parts thereof (Itu, paragraph [0015] “At step 102, one or more input medical images are received.” [0018] “At step 104, measures of interest for a primary task and one or more secondary tasks are predicted from the one or more input medical images (and optionally the patient data, if any) using a trained machine learning model.” Baron, paragraph [0038] “FIG. 2A illustrates an example of using a machine learning model to perform a prediction of survival rate of a patient at a pre-determined time after diagnosis of a cancer… The survival rate can provide a likelihood that the patient survives at a pre-determined time (e.g., 500 days, 1000 days, 1500 days, etc.) after the patient is diagnosed of a medical condition (e.g., an advanced stage cancer).” [0041] “ It is understood that other categories of clinical data not shown in FIG. 2B, such as biopsy image feature data, can also be input to machine learning prediction model 200 to perform the clinical prediction.” [0093] “Three patient cohorts, one for each of metastatic colorectal cancer, metastatic breast cancer, and advanced lung cancer, are defined from three Flatiron DataMarts: Metastatic CRC (colorectal cancer); ii) Advanced NSSLC (non-small cell lung cancer) iii) Metastatic Breast Cancer.” [0097] “ Few if any patients had data for all of the potential predictors (lab tests, molecular biomarkers and clinical/demographic variables) for survival rate prediction as shown in FIG. 4A… For example, the prediction model LC.sub.A for advanced lung cancer is trained based on gender and race data of patients and have the data categories for gender and race labelled as “1” in FIG. 4A.” – Itu teaches the based medical-image machine learning system including predicting measures for input medical images using a trained machine learning model. Baron teaches using machine learning to predict a survival rate of a patient after diagnosis of cancer and specifically teaches prediction models for advanced non-small cell lung cancer/advanced lung cancer, where survival outcome data is extracted. Baron further teaches that biopsy image feature data can be input to the machine learning prediction model. Under BRI, biopsy image data associated with lung cancer corresponds to medical images showing human lungs or parts thereof. Accordingly, Itu in view of Baron teaches the claimed limitation.).
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having a combination of Itu and Baron before them, to apply the medical image machine learning feedback system of Itu to the lung-cancer survival rate prediction context taught by Baron. One would have been motivated to make such a combination in order to use feedback based improvement of a medical image machine learning model in a known clinical prediction application including survival rate prediction for lung cancer patients. This would allow the machine learning model to predict survival rates for lung cancer patients based on medical image information and improve the model through user feedback as taught by Itu.
Claim 18 are rejected under the 35 U.S.C. 103 as being unpatentable over Itu et al., (Pub. No.: EP3786972A1 (Published: 2021)) in view of Epstein et al., (Pub. No.: US 20230004889 A1 (Filed: 2022)).
Regarding claim 18, Itu teaches all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1. Itu does not teach but Itu in view Epstein teaches:
wherein the method is used in a system to assess whether or not to accept new clients of a financial institute in view of sanctions (Epstein, paragraph [0041] “Through embodiments of the present invention, financial institutions, multinational corporations, legal professionals, and other stakeholders can access visual graphs depicting relationships between relevant actors in order to evaluate their possible exposure to financial, trade, or other business-related risks.” [0042] “With respect to the application to sanctions, sanctions refers to laws or regulations issued by a government that places commercial restrictions on certain identified entities or entities that are described in more general terms or indirectly (e.g., all subsidiaries of an entity that is specifically identified) and also place restrictions on commercial activities involving the restricted entities.” [0077] “FIG. 8 depicts another illustrative computer-implemented system 800 for providing a visual interactive software tool that permits user to investigate financial, trade and other business risks, which as described in this illustrative application is directed to investigating and evaluating financial crime and sanctions-related risks.” [0079] “The data are used in a different operational use case where financial institutions and companies screen customers, transactions, and/or counterparties to determine if they have material associations with sanctioned actors.” – Epstein teaches a sanctions-risk related system used by financial institutions to evaluate exposure to sanctions-related risks. Epstein further teaches that sanctions include legal or regulatory restrictions on identified entities or indirectly restricted entities, and that financial institutions screen customers, transactions, and counterparties to determine whether they have material associations with sanctioned actors. Under BRI, screening customers to determine whether they are associated with sanctioned actors corresponding to assessing whether or not to accept new clients of a financial institute in view of sanctions.).
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having a combination of Itu and Epstein before them, to apply the feedback based machine learning improvement method of Itu to the sanctions-risk screening system of Epstein. One would have been motivated to make such a combination in order to improve a financial institution’s customer screening process by using feedback based model improvement in a known sanctions related risk environment. This would allow the system to assess whether customers or potential clients should be accepted or rejected in view of sanctions-related risk while improving the model through user feedback as taught by Itu.
Conclusion
The prior art of record and not relied upon is considered pertinent to Applicant’s disclosure:
1. Teso, Stefano, and Kristian Kersting. "Explanatory interactive machine learning." Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. 2019. – Teso teaches explanatory interactive machine learning, including presenting the model predictions/explanations to a user, receiving user corrections as feedback, and updating the model based on feedback.
2. Guo, Lijie, et al. "Building trust in interactive machine learning via user contributed interpretable rules." Proceedings of the 27th international conference on intelligent user interfaces. 2022. – Guo teaches an explanation-driven interactive machine learning system which users provide feedback by correcting system predictions and/or modifying interpretable rules, and the system evaluates how the updated rule increases or decreases model accuracy.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Daravanh Phakousonh whose telephone number is (571)272-6324. The examiner can normally be reached Mon - Thurs 7 AM - 5 PM, Every other Friday 7 AM - 4PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at 571-272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Daravanh Phakousonh/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights