Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
A. Claims 1-3 and 5 are rejected under 35 U.S.C. 103 as being unpatentable over Koehrsen (“Beyond Accuracy: Precision and Recall”, 2018) in view of Bernard (US 2018/0341752)
As for claim 1, Koehrsen teaches
A method, comprising:
tuning an operating point of a static model configured to output a presence or absence of a specific finding in one or more [data elements] (selecting a classifier threshold, p 9 as discussed below)
by executing the static model on a set of annotated [data elements] (p 7 par 1, “we put in information about patients and receive a score between 0 and 1”) and
comparing output from the static model for each annotated [data elements] of the set of annotated [data elements] to each of a plurality of possible operating points (p 7 table “Outcome of model at each threshold”, each threshold can be called an “operating point”; p 7 par 1 “evaluate thresholds”: pg 5 par 1 (bold text) discusses how the threshold is used – a data point or score is determined to be above or below the threshold, i.e. “compared”; p 7 table lists a plurality of thresholds and discusses how each is applied to the data point), and
selecting the operating point from the plurality of possible operating points that results in a target tuning metric (p 9 table and par below, determining that the threshold of 0.5 yields the best F1 score, e.g. “a target tuning metric”); and
executing the static model on a subsequent [data element] to determine a presence or absence of the specific finding in the subsequent [data element] by comparing output of the static model for the subsequent [data elements] to the selected operating point (pg 9, it is clear that selecting the threshold (e.g. 0.5) is to classify subsequent data elements; in addition pg 7 discusses processing 100 data elements, so any one of them could be called “subsequent”)
Koehrsen doesn’t specifically teach, Bernard however teaches
[classifying] medical images (Fig 1, medical image classification)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention, to modify the data classification method of Koehrsen by applying it to medical images of Bernard, as both pertain to the art of classifying data elements. The motivation to do so would have been, producing reliable results by applying well-known classification techniques.
As for claim 2, the combination of Koehrsen and Bernard teaches
the target tuning metric comprises a target sensitivity, a target specificity, a target accuracy, a target positive predictive value, and/or a target negative predictive value (Koehrsen p 6 summarizes a number of classification accuracy metrics taught throughout the reference)
As for claim 3, the combination of Koehrsen and Bernard teaches
the target tuning metric comprises maximum accuracy and wherein selecting the operating point from the plurality of possible operating points that results in maximum accuracy comprises:
comparing the output from the static model for each annotated medical image of the set of annotated medical images to a first possible operating point to determine, for each annotated medical image, whether that annotated medical image is positive or negative for the specific finding (Koehrsen p 7 as previously discussed);
assigning a first tuning metric value to each annotated medical image based on whether the determination of the positive or negative for the specific finding for each annotated medical image matches an indication of whether that annotated medical image is positive or negative for the specific finding as conveyed by an annotation of that annotated medical image (Koehrsen. P 6 “Recap”, discusses True positives, False positives, True negatives and False negatives – data labeled as positive or negative, i.e. “determination”, that are actually positive or negative, i.e. “annotation”);
summing each tuning metric value to determine a summary score for the first operating point; (Koehrsen p2 2nd - 3rd full paragraph, recall and precision are based on calculating true positives, i.e. “successes”, and false positives and negatives, i.e. “failures”)
determining a summary score for each additional possible operating point by comparing the output from the static model for each annotated medical image to each additional possible operating point and assigning a respective second tuning metric value to each annotated medical image for each additional possible operating point (p 9 table, various scores measured for multiple thresholds); and
selecting the possible operating point that has the lowest summary score (text below p 9 table)
As for claim 5, the combination of Koehrsen and Bernard teaches
the static model is configured to output a presence or absence of a specific diagnostic finding in one or more x-ray images, wherein the set of annotated medical images comprises a set of annotated x-ray images, each x-ray image of the set of annotated x-ray image including an annotation from an expert indicating a presence or absence of the specific diagnostic finding in that x-ray image (Bernard [0051] x-ray for automated diagnostics)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention, to modify the teaching of Koehrsen and Bernard, to further include the x-ray images in the analysis, as all pertain to classifying data elements. The motivation to do so would have been, to apply well-known classification techniques to well-known medical imaging systems.
B. Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Koehrsen in view of Bernard in further view of Markham (“Roc curves and Area Under the Curve explained”, DataSchool, 2014, https://www.dataschool.io/roc-curves-and-auc-explained/)
As for claim 4, the combination of Koehrsen and Bernard teaches
selecting the operating point from the plurality of possible operating points that results in the target tuning metric comprises:
determining a specificity value and a sensitivity value for each possible operating point based on the output from the static model for each annotated medical image relative to each of a plurality of possible operating points and further based on, for each annotated medical image, whether that annotated medical image is positive or negative for the specific finding as conveyed by an annotation of that annotated medical image (pg 9 table as discussed previously);
The combination of Koehrsen and Bernard doesn’t specifically teach, Markham however teaches
plotting each specificity value as a function of a corresponding sensitivity value to form a metric curve; (Markham 0:00 the ROC curve plots the true positive rate and false positive rate on a user interface)
outputting the metric curve for display on a display device; (a display device is inherent to displaying a user interface)
receiving a user input selecting a point on the metric curve (6:16, 7:05 the threshold bar is moved, and simultaneously the red dot on the ROC curve is moved accordingly); and
setting the selected operating point as the operating point corresponding to the selected point on the metric curve (moving the threshold bar sets the new threshold value)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention, to modify the classification method of Koehrsen and Bernard by further including the ROC curve user interface of Markham, as all pertain to classifying data elements. The motivation to do so would have been, to allow the user to conveniently visualize the effects of different threshold choices.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARK ROZ whose telephone number is (571)270-3382. The examiner can normally be reached on 9AM-5PM M-F.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Moyer can be reached on (571)272-9523. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MARK ROZ/
Primary Examiner, Art Unit 2669
Prosecution Insights