DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 have been examined.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3, 10 and 17 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 3 recites “accumulating the model class scores in a decreasing ranking of the plurality of conformal scores until an index of the class in the decreasing ranking.” This limitation is not understood. There appears to be a missing object to for the phrase “until an index of the class …” That is, “until an index of the class” … what? This phrase will be interpreted as “until an index of the class in the decreasing ranking is reached.”
Claims 10 and 17 include limitations similar to claim 3 and are rejected for the same reason.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 4-9, 11-16 and 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication 20230334245 by Kumar et al. (“Kumar”) in view of U.S. Patent Application Publication 20130070986 by Peleg et al. (“Peleg”).
Regarding claim 1, Kumar discloses:
1. A system for selective model intervention, comprising: a processor configured to execute instructions; and a computer-readable medium having instructions executable by the processor for: See Kumar, Fig. 3, depicting a system.
applying a computer model to a data sample to determine a plurality of model class scores corresponding to a plurality of class outputs of a plurality of classes; Kumar, Fig. 5 element 504 and ¶ 0059, “… generating, via a base classifier model (e.g., 220 in FIG. 2), a plurality of predicted labels (e.g., prediction labels 204 in FIG. 2) corresponding to an input of the plurality of texts from the calibration dataset.”
determining a plurality of conformal scores corresponding to the plurality of class outputs based on the plurality of model class scores; Kumar, Fig. 5 element 506 and ¶ 0060, “computing a first set of non-conformity scores (e.g., scores 206 in FIG. 2) by comparing the plurality of predicted labels (e.g., label predictions 204 in FIG. 2) and the corresponding labels from the calibration dataset (e.g., 202 in FIG. 2).”
determining a class membership set based on the plurality of conformal scores and a conformal threshold; Kumar, Fig. 5 elements 514 and ¶ 0064, “determining a reduced set of classification labels (e.g., reduced label set 216 in FIG. 2) by selecting classification labels with corresponding scores from the second set of non-conformity scores less than the non-conformity threshold.”
Kumar does not expressly disclose the following limitations, but they are taught by Peleg:
determining whether the class membership set consists of one class of the plurality of class outputs; See Peleg: ¶ 0033, “The classification tool may classify the images between two groups such as, for example, healthy or unhealthy, interesting or non-interesting, relevant or irrelevant (to a certain medical situation) etc.
automatically performing an action when the class membership set is determined to consist of one class of the plurality of class outputs; and See Peleg, ¶ 0033, “Such classifications may be presented to a user for example, on a monitor (e.g., monitor 20).”
when the class membership set is determined not to consist of one class, providing information about the data sample for manual review. See Peleg Fig. 8 along with ¶ 0064, “As indicated in block 830, a user input about a border case image from the image stream may be requested. … For example, a border case image may be one of the images for which the initial classification algorithm is least sure about its classification, namely the closest images to the classification border and/or images that are closer than a certain determined distance to the classification border, e.g. images which received uncertain classification scores by the initial classification algorithm, e.g. soft margins whose absolute value is relatively low. As indicated in block 840, the border case image may be classified according to the user inputs.”
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Peleg’s border case processing with Kumar’s classification in order to utilize a relevant training set for creation of an adapted classifier as suggested by Peleg (see ¶ 0064).
Regarding claim 2, Kumar does not expressly disclose:
2. The system of claim 1, wherein the information provided about the data sample for manual review includes the plurality of model class scores. This is taught by Peleg. See Peleg, Fig. 5 along with ¶ 0042, “In some embodiments of the present invention, the scores of the second image may render it proximate or close (in terms of the geometric distance on the X-Y grid) to the predetermined border, and a user may be asked for indication regarding classification of the second image in order to determine more accurately the border, for example the shape of the border in the surroundings of the point representing the second image.”
Regarding claim 4, Kumar also discloses:
4. The system of claim 1, wherein each class has an associated conformal threshold for determining membership of that class in the class membership set. Kumar, ¶ 0061, “At step 508, method 500 performs computing a non-conformity threshold based on the first set of non-conformity scores (e.g., scores 206 in FIG. 2) and a pre-defined error rate (e.g., rate a as described in relation to FIG. 2).”
Regarding claim 5, Kumar also discloses:
5. The system of claim 1, wherein the conformal threshold is based on an error rate. Kumar, ¶ 0061, “At step 508, method 500 performs computing a non-conformity threshold based on the first set of non-conformity scores (e.g., scores 206 in FIG. 2) and a pre-defined error rate (e.g., rate a as described in relation to FIG. 2).”
Regarding claim 6, Kumar also discloses:
6. The system of claim 1, wherein the instructions are further executable for: training the computer model with a training set; and determining the conformal threshold based on a calibration dataset and an error rate. Kumar, ¶ 0060-0061, “For another example, the non-conformity scores may be computed as negative of class logits generated by the base classification model in response to a specific text from the calibration dataset. … At step 508, method 500 performs computing a non-conformity threshold based on the first set of non-conformity scores (e.g., scores 206 in FIG. 2) and a pre-defined error rate (e.g., rate a as described in relation to FIG. 2).”
Regarding claim 7, Kumar also discloses:
7. The system of claim 6, wherein the instructions are further executable for determining a plurality of conformal thresholds based on a plurality of error rates associated with the plurality of classes. Kumar, ¶ 0027, “Thus, scores 212, s(xtest, y1), . . . , s(xtest, yK) may in turn be computed for the K actual target labels using the prediction ŷtest generated by the base classifier 220.”
Regarding claim 8, Kumar discloses:
8. A method for selective model intervention, comprising: See Kumar, Fig. 5, broadly depicting a method.
All further limitations of claim 8 have been addressed in the above rejection of claim 1.
Regarding claims 9 and 11-14:
Parent claim 8 is addressed above. All further limitations of claims 9 and 11-14 have been addressed in the above rejections of claims 2 and 4-7, respectively.
Regarding claim 15, Kumar discloses:
15. A non-transitory computer-readable medium for selective model intervention, the non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to: See Kumar, Fig. 3, depicting medium 320 with instructions for processor 310.
All further limitations of claim 15 have been addressed in the above rejection of claim 1.
Regarding claims 16 and 18-20:
Parent claim 15 is addressed above. All further limitations of claims 16 and 18-20 have been addressed in the above rejections of claims 2 and 4-6, respectively.
Claim(s) 3, 10 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kumar in view of Peleg as applied above, and further in view of “Fair conformal predictors for applications in medical imaging” by Lu et al. (“Lu”).
Regarding claim 3, Kumar does not expressly disclose:
3. The system of claim 1, wherein a conformal score for a class of the plurality of conformal scores is determined by accumulating the model class scores in a decreasing ranking of the plurality of conformal scores until an index of the class in the decreasing ranking [is reached]. However, Lu teaches this. See Lu, p. 12011, bottom left column: “The scoring function s sorts the classes according to their softmax score in descending order for each example and outputs the cumulative sum until the softmax score of the true class is reached.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Lu’s scoring function with Kumar’s conformal score in order to adapt better to differences between subgroups than aggregate conformal methods as suggested by Lu (see p. 12011, 1st paragraph under “Methods”).
Regarding claim 10:
Parent claim 8 is addressed above. All further limitations of claim 10 have been addressed in the above rejection of claim 3.
Regarding claim 17:
Parent claim 15 is addressed above. All further limitations of claim 17 have been addressed in the above rejection of claim 3.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
U.S. Patent 11501210 to Zhdanov et al. See col. 2, lines 49-61, e.g. “ If the ML models determine that the confidence score of a prediction is less than a defined confidence (e.g., threshold), the content (or a portion thereof) may be sent for human review.”
Any inquiry concerning this communication or earlier communications from the examiner should be directed to James D Rutten whose telephone number is (571)272-3703. The examiner can normally be reached M-F 9:00-5:30 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at (571)272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/James D. Rutten/Primary Examiner, Art Unit 2121
Prosecution Insights