Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 5, 12 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Below is a claim-by-claim analysis.
Claim 5, 12
Step 1: Recites a method (claim 5) and an apparatus (claim 12). Therefore, it is directed to a statutory category of invention.
Step 2A Prong 1: The claim recites:
recognize an image
Recognition of an image can be broadly interpreted as a generic mental process
classify the recognized image into one of a plurality of categories
Classification of an image can be broadly interpreted as a generic mental process
Step 2A Prong 2: The judicial exception is not integrated into a practical application. The remaining
limitations of the claim are directed to insignificant extra-solution activity (“receiving… a request to generate…”, “receiving information associated…”, “generating… the plurality of adversarial examples…1” “sending … the plurality of adversarial examples…”).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis mirrors
the analysis of step 2A prong 2.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5, 8-10, 12, 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Nie et al. (“Adversarial Confidence Learning for Medical Image Segmentation and Synthesis”, 2020), in view of Walters et al. (“US 10540798”, 2019).
Regarding claim 1, Nie teaches generating a plurality of adversarial examples for a first machine learning model2 (Page 2498, Figure 13, Page 2502, Column 2, Paragraphs 2-3, “To evaluate the proposed method, we apply our algorithm on two different datasets. The first dataset is our own pelvic dataset… The pelvic dataset consists of 50 prostate cancer patients from a Cancer Hospital, each with one T2-weighted MR image and its corresponding manually-labeled map by a medical expert.”4), wherein the first machine learning model is configured to output a classification for an input (Page 2498, Figure 25), and wherein the plurality of adversarial examples are configured to be input to the first machine learning model (Page 2498, Figure 1) and cause misclassification by the first machine learning model (Page 2501, Column 2, Paragraph 7, “At the same time, the generated image will be constrained to be as realistic as possible so that it can fool the discriminator”);
generating… and using a second machine learning model6 (Page 2498, Figure 1), the plurality of adversarial examples, wherein each of the plurality of adversarial examples is modified from a ground truth example (Page 2501, Column 1, Equation 12 and Paragraph 6, “…LG(X,Y) = ∥ Y −G(X)∥ p … G(X) is the generated target image from the source image X by the Generator network G7);
and sending, to the first machine learning model, the plurality of adversarial examples, wherein the first machine learning model is configured to be adjusted (Page 2498, Column 1, Paragraph 1, “…introduces our proposed adversarial confidence learning framework which retains the adversarial learning and imposes confidence learning to enhance the supervised generator”): based on a comparison between:
a respective output classification for each of the plurality of adversarial examples;
and data indicating a correct classification for each of the plurality of adversarial examples (Page 2498, Figure 28)
Nie fails to teach receiving, by a computing device, a request, and receiving information associated with the first machine learning model. However, Walters teaches receiving, by a computing device, a request to generate… (Column 6, Lines 30-32, “a customer, represented herein by a customer device 1040… request generation”) and receiving information associated with [a]9 first machine learning model (Column 14, Lines 54-56, “In further embodiments, the model selector 2032 may select a random number of the models 2010 at random based on parameters provided by a user or parameters related to the training of the models”10)
Nie and Walters are considered analogous to the invention because all are directed towards system of adversarial training. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified Nie to incorporate the teachings of Walters, and included a means of receiving a request for generating adversarial samples, as well as receiving information associated with the model. Doing so allows the adversarial generation to be implemented within a larger system, and receiving information associated with the model can improve the training process.
Regarding claim 2, Nie teaches receiving, from the first machine learning model, the respective output classification for each of the plurality of adversarial examples (Page 2498, Figure 2, Page 2498, Column 2, Paragraph 2, “Given a labeled input image X ∈ RH×W×T with corresponding ground-truth output map (segmentation or output modality) Y∈ ZH×W×T. For segmentation map, we encode it to one hot format P ∈ RH×W×T×C (by converting the label map Y into C binary label maps with one-hot encoding), where C is the number of semantic categories in the dataset.”11); and retraining, based on adversarial examples misclassified by the first machine learning model, the second machine learning model (Page 2499, Equations 1-312)
Regarding claim 3, Nie teaches wherein the first machine learning model is further configured to output a respective confidence level associated with each output classification (Page 2498, Caption of Figure 2, “0 means the lowest confidence for the prediction of voxel (the predicted category for the voxel is not consistent with the ground-truth category at all) and 1 means the highest confidence for the prediction of the voxel (the predicted category is fully consistent with the ground-truth category)”), and wherein the retraining the second machine learning model is further based on a respective confidence level corresponding to each of the output classification associated with the plurality of adversarial examples (Page 2498, Caption of Figure 2, “we pursue a perfect D so that we can obtain the confidence map (M) to guide the supervised training of the S, which means we can inject confidence learning besides the adversarial learning”)
Regarding claim 5, Nie teaches wherein the first machine learning model is configured to: recognize an image13; and classify the recognized image into one of a plurality of categories (see claim 2 analysis).
Claim 8 is a system claim corresponding to method claim 1 and is rejected for the same reasons as given in the rejection of that claim.
Claim 9 is a system claim corresponding to method claim 2 and is rejected for the same reasons as given in the rejection of that claim.
Claim 10 is a system claim corresponding to method claim 3 and is rejected for the same reasons as given in the rejection of that claim.
Claim 12 is a system claim corresponding to method claim 5 and is rejected for the same reasons as given in the rejection of that claim.
Claim 15 is a non-transitory computer readable medium claim corresponding to method claim 1 and is rejected for the same reasons as given in the rejection of that claim.
Claim 16 is a non-transitory computer readable medium claim corresponding to method claim 2 and is rejected for the same reasons as given in the rejection of that claim.
Claim 17 is a non-transitory computer readable medium claim corresponding to method claim 3 and is rejected for the same reasons as given in the rejection of that claim.
Claims 4, 11, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Nie et al. (“Adversarial Confidence Learning for Medical Image Segmentation and Synthesis”, 2020), in view of Walters et al. (“US 10540798”, 2019) and in further view of Merling et al. (“US 12222909”, 2023).
Regarding claim 4, Nie fails to teach the further limitations of the claim. However, Merling teaches receiving information associated with [a]14 first machine learning model that comprises metadata associated with input fields of a first machine learning model (Paragraph 7, “ the system may receive user data, wherein the user data comprises a first data structure with a plurality of metadata types… system may select, based on the first metadata type, a first model of a plurality of models for extracting data from the first subset”).
Nie and Merling are considered analogous to the invention because all are directed towards system of machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified Nie to incorporate the teachings of Merling, and included a means for receiving metadata associated with input fields of the first machine learning model. Doing so provides the system with an additional source of information to focus the training of adversarial samples and understand the context of the images being used.
Claim 11 is a system claim corresponding to method claim 4 and is rejected for the same reasons as given in the rejection of that claim.
Claim 18 is a non-transitory computer readable medium claim corresponding to method claim 4 and is rejected for the same reasons as given in the rejection of that claim.
Claims 6, 13, 19 are rejected under 35 U.S.C. 103 as being unpatentable over Nie et al. (“Adversarial Confidence Learning for Medical Image Segmentation and Synthesis”, 2020), in view of Walters et al. (“US 10540798”, 2019) and in further view of Waghela et al. (“A Modified Word Saliency-Based Adversarial Attack on Text Classification Models”, 2024).
Regarding claim 6, Nie fails to teach the further limitations of the claim. However, Waghela teaches determining one or more data points that are assigned, by [a]15 first machine learning model, a weight exceeding a threshold (Page 4, Paragraph 6, “Computation of word saliency: Before initiating replacements, the saliency of each word in the input text is computed. The saliency metric reflects the contribution of each word to the model’s prediction or output”); and modifying a portion, in each of a plurality of ground truth examples, that corresponds to the one or more data points (Page 9, Tables 5-7).
Nie and Waghela are considered analogous to the invention because all are directed towards adversarial attack systems. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified Nie to incorporate the teachings of Waghela, and created adversarial examples by modifying higher weighted portions of the input more. Doing so can produce more robust adversarial examples that are better at fooling the discriminator (see Page 9, Tables 5-7 of Waghela).
Claim 13 is a system claim corresponding to method claim 6 and is rejected for the same reasons as given in the rejection of that claim.
Claim 19 is a non-transitory computer readable medium claim corresponding to method claim 6 and is rejected for the same reasons as given in the rejection of that claim.
Claims 7, 14, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Nie et al. (“Adversarial Confidence Learning for Medical Image Segmentation and Synthesis”, 2020), in view of Walters et al. (“US 10540798”, 2019) and in further view of Putman et al. (“US 20210103654”).
Regarding claim 7, Nie teaches receiving, from the first machine learning model, a plurality of confidence levels, each associated with a respective classification for a corresponding second input of a plurality of second inputs (see claim 1-3 analysis16), as well as the generation of adversarial examples (see claim 1 analysis). Nie fails to teach triggering, based on the plurality of confidence levels satisfying a threshold, the request to generate the plurality of adversarial examples. However, Putman teaches a system wherein based on… [a confidence level]17 satisfying a threshold, a request to generate… (Paragraph 83, “Whereas if the anomalous activity has a confidence level that corresponds to a lower interval, an alert protocol can trigger a more moderate action like generating a report”).
Nie and Putman are considered analogous to the invention because all are directed towards systems of machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified Nie to incorporate the teachings of Putman, and included a mechanism for triggering generation based on satisfying a confidence threshold. Doing so automates the system and reduces unnecessary computations.
Claim 14 is a system claim corresponding to method claim 7 and is rejected for the same reasons as given in the rejection of that claim.
Claim 20 is a non-transitory computer readable medium claim corresponding to method claim 7 and is rejected for the same reasons as given in the rejection of that claim.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEWOS MESFIN whose telephone number is (571)270-0782. The examiner can normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571) 272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEWOS MESFIN/Examiner, Art Unit 2145
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145
1 Can be broadly interpreted as a step of data gathering, to be processed by further limitations
2 We understand the first model as being a “discriminator” in a generative adversarial network
3 The adversarial example being ŷ, being fed into D (the discriminator).
4 This specific reference teaches multiple images as inputs, which, in combination with the figure teaches a plurality of adversarial examples
5 The predicted mask in the figure is an output classification
6 We understand the second model as being a “generator” in a generative adversarial network
7 A function output where the ground truth is the input can be considered a modification
8 The predicted mask (output classification) and real mask (correct classification) are input to the confidence network (a form of discriminator) which then outputs confidence levels that help to improve the segmentation network (a form of generator)
9 Nie teaches the first machine learning model (i.e. a discriminator). In combination with what is taught in Walters the whole limitation is covered
10 Can broadly be interpreted as information relating to a first machine learning model (of many machine learning models)
11 The paragraph and the figure show that the first model (segmentation/generator model) outputs a label in the form of a one-hot encoding.
12 Showcases the different loss functions (including cross entropy loss, which calculates misclassification) that are used to update the generator model. This runs for multiple epochs (i.e. retrains) as shown by the caption for Figure 6 on Page 2505
13 Recognition can be broadly interpreted as the ability to process it and create meaningful output. This is apparent in the system framework show in figure 2
14 Nie teaches the first machine learning model (i.e. a discriminator). In combination with what is taught in Merling the whole limitation is covered
15 Nie teaches the first machine learning model (i.e. a discriminator). In combination with what is taught in Waghela the whole limitation is covered
16 There is no specific citation that states this. The reference teaches an adversarial training system with confidence outputs that are used to update a generator as it creates more realistic adversarial images. Thus, it implies that this confidence levels would be output as well.
17 Putman teaches a singular confidence level. The plurality of confidence levels is taught by Nie.
Prosecution Insights