DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 3-9. 11-13, 15 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhou (U.S. PG-PUB NO. 2022/0300769) in view of Lim (“Fast AutoAugment”, 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada).
-Regarding claims 1, 15 and 16, Zhou discloses an information processing apparatus that executes active learning by repeating image selection and retraining of a learning model with the selected images (the ACFT algorithm iteratively selects a set of candidates for labeling as illustrated in Algorithm 1 (FIG. 1E, element 195), [0060]; ACFT(HQ) denotes the optimized learning strategy, which continually fine-tunes the current CNN model Mt-1 using newly annotated candidates enlarged by those misclassified candidate, [0086]), the information processing apparatus comprising: at least one memory storing a program (memory 1195, [0135]); and at least one processor that (processor 1190, [0135]), upon execution of the program is configured to operate as: an acquisition unit configured to acquire a trained learning model (the ACFT begins with an empty training dataset and directly uses pre-trained CNNs (AlexNet and GoogLeNet) on ImageNet, [0086]); by executing image transformation based on the image transformation method candidate on an annotated image including an annotation representing ground truth information (To boost CNN performance for CAD systems, multiple patches are usually generated automatically for each candidate through data augmentation; those patches that are generated from the same candidate inherit the candidate's label, [0060]; Each resulting patch was further rotated eight (8) times by mirroring and flipping, [0080]), and a second selection unit configured to, for each of a plurality of unannotated images, generate a transformed unannotated image by executing the selected image transformation method on the unannotated image, provide the transformed unannotated image to the acquired trained learning model, calculated an uncertainty of an output result obtained from the acquired trained learning model for the transformed unannotated image, and-select, based on the calculated uncertainty, an unannotated image used to retrain the learning model (the described ACFT method starts directly with a pre-trained CNN to seek “salient” samples from the un-annotated pool for annotation, [0041]; all patches augmented from the same candidate (FIG. 2A) share the same label; therefore, they are expected to have similar predictions by the current CNN. As a result, their entropy and diversity provide a useful indicator of the “power” of a candidate for elevating the performance of the current CNN, [0071]; Suppose b number of candidates are required for annotation. Instead of selecting top b candidates, the method extends the candidate selection pool to ωb, [0076]).
Zhou is silent to teaching that a first selection unit configured to, for each of a plurality of image transformation method candidates, generate a transformed annotated image; provide the transformed annotated image to the acquired trained learning model, calculate an uncertainty of an output result obtained from the acquired trained learning model relative to the ground truth information, calculate a score for the image transformation method candidate based on the calculated uncertainty, and select an image transformation method from the plurality of image transformation method candidates based on the calculated scores. However, the claimed limitation is well known in the art as evidenced by Lim.
In the same field of endeavor, Lim teaches a first selection unit configured to, for each of a plurality of image transformation method candidates, generate a transformed annotated image (Note that each sub-policy ꚍ is a random sequence of image transformations which depend on p and λ, and this enables to cover a wide range of data augmentations, section 3.1, page 3; Since the evaluation of the model performance for every candidate policies is computationally expensive, we apply Bayesian optimization to the exploration of augmentation strategies, section 3.2.2, page 5; Shuffle (Line 1): We split training sets while preserving the percentage of samples for each class (stratified shuffling) using StratifiedShuffleSplit method in sklearn [27], section 3.3, page 5); provide the transformed annotated image to the acquired trained learning model (Next, we train model parameter θ on DM from scratch without data augmentation; Note that, during the policy exploration-and-exploitation procedure, the proposed algorithm does not train model parameter from scratch again, hence the proposed method find augmentation policies significantly faster than AutoAugment, section 3.2.1, page 5), calculate an uncertainty of an output result obtained from the acquired trained learning model relative to the ground truth information (For a given augmentation policy T , L(
θ
D
) denotes the expected loss of model for augmented images of data by (2), section 3.2, page 4; In practice, we minimize the categorical cross-entropy loss L(
θ
D
A
) instead of maximizing accuracy in (3), section 3.2.1, page 4), calculate a score for the image transformation method candidate based on the calculated uncertainty, and select an image transformation method from the plurality of image transformation method candidates based on the calculated scores (Therefore, we perform this evaluation by measuring how much one dataset follows the pattern of the other by making use of the model predictions on both datasets, section 3.2.1, page 4; As the algorithm completes the exploration step, we select top-N policies over B and denote them Tt collectively, section 3.2.1, page 5; Merge (Line 7-9): Select the top N best policies for each split and then combine the obtained policies from all splits, section 3.3, page 6).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to incorporate Lim’s selection of a transformation, by scoring candidate transformation with a trained model’s loss on labeled images, into Zhou’s active learning apparatus. Zhou identifies the choice of data augmentation as critical (Zhou, [0120]), and Lim supplies a known way to make that choice. In the combination, the transformation selected per Lim is the one Zhou executes on unannotated candidates before computing entropy. This is the use of a known technique to improve a similar system in the same way, with predicable results.
-Regarding claim 3, the combination further discloses execution of the stored program further configures the first selection unit to operate as a first calculation unit that provides, as input to the acquired learning model, a transformed image of the annotated image, the transformed image having been obtained by executing image transformation based on an image transformation method candidate and calculates an uncertainty of an output result obtained by the acquired learning model, and a second calculation unit that calculates the score for an individual image transformation method candidate based on an uncertainty calculated by the first calculation unit (Lim, In practice, we minimize the categorical cross-entropy loss L
θ
T
(
D
A
) instead of maximizing accuracy in (3), section 3.2.1, page 4; As the algorithm completes the exploration step, we select top-N policies over B and denote them Tt collectively, section 3.2.1, page 5).
-Regarding claim 4, the combination further discloses the first calculation unit calculates the uncertainty by comparing the output result and the ground truth information (Lim, For a given classification model M
⟨
∙
│
θ
⟩
) : X
→
Y that is parameterized by θ, the expected accuracy and the expected loss of model M
⟨
∙
│
θ
⟩
) on dataset D are denoted by R
θ
D
and L
θ
D
, respectively, section 3.2, page 4).
-Regarding claim 5, the combination further discloses the first selection unit selects an image transformation method whose score is high or is equal to or more than a threshold (Lim, Merge (Line 7-9): Select the top N best policies for each split and then combine the obtained policies from all splits, section 3.3, page 6; Ly in (4) denotes the constant threshold of loss value determined by the quantile of observations among previously explored policies, section 3.2.2, page 5).
-Regarding claim 6, the combination execution of the stored program further configures the second selection unit to select an image from unannotated images having no ground truth information added, and use the selected unannotated image an annotation addition target (Zhou, The described ACFT method starts directly with a pre-trained CNN to seek “salient” samples from the un-annotated pool for annotation, [0041]; block 1025, processing logic annotates each of the image candidates in the selected set of worthy image candidates with a label, [0127]).
-Regarding claim 7, the combination further discloses execution of the stored program further configures the second selection unit to include a third calculation unit that provides, as an input to the acquired learning model, a transformed image of the unannotated image, the transformed image having been obtained by executing image transformation based on a selected image transformation method and calculates an uncertainty of an output result obtained by the learning model, and wherein the second selection unit includes a fourth calculation unit that calculates a priority of the unannotated image based on the calculated uncertainty (Zhou, entropy captures classification certainty—a higher uncertainty value denotes a greater degree of information, and combining entropy and diversity yields as is defined at equation 2 [0071]; Candidates are then selected from this pool with their sampling probabilities ais to inject randomization, [0076]; Lim, since augmentation policy T
*
works as optimized inference-time augmentation [33, 34] to make the model robustly predict correct answers, section 3.2.1, page 5).
-Regarding claim 8, the combination further discloses the learning model is used to execute a task including at least one of image classification, object detection, and segmentation (Zhou, Image quality assessment in colonoscopy can be viewed as an image classification task whereby an input image is labeled as either informative or non-informative, [0079]).
-Regarding claim 9, the combination further discloses in a case where the output result includes a classification result, the first calculation unit calculates the uncertainty based on a probability distribution distance between a probability distribution obtained from the ground truth information and the classification result transformed into a probability distribution (Lim, In practice, we minimize the categorical cross-entropy loss L(
θ
D
A
) instead of maximizing accuracy in (3), section 3.2.1, page 4; Zhou, entropy captures classification certainty—a higher uncertainty value denotes a greater degree of information, and combining entropy and diversity yields as is defined at equation 2 [0071]).
-Regarding claim 11, the combination further discloses in a case where the output result includes both a classification result and location or area information, the second calculation unit calculates the score based on a combination of the uncertainty calculated based on the classification result and the uncertainty calculated based on the location or area or based on one of the uncertainties (Zhou, In this application, each polyp detection is regarded as a candidate, [0080]; A CAD system typically employs a candidate generator, which can quickly produce a set of candidates, among which some are true positives and others are false positives, [0060]; Lim, n practice, we minimize the categorical cross-entropy loss L(
θ
D
A
) instead of maximizing accuracy in (3), section 3.2.1, page 4).
-Regarding claim 12, the combination further discloses execution of the stored program further configures the first selection unit to select an image transformation method from candidates including at least one of geometrical transformation, color tone transformation, noise addition, blurring, and mosaic (Lim, the search space consists of the same 16 operations (ShearX, ShearY, TranslateX, TranslateY, Rotate, AutoContrast, Invert, Equalize, Solarize, Posterize, Contrast, Color, Brightness, Sharpness, Cutout, Sample Pairing), section 4, page 6; Zhou, Each resulting patch was further rotated eight (8) times by mirroring and flipping, [0080]).
-Regarding claim 13, the combination further discloses execution of the stored program further configures the at least one processor to operate as an update that updates the acquired learning model by retraining the acquired learning model using the selected image, wherein the acquisition unit acquires the updated learning model (Zhou, ACFT(HQ) denotes the optimized learning strategy, which continually fine-tunes the current CNN model Mt-1 using newly annotated candidates enlarged by those misclassified candidates; that is, Q
∪
H, [0086]; the ACFT algorithm iteratively selects a set of candidates for labeling as illustrated in Algorithm 1 (FIG. 1E, element 195), [0060]).
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhou (U.S. PG-PUB NO. 2022/0300769) in view of Lim (“Fast AutoAugment”, 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada) and further in view of Shiens (U.S. PG-PUB NO. 2019/0354817).
-Regarding claim 10, the combination of Zhou and Lim further discloses a detection task using bounding boxes (Zhou, in a case where the output result includes location information or area information, the first calculation unit calculates the uncertainty based on a degree of overlapping between a location or an area obtained from the ground truth information and a location or an area included in the output result, [0080]).
The combination is silent to teaching that in a case where the output result includes location information or area information, the first calculation unit calculates the uncertainty based on a degree of overlapping between a location or an area obtained from the ground truth information and a location or an area included in the output result. However, the claimed limitation is well known in the art as evidenced by Shiens.
In the same field of endeavor, Shiens teaches in a case where the output result includes location information or area information, the first calculation unit calculates the uncertainty based on a degree of overlapping between a location or an area obtained from the ground truth information and a location or an area included in the output result (the annotation for each image 12 can include a bounding shape, where a location of the bounding shape corresponds to the location of the target object within the image 12, [0030]; The bounding box is adjusted to maintain consistency with the applied augmentation, [0090]; A performance metric 20 (e.g., average precision, accuracy, latency, model data size, and/or various other measures of model performance) can be evaluated for the trained model 18, [0055]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of the combination with the teaching of Shiens. Lim’s evaluation of candidate transformation would then extend to Zhou’s poly-detection task using the standard measure of a predicted location against a ground-truth location.
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhou (U.S. PG-PUB NO. 2022/0300769) in view of Lim (“Fast AutoAugment”, 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada) and further in view of Park (U.S. PG-PUB NO. 2022/0114441).
-Regarding claim 14, Zhou discloses an information processing apparatus that executes active learning by repeating image selection and retraining of a learning model with the selected images (the ACFT algorithm iteratively selects a set of candidates for labeling as illustrated in Algorithm 1 (FIG. 1E, element 195), [0060]; ACFT(HQ) denotes the optimized learning strategy, which continually fine-tunes the current CNN model Mt-1 using newly annotated candidates enlarged by those misclassified candidate, [0086]), the information processing apparatus comprising: at least one memory storing a program (memory 1195, [0135]); and at least one processor (processor 1190, [0135]) that, upon execution of the program is configured to operate as: an acquisition unit configured to acquire a trained learning model (the ACFT begins with an empty training dataset and directly uses pre-trained CNNs (AlexNet and GoogLeNet) on ImageNet, [0086]); and a selection unit configured to, for each of a plurality of unannotated images, generate a transformed unannotated image by executing the selected image transformation method on the unannotated image, provide the transformed unannotated image to the acquired trained learning model, calculate an uncertainty of an output result obtained from the acquired trained learning model for the transformed unannotated image, and select, based on the calculated uncertainty, an unannotated image used to retrain the learning model (the described ACFT method starts directly with a pre-trained CNN to seek “salient” samples from the un-annotated pool for annotation, [0041]; all patches augmented from the same candidate (FIG. 2A) share the same label; therefore, they are expected to have similar predictions by the current CNN. As a result, their entropy and diversity provide a useful indicator of the “power” of a candidate for elevating the performance of the current CNN, [0071]; Suppose b number of candidates are required for annotation. Instead of selecting top b candidates, the method extends the candidate selection pool to ωb, [0076]).
Zhou is silent to teaching that generating a transformed annotated image by executing image transformation based on the image transformation method candidate on an annotated image including an annotation representing ground truth information, providing the transformed annotated image to the acquired trained learning model, calculating an uncertainty of an output result obtained from the acquired trained learning model relative to the ground truth information, calculating a score for the image transformation method candidate based on the calculated uncertainty, and selecting an image transformation method from the plurality of image transformation method candidates based on the calculated scores. However, the claimed limitation is well known in the art as evidenced by Lim.
In the same field of endeavor, Lim teaches generating a transformed annotated image by executing image transformation based on the image transformation method candidate on an annotated image including an annotation representing ground truth information (Note that each sub-policy ꚍ is a random sequence of image transformations which depend on p and λ, and this enables to cover a wide range of data augmentations, section 3.1, page 3; Since the evaluation of the model performance for every candidate policies is computationally expensive, we apply Bayesian optimization to the exploration of augmentation strategies, section 3.2.2, page 5; Shuffle (Line 1): We split training sets while preserving the percentage of samples for each class (stratified shuffling) using StratifiedShuffleSplit method in sklearn [27], section 3.3, page 5), providing the transformed annotated image to the acquired trained learning model (Next, we train model parameter θ on DM from scratch without data augmentation; Note that, during the policy exploration-and-exploitation procedure, the proposed algorithm does not train model parameter from scratch again, hence the proposed method find augmentation policies significantly faster than AutoAugment, section 3.2.1, page 5), calculating an uncertainty of an output result obtained from the acquired trained learning model relative to the ground truth information (For a given augmentation policy T , L(
θ
D
) denotes the expected loss of model for augmented images of data by (2), section 3.2, page 4; In practice, we minimize the categorical cross-entropy loss L(
θ
D
A
) instead of maximizing accuracy in (3), section 3.2.1, page 4), calculating a score for the image transformation method candidate based on the calculated uncertainty, and selecting an image transformation method from the plurality of image transformation method candidates based on the calculated scores (Therefore, we perform this evaluation by measuring how much one dataset follows the pattern of the other by making use of the model predictions on both datasets, section 3.2.1, page 4; As the algorithm completes the exploration step, we select top-N policies over B and denote them Tt collectively, section 3.2.1, page 5; Merge (Line 7-9): Select the top N best policies for each split and then combine the obtained policies from all splits, section 3.3, page 6).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to incorporate Lim’s selection of a transformation, by scoring candidate transformation with a trained model’s loss on labeled images, into Zhou’s active learning apparatus. Zhou identifies the choice of data augmentation as critical (Zhou, [0120]), and Lim supplies a known way to make that choice. In the combination, the transformation selected per Lim is the one Zhou executes on unannotated candidates before computing entropy. This is the use of a known technique to improve a similar system in the same way, with predicable results.
The combination of Zhou and Lim is silent to teaching that a setting unit configured to set an image transformation method, from a plurality of image transformation method candidates, for an image by for each of a plurality of image transformation method candidates, and wherein the setting unit changes a currently set image transformation method, depending on progress of training of the learning model acquired by the acquisition unit. However, the claimed limitation is well known in the art as evidenced by Park.
In the same field of endeavor, Park teaches a setting unit configured to set an image transformation method, from a plurality of image transformation method candidates, for an image by for each of a plurality of image transformation method candidates, and wherein the setting unit changes a currently set image transformation method, depending on progress of training of the learning model acquired by the acquisition unit (The technique user 1451 may compare the verification accuracy of each of the plurality of second neural network models for each preset unit of training, and determine the data augmentation technique applied to the model having the maximum verification accuracy as an optimal data augmentation technique to be used for the remaining models during the next training, [0098]; In this case, the updater 145 updates the data augmentation technique between the plurality of second neural network models 710 for each epoch during the training process, [0133]; the schedule determinator 150 may provide the determined optimal schedule during training of the main model, [0076]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of the combination with the teaching of Park. Lim’s evaluation of candidates would then be repeated as Zhou’s model is fine-tuned, keeping the transformation suited to the model’s current state. Zhou observes that prediction patterns change between early and late stages of training [0099], which yields predictable results.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1 and 3-16 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PING Y HSIEH whose telephone number is (571)270-3011. The examiner can normally be reached Monday-Friday, 9am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PING Y HSIEH/ Primary Examiner, Art Unit 2664