DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/12/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered and attached by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 5-8, 10, 11, 14-17, 19, 20, and 23-26 are rejected under 35 U.S.C. 103 as being unpatentable over G. Han and S. -N. Lim, "Few-Shot Object Detection with Foundation Models," 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2024, pp. 28608-28618 hereinafter referred to as (Han) in view of Huang (U.S. Patent Pub. No. 2024/0071056).
Regarding Claim 1, Han teaches a method for outputting a fine-tuned model for few-shot object detection, the method comprising (Figure 1. Overview of the proposed model. The diagram shows the high-level framework of our model for FSOD:)
creating a dataset of K-shot classes for fine-tuning a pretrained vision language model (VLM) (Section 3.1 For K-shot (e.g., K = 1, 5,10) object detection, we have exactly K bounding box annotations for each novel class c ∈ Cnovel as the training data; Section 3.2: We propose to study FSOD with foundation models in this work. The idea is to make full use of the knowledge in the pre-trained large-scale vision/language foundation models for downstream few-shot learning tasks, and simplify the human efforts for model design)
performing concept alignment between the dataset of K-shot classes and the VLM (Section 3.2 As shown in Figure 2, our model mainly consists of the following three submodules: (1) Visual Feature Extraction to extract feature representations for both query images and few-shot support images, (2) Proposal Generation to generate support-class-aware object regions from the query image, and (3) Few-Shot Proposal Classification to classify each of the proposal given the mapping of the categories and their visual prototypes;)
fine-tuning the VLM using the dataset of K-shot classes with a pseudo-negative federated loss to generate a few-shot object detection (FSOD) model; and (See entire Section 3.2; Section 3.2 Few-Shot Proposal Classification: We follow the original loss functions defined in DETR [4] by first finding the optimal bipartite matching between the predicted objects set and ground truth objects set, and then optimizing the model towards this optimal assignment; Then, we can train our LLM end-to-end using the next-token pre diction loss, calculated over the ground-truth proposal labels. In this step, the proposal generation module is also fine-tuned with the DETR loss.)
outputting the FSOD model for use in object detection of the K-shot classes in image data received from one or more sensors (It would be obvious to one with ordinary skill in the art that this updated model would then be used as the FSOD, but Han does not explicitly disclose that. See Huang below for a model updated with pseudo-negative loss and output for detection on images from a sensor.)
Han does not explicitly disclose fine-tuning a model using the dataset of classes with a pseudo-negative federated loss to generate a model; and
outputting a model for use in object detection of the classes in image data received from one or more sensors.
Huang is in the same field of art of image analysis. Further, Huang teaches fine-tuning a model using the dataset of classes with a pseudo-negative federated loss to generate a model; and (¶55 At step 410, the multiple weakly augmented photograph class predictions are converted into positive pseudo-labels or negative pseudo-labels. In this regard, a first class prediction, of the weakly augmented photograph class predictions, is converted to the positive pseudo-label when the probability for the first class prediction is above a positive score threshold; ¶57 At step 414, the model is trained to make the strongly augmented photograph label prediction match the positive pseudo-label via a cross-entropy loss. Such training is based on the positive pseudo-labels and the negative pseudo-labels.)
outputting a model for use in object detection of the classes in image data received from one or more sensors (Fig. 5; Abstract: The trained model is then utilized to label the unlabeled photograph with multiple labels.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Han by generating a pseudo negative loss to update a model and then using that model that is taught by Huang; thus, one of ordinary skilled in the art would be motivated to combine the references to improve labeling efficiency (Huang ¶21).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 2, Han in view of Huang discloses the method of claim 1, wherein creating the dataset of K-shot classes includes:
selecting an image associated with one of a set of target classes; and adding the image to the dataset of K-shot classes if a total count of annotations for the target class in the image are less than or equal to K, until K annotations per target class of the set of target classes are added to the dataset of K-shot classes (Han, Section 3.1: For K-shot (e.g., K = 1, 5,10) object detection, we have exactly K bounding box annotations for each novel class c ∈ Cnovel as the training data.)
Regarding Claim 5, Han in view of Huang discloses the method of claim 1, wherein computing the pseudo-negative federated loss includes:
generating pseudo-positive predictions for each image in the dataset of K-shot classes; filtering the pseudo-positive predictions by confidence threshold to identify pseudo-positive classes; and (Huang, ¶55 a first class prediction, of the weakly augmented photograph class predictions, is converted to the positive pseudo-label when the probability for the first class prediction is above a positive score threshold)
identifying pseudo-negative classes by determining classes not included in the pseudo-positive predictions (Huang, ¶55 A second class prediction, of the weakly augmented photograph class predictions, is converted to the negative pseudo-label when the probability for the second class prediction is below a negative score threshold. Such a negative score threshold is also a fixed percentile threshold that unifies fixed and variable score thresholds. In addition, the negative pseudo-label indicates an absence of a class.)
The reasons for combining Han and Huang are similar to that stated in the rejection of claim 1. In addition, this same reasoning is pertinent and applicable to the rejections of claim 6 below.
Regarding Claim 6, Han in view of Huang discloses the method of claim 5, wherein computing the pseudo-negative federated loss further includes:
combining the pseudo-negative classes with ground truth classes to form a set of selected classes (Huang, ¶55 In one or more embodiments, the fixed percentile thresholds are based on a ground truth ratio in labeled data that avoids having an incorrectly high ratio of the positive pseudo-labels;)
iterating over the selected classes to compute a binary cross-entropy (BCE) loss for each class by comparing FSOD model predictions with ground truth annotations; and summing the computed losses to obtain a total pseudo-negative federated loss (Huang, ¶57 At step 414, the model is trained to make the strongly augmented photograph label prediction match the positive pseudo-label via a cross-entropy loss. Such training is based on the positive pseudo-labels and the negative pseudo-labels; ¶59 a total loss for converting to the positive pseudo-label may be determined as a weighted combination of supervised learning and the unlabeled loss weight. Such a total loss may then be used during the training. Further, the use of the unlabeled weight loss may improve performance due to an increasing gap between the positive score threshold (which increases) and the negative score threshold (which decreases) during/as the training progresses.)
Regarding Claim 7, Han in view of Huang discloses the method of claim 6, further comprising determining the fine-tuning has converged based on stability of the total pseudo-negative federated loss and/or performance of the FSOD model on the object detection of the K-shot classes (Han, Section 4: (4)Using LLM as few-shot learner can further improve the performance, compared with the Deformable DETR only model. This verifies the effectiveness of our model by introducing additional context information and prior knowledge for few-shot learning; We further ablate the importance of our contextualized few-shot learning in Table5. We show the experiments of using LLM to classify for each proposal separately, and using LLM to classify for each class separately. The performance decreases slightly in the two models. More importantly, the running speed of the two models is much slower. This shows the effectiveness of our contextual modeling.)
Regarding Claim 8, Han in view of Huang discloses the method of claim 1, wherein the pretrained VLM comprises a Detic segmentation model or a Contrastive Language-Image Pretraining (CLIP) model trained on large-scale multi-modal data (Han, Section 1: we comprehensively evaluate multiple pre-trained vision foundation models, including MAE [19], CLIP [40], SAM [24] and DINOv2 [37], and with different detection architectures, RCNN-based framework ViTDet [28] and Transformer-based framework Deformable DETR [71].)
Regarding claim 10, claim 10 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Huang further teaching on: A system for outputting a fine-tuned model for few-shot object detection, the system comprising: one or more processors including instructions installed to one or more memories (Han, ¶71 the computer 502 operates by the hardware processor 504A performing instructions defined by the computer program 510 (e.g., a computer-aided design [CAD] application) under control of an operating system 508.)
Claim 11 recites limitations similar to claim 2 and is rejected under the same rationale and reasoning.
Claim 14 recites limitations similar to claim 5 and is rejected under the same rationale and reasoning.
Claim 15 recites limitations similar to claim 6 and is rejected under the same rationale and reasoning.
Claim 16 recites limitations similar to claim 7 and is rejected under the same rationale and reasoning.
Claim 17 recites limitations similar to claim 8 and is rejected under the same rationale and reasoning.
Regarding claim 19, claim 19 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Huang further teaching on: A non-transitory computer-readable medium comprising instructions for providing a fine-tuned model for few-shot object detection that, when executed by one or more processors, cause the one or more processors to perform operations (¶77 In one embodiment, instructions implementing the operating system 508, the computer program 510, and the compiler 512 are tangibly embodied in a non-transitory computer-readable medium.)
Claim 20 recites limitations similar to claim 2 and is rejected under the same rationale and reasoning.
Claim 23 recites limitations similar to claim 5 and is rejected under the same rationale and reasoning.
Claim 24 recites limitations similar to claim 6 and is rejected under the same rationale and reasoning.
Claim 25 recites limitations similar to claim 7 and is rejected under the same rationale and reasoning.
Claim 26 recites limitations similar to claim 8 and is rejected under the same rationale and reasoning.
Claims 3-4, 12-13, and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over (Han) in view of Huang (U.S. Patent Pub. No. 2024/0071056) in view of Volpi (U.S. Patent Pub. No. 2025/0356625).
Regarding Claim 3, Han in view of Huang teaches the method of claim 2, wherein performing the concept alignment includes:
compiling multimodal annotations for each target class, the multimodal annotations including textual descriptions and visual examples of the target class; and (Han, Section 3.1: For K-shot (e.g., K = 1, 5,10) object detection, we have exactly K bounding box annotations for each novel class c ∈ Cnovel as the training data; We use RoIAlign [18] to calculate the representation of the object given the bounding box annotation of the object; Section 4.2: We have two projection layers in our model: one connecting proposals to the LLM and the other connecting class prototype to the LLM.)
Han in view of Huang does not explicitly disclose augmenting the textual descriptions with synonyms generated by querying a large language model (LLM) for descriptions of bounding box regions in the images of the target class.
Volpi is in the same field of art of image analysis. Further, Volpi teaches augmenting the textual descriptions with synonyms generated by querying a large language model (LLM) for descriptions of bounding box regions in the images of the target class (¶152 an LLM may be instantiated by embedding a task instruction and the vocabulary C, enriched with synonyms, into its system prompt. These synonyms, queried from an LLM in advance, help the system recognize classes phrased in different ways, such as “TV” vs. “Television”. This may improve selection quality.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Han in view of Huang by augmenting the textual descriptions using synonyms that is taught by Volpi; thus, one of ordinary skilled in the art would be motivated to combine the references to improve selection quality (Volpi ¶152).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 4, Han in view of Huang in view of Volpi discloses the method of claim 3, wherein the multimodal annotations include materials used by human annotators for annotating images in the image set from which the K-shot classes are selected (Han, Section 3.2: Specifically, we first add several class tokens (e.g., from <class_1> to class <class_80> in MSCOCO dataset) ,and the background class token to the LLM tokenizer. Then we design the following language instructions for the LLM to perform classification for each of the proposals.) (Volpi, ¶157 Including synonyms for class names in the user-defined vocabulary when prompting the LLM-based example further improves performance.)
Claim 12 recites limitations similar to claim 3 and is rejected under the same rationale and reasoning.
Claim 13 recites limitations similar to claim 4 and is rejected under the same rationale and reasoning.
Claim 21 recites limitations similar to claim 3 and is rejected under the same rationale and reasoning.
Claim 22 recites limitations similar to claim 4 and is rejected under the same rationale and reasoning.
Claims 9, 18, and 27 are rejected under 35 U.S.C. 103 as being unpatentable over (Han) in view of Huang (U.S. Patent Pub. No. 2024/0071056) in view of Byravan (U.S. Patent Pub. No. 2026/0097491).
Regarding Claim 9, Han in view of Huang teaches the method of claim 1.
Han in view of Huang does not explicitly disclose capturing pixel data using one or more sensors of a robot;
applying the pixel data as input to the FSOD model to perform the object detection of the K-shot classes; and
controlling one or more actuators of the robot based on a result of the object detection.
Byravan is in the same field of art of image analysis. Further, Byravan teaches capturing pixel data using one or more sensors of a robot (¶24 the system 100 receives an input observation 110 that includes an image captured by a camera of the robot 104 and causes the robot 104 to perform an action from a set of actions;)
applying the pixel data as input to the FSOD model to perform the object detection of the K-shot classes; and (¶58 the system 190 can train the policy neural network 120 in simulation and then further train the policy neural network 120 while controlling the robot 104 in the environment 106, thereby performing few-shot transfer from simulation to the real-world)
controlling one or more actuators of the robot based on a result of the object detection (¶45 The system 100 can then control the robot 104 by providing the action 108 defined by the policy output 122 as a control input for the robot 104.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Han in view of Huang by implementing the method using a robot that is taught by Byravan; thus, one of ordinary skilled in the art would be motivated to combine the references to successfully complete tasks in the real world (Byravan ¶10).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 18 recites limitations similar to claim 9 and is rejected under the same rationale and reasoning.
Claim 27 recites limitations similar to claim 9 and is rejected under the same rationale and reasoning.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUSTIN BILODEAU whose telephone number is (571)272-1032. The examiner can normally be reached 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUSTIN BILODEAU/Examiner, Art Unit 2664
Prosecution Insights