DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments and amendments in the Amendment filed December 9, 2025 (herein “Amendment”), with respect to the rejection of claims 1–20 under 35 U.S.C. 101 for being directed towards an abstract idea without a practical application or significantly more have been fully considered and are persuasive. The rejection of claims 1–20 under 35 U.S.C. 101 has been withdrawn.
Applicant’s terminal disclaimer, filed along with the Amendment, to obviate the double patenting rejections against claims 1, 4–8, 11–15 and 17–20 has been approved. The double patenting rejections against claims 1, 4–8, 11–15 and 17–20 have been withdrawn.
Applicant’s arguments and amendments in the Amendment with respect to the rejection(s) of claim(s) 1–20 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Saito et al., "Strong-Weak Distribution Alignment for Adaptive Object Detection," 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp. 6949-6958, doi: 10.1109/CVPR.2019.00712, and in further view of Mohr et al., US Patent Application Publication No. US 2023/0306747 A1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1–4, 6, 8–11, 13, 15–17 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Lin et al., US Patent No. 10,755,099 B2 (herein “Lin”), in view of Rohrbach et al., "Translating Video Content to Natural Language Descriptions," 2013 IEEE International Conference on Computer Vision, Sydney, NSW, Australia, 2013, pp. 433-440, doi: 10.1109/ICCV.2013.61 (herein “Rohrbach”) in view of Saito et al., "Strong-Weak Distribution Alignment for Adaptive Object Detection," 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp. 6949-6958, doi: 10.1109/CVPR.2019.00712 (herein “Saito”), and Mohr et al., US Patent Application Publication No. US 2023/0306747 A1 (herein “Mohr”).
Regarding claims 1, 8 and 15, with substantive differences between the claims noted in square brackets {}, with claim 1 as illustrative, and with deficiencies of Lin noted in square brackets [], Lin teaches {A computer-implemented method for training a neural network to predict object categories without manual annotation, the method comprising: - claim 1/ A computer program product for training a neural network to predict object categories without manual annotation, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising: - claim 8 / A computer processing system for training a neural network to predict object categories without manual annotation, comprising: a memory device for storing program code; and a processor device, operatively coupled to the memory device, for running the program code to: - claim 15 (Lin col. 1, l. 50–col. 2, l. 17, techniques, systems and devices for an object detector trained to detect unseen object categories (thus without any kind of annotation, never mind human annotation), where col. 29, ll. 23–41 and 58–60, and col. 30, ll. 18–3, and col. 31, ll. 30–54 teach systems including processors to perform the disclosed techniques, memory and computer readable media storing computer readable instructions)} feeding training datasets including at least images and data annotations to an object detection neural network (Lin col. 20, ll. 33–36, and 55–57, col. 21, ll. 4–11, and col. 22, ll. 1–4, image module obtains images from a database of training datasets that can be used to train an adaptive model of an object detector neural network, and the input condition module obtains an input condition as metadata of an image such as annotations describing objects in an image, the annotations being input to the detection module downstream as embeddings);
converting, by a text prompter, the data annotations into natural text inputs (Lin col. 21, ll. 4–18, input condition module which obtains an input condition such as text typed in from a user (thus being a text prompter) also processes the input condition of extracted metadata from an image such as annotations describing an object, to have corrected spelling and corrected grammar, thus converting to natural text input);
converting, by a text embedder, the natural text inputs into embeddings (Lin col. 21, ll. 45–51, a word based concept is received by the word embedding module and a word embedding is generated therefrom);
minimizing objective functions, including a localization loss, detection classification loss (Lin col. 16, ll. 40–62, training module trains any suitable network according to any suitable loss function, giving an example of a binary classifier assigning labels to
detection outputs (detection classification loss) of the conditional detection network that substantially overlap with a ground truth bounding box (localization loss) that corresponds to a word-based concept) [weak alignment loss, and a global alignment loss, that utilize the embeddings based on a determined alignment] for summarizing the images and data annotations during training to adjust parameters of the object detection neural network (Lin col. 22, ll. 28–35, training module generates suitable training updates such as weights of (parameters) neural networks updated by stochastic gradient descent that minimizes any suitable loss function, where col. 16, ll. 40–62 teach that the binary loss used in the training objection functions provides word-based concept labels (data annotations) corresponding to the image (summarizing the images)); and
[generating control instructions for an autonomous] vehicle (Lin col. 29, ll. 23–54, fig. 10, computing devices implementing the disclosed object detection in images including a vehicle 1002-5) [to avoid a collision by] predicting, by the object detection neural network, objects within images [and videos] [of a traffic scene that includes the autonomous vehicle] (Lin col. 26, ll. 17–28, col. 27, ll. 1–19 and 39–43, an image is obtained and the conditional detection network of the detection module detects a region where an object is present).
While Lin teaches predicting objects in an image, and that the image can come from a sequence of video images (see col. 9, ll. 36–39) Lin does not explicitly teach predicting objects within videos.
Further, while Lin teaches objective functions at least including a localization loss and detection classification loss, Lin does not teach the objection functions being all of a localization loss, detection classification, weak alignment loss, and a global alignment loss, that utilize the embeddings based on a determined alignment.
Still further while Lin teaches that its object detection in images can be performed in a vehicle, Lin does not explicitly teach generating control instructions for an autonomous vehicle, to avoid a collision by predicting objects of a traffic scene that includes the autonomous vehicle.
Rohrbach teaches predicting objects within videos (Rohrbach page 435, sections 3 and 3.1, a semantic representation is extracted from a video including objects in the video such as a carrot and a knife).
Saito teaches objective functions, including a localization loss, detection classification loss, weak alignment loss, and a global alignment loss, that utilize the embeddings based on a determined alignment (Saito pages 6951–6953, section 3, loss functions for training a machine learning model including equations 1, 5, 6–10 and 11–12, including weak global feature alignment loss Lcls (weak and global alignment loss), detection loss including classification loss (detection classification loss) Ldet, a local feature alignment loss Lloc (localization loss), using feature vectors F (embeddings) based on local and global alignment).
Mohr teaches generating control instructions for an autonomous vehicle, to avoid a collision by predicting objects of a traffic scene that includes the autonomous vehicle (Mohr Abstract, ¶¶ 39, 54, image detection for managing traffic including detecting at least one operational vehicle in the images, and control a trajectory of the operational vehicle based on the image detection to avoid accidents or faults (avoid collision)).
Therefore, taking the teachings of Lin and Rohrbach together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the object detection of Lin to include detecting objects in videos as disclosed in Rohrbach at least because doing so would allow for more accurate object detection by using object and activity detection to determine the different components of visual input. See Rohrbach Abstract, and page 435 section 3.
Further, taking the teachings of Lin and Saito together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the object detection of Lin to include the loss functions as disclosed in Saito at least because doing so would improve object detection performance and allow for adaptation between dissimilar domains. See Saito Abstract and section 4.1.
Still further, taking the teachings of Lin and Mohr together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the object detection of Lin to include the controlling of an autonomous vehicle using images of a traffic scene as disclosed in Mohr at least because doing so would allow for safe and robust driving performance. See Mohr ¶ 44.
Regarding claims 2, 9 and 16, with claim 2 as a representative, and deficiencies of Lin noted in square brackets [], Lin teaches wherein the training datasets include at least detection data, image caption data, attribute data, and [action data] (Lin col. 20, ll. 33–36, 45–54, and col. 21, ll. 8–10, the database of training datasets including images and annotations of objects (attribute data), bounding boxes of objects (detection data), annotations describing objects in an image as part of image metadata (image caption data)).
Lin does not explicitly teach action data.
Rohrbach teaches action data (Rohrbach page 437, TACoS dataset which contain annotations on activity (action) as part of SR (semantic relationship) data).
Therefore, taking the teachings of Lin and Rohrbach together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the object detection of Lin to include training data with action data as disclosed in Rohrbach at least because doing so would allow for more accurate object detection by using object and activity detection to determine the different components of visual input. See Rohrbach Abstract, and page 435 section 3.
Regarding claims 3, and 10, with claim 3 as a representative, Lin does not teach the limitations of claims 3, and 10, but Rohrbach teaches wherein the detection data is converted into a simple sentence concatenating all object categories (Rohrbach pages 435–436, section 3.2, SR data translated into a description (simple sentence), including the object and tool in the SR data (all object categories) by concatenating the concepts using spaces as delimiters).
Therefore, taking the teachings of Lin and Rohrbach together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the object detection of Lin to include converting detection data as disclosed in Rohrbach at least because doing so would allow for more accurate object detection by using object and activity detection to determine the different components of visual input. See Rohrbach Abstract, and page 435 section 3.
Regarding claims 4, 11 and 17, with claim 4 as a representative, and deficiencies of Lin noted in square brackets [], Lin teaches wherein each object is described with a bounding box and [some semantic description] (Lin col. 22, ll. 1–14, output image includes bounding boxes surrounding objects in the image). Lin does not explicitly teach, but Rohrbach teaches some semantic description (Rohrbach pages 435–436, section 3.2, the semantic relation is converted to a description for the video, thus the description being a semantic description).
Therefore, taking the teachings of Lin and Rohrbach together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the object detection of Lin to include describing each object with a semantic description as disclosed in Rohrbach at least because doing so would allow for more accurate object detection by using object and activity detection to determine the different components of visual input. See Rohrbach Abstract, and page 435 section 3.
Regarding claims 6, 13, and 19, Lin teaches wherein a classification loss is applied when category annotations are available for each object (Lin col. 22, ll. 24–35, and col. 21, ll. 9–10, based on comparing detection results with training data including images with annotations describing objects in the image (category annotations), the training module applies a stochastic gradient descent that minimizes any suitable loss function (classification loss)).
Claims 5, 7, 12, 14, 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Lin, Rohrbach, Saito and Mohr as set forth above regarding the independent claims, further in view of D. Ko et al., "Video-Text Representation Learning via Differentiable Weak Temporal Alignment," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, June 24, 2022, pp. 5006-5015, doi: 10.1109/CVPR52688.2022.00496 (herein “Ko”).
Regarding claims 5, 12, and 18, Lin as modified by Rohrbach above teaches the claimed “the object detection neural network” but does not explicitly teach, where Ko teaches wherein localization is applied to minimize localization ability (Ko page 5006, right column, alignment algorithm allows flexibility by taking into account a globally optimal path as well as locally optimal paths by introducing local neighborhood smoothing (localization), since the localization is part of the alignment it would reduce the need for localization to be done elsewhere in an object detection system (with an intended result of “to minimize localization ability”)).
Therefore, taking the teachings of Lin and Rohrbach and Ko together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the object detection of Lin as modified by Rohrbach to include alignment that takes into account a globally optimal path as well as locally optimal paths by introducing local neighborhood smoothing as disclosed in Ko at least because doing so would provide significant improvements in multi-modal representation learning and better performance on various challenging downstream tasks. See Ko, Abstract.
Regarding claims 7, 14 and 20, Lin as modified by Rohrbach does not teach but Ko teaches wherein a weak alignment objective function and a global alignment objective function are applied, the weak alignment objective function trained from image-caption pairs without localization annotation (Ko page 5010, Algorithm 1 and Weak Alignment section, and page 5011, fig. 4, the algorithm including an outer loop DTW alignment processing (global alignment objective function) and a weak alignment objective function shown in Algorithm 1 to be a merge function, aligning clip (image) and caption (caption) sequences (pairs) using dummy elements and a pair distance, but there is no mentioning of localization annotations within the weak alignment portion of the algorithm, thus “without” localization annotation).
Therefore, taking the teachings of Lin and Rohrbach and Ko together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the object detection of Lin as modified by Rohrbach to include the weak alignment function to be trained from image-caption pairs without localization annotation as taught by Ko at least because doing so would provide significant improvements in multi-modal representation learning and better performance on various challenging downstream tasks. See Ko, Abstract.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE M KOETH whose telephone number is (571)272-5908. The examiner can normally be reached Monday-Thursday, 09:00-17:00, Friday 09:00-13:00, EDT/EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at 571-272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
MICHELLE M. KOETH
Primary Examiner
Art Unit 2671
/MICHELLE M KOETH/Primary Examiner, Art Unit 2671