DETAILED ACTION
Notice of AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
2. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
3. Claims 1, 6-8, 11, 12, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al (Joint Visual Grounding and Tracking with Natural Language Specification, March 21, 2023) (Applicant disclosed reference) and in further view of Zhang et al (All in One: Exploring Unified Vision-Language Tracking with Multi-Modal Alignment, Feb. 28th, 2025), and Cheng et al (Segment and Track Anything, May 11, 2023) (Applicant disclosed reference).
Regarding claim 1, Zhou et al teaches: A data processing system comprising: a processor; and a machine-readable medium storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations comprising: obtaining a first frame of video content comprising a plurality of frames over which a target object is to be tracked [page 2: 2.1, p01; page 4: p01]; obtaining a natural language description of the target object [fig. 2]; encoding the first frame of the video content, and the natural language description of the target object as fused encoding information using a single object tracking pipeline [abstract, page 3: 3.1: p02]; and tracking the target object with the single object tracking pipeline using the fused encoding information [page 2: p03-p05].
For a redundant teaching in the same field of endeavor, Zhang et al teaches a framework with vision language tracker using a unified transformer embedded with visual and language signals: A data processing system comprising: a processor; and a machine-readable medium storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations comprising: encoding the first frame of the video content, and the natural language description of the target object as fused encoding information using a single object tracking pipeline [abstract, page 3: 3, fig. 2].
Zhou et al in view of Zhang et al does not disclose the first point input. In the same field of endeavor, Cheng et al teaches: obtaining a first point input denoting a point on the first frame of the video content representing a location of the target object on the first frame of the video content; and tracking the target object with the single object tracking pipeline using the fused encoding information [abstract, page 2: p03].
Therefore, given Zhou et al’s teaching on unified visual language grounding and tracking model to locate and track an object using NL descriptions and video information, Zhang et al’s framework of a single object tracking pipeline with fused and encoded visual content and NL description, and Cheng et al’s prescription on first frame multimodal user click and text interaction to select and track the object, it would have been obvious for an ordinary skilled in the art before the effective filing date of the claimed invention to combine the teaching of all to encode video content , NL description, and first point input as fused encoding information as a single tracking pipeline for object selecting and tracking for improved accuracy and efficiency.
Regarding claim 6, the rationale applied to the rejection of claim 1 has been incorporated herein. Zhang et al further teaches: The data processing system of claim 1, wherein encoding the first frame of the video content, the first point input, and the natural language description of the target object as the fused encoding information further comprises: tokenizing the natural language description using a tokenizer to obtain a list of tokens [page 5: 4.1]; adding a classification token to a beginning of the list of tokens and a separator token at an end of the list of tokens [page 4: 3.2]; encoding the list of tokens using a language model to obtain language embeddings representing the natural language description [page 4: 3.2]; and providing the language embeddings as an input to a unified fusion encoder of the single object tracking pipeline [fig. 2].
Regarding claim 7, the rationale applied to the rejection of claim 1 has been incorporated herein. Zhou et al further teaches: The data processing system of claim 1, wherein encoding the first frame of the video content, the first point input, and the natural language description of the target object as the fused encoding information further comprises: analyzing the first frame of the video content using a Swin Transformer model to generate image embeddings [page 4: 3.2]; and providing the image embeddings as an input to a unified fusion encoder of the single object tracking pipeline [page 4: 3.3: p02].
Regarding claim 8, the rationale applied to the rejection of claim 1 has been incorporated herein. Zhang et al further teaches: The data processing system of claim 1, wherein encoding the first frame of the video content, the first point input, and the natural language description of the target object as the fused encoding information further comprises: generating embeddings associated with the first frame of the video content, the first point input, and the natural language description [page 4: 3.2 (vision token, language token)]; and providing the embeddings as an input to a unified fusion encoder trained to analyze the embeddings and generate features associated with the first frame of the video content, the first point input, and the natural language description [page 4: 3.2].
Regarding claim 11, the rationale applied to the rejection of claim 1 has been incorporated herein. Zhou et al further teaches: The data processing system of claim 1, wherein tracking the target object with the single object tracking pipeline further comprises: analyzing the fused encoding information using a unified fusion decoder to predict a bounding box for the target object in the first frame of the video content [page 3: fig. 2, page 4: 3.4].
Claim 12 has been analyzed and rejected with regard to claim 1.
Claim 17 has been analyzed and rejected with regard to claim 1 and in accordance with Zhou et al, in view of Zhang et al and Cheng et al’s further teaching on: A data processing system comprising: a processor; and a machine-readable medium storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations comprising [Zhang: page 3: 3]:
receiving, at a single object tracking pipeline, a request to track a target object from an object tracking application [Cheng: page 2: p02],
providing the image embeddings, the language embeddings, and the click embeddings as an input to a unified fusion encoder to obtain fused encoding information [Zhang: page 3: 3, page 4: 3.2]; and
providing the fused encoding information to a unified fusion decoder to obtain bounding box information for the target object, the bounding box information surrounding a predicted location of the target object within the first frame of the video content [Zhou: fig. 2, page 4: p01, 3.4].
4. Claims 2, 4, 13, 15, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al (Joint Visual Grounding and Tracking with Natural Language Specification, March 21, 2023) (Applicant disclosed reference), Zhang et al (All in One: Exploring Unified Vision-Language Tracking with Multi-Modal Alignment, Feb. 28th, 2025), and Cheng et al (Segment and Track Anything, May 11, 2023) (Applicant disclosed reference); and in further view of Kirillov et al (Segment Anything, Apr. 05, 2023) (Applicant disclosed reference) and Tancik et al (Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, Jun. 18th, 2020) (Applicant disclosed reference).
Regarding claim 2, the rationale applied to the rejection of claim 1 has been incorporated herein. Zhou et al in view of Zhang et al and Cheng et al is lack of details on encoding. In the same field of endeavor, Kirillov et al teaches: The data processing system of claim 1, wherein encoding the first frame of the video content, the first point input, and the natural language description of the target object as the fused encoding information further comprises: analyzing the first point input using a click encoder configured to generate point embeddings by encoding the first point input using Gaussian Random Fourier features and a learnable embedding vector [page 2: p03, page 5: p03 (Prompt encoder)]; and providing the point embeddings as an input to a unified fusion encoder of the single object tracking pipeline [page 5: Mask decoder]. Therefore, given Kirillov et al’s prescription on prompt embeddings to mask decoder and Zhou et al in view of Zhang et al and Cheng et al’s teaching on the single object tracking pipeline, it would have been obvious for an ordinary skilled in the art before the effective filing date of the claimed invention to feed Kirillov et al’s prompt embeddings to the object tracking pipeline to combine user prompt, NL description, and visual information to produce tracking result.
Kirillov et al cites Tancik et al for Gaussian Random Fourier Features. In the same field of endeavor, Tancik et al teaches: encoding the first point input using Gaussian Random Fourier features [abstract, introduction, page 2: p01; page 19: p01]. Therefore, given Tancik et al’s teaching on encoding a coordinate input using Gaussian Random Fourier features, it would have been obvious for an ordinary skilled in the art before the effective filing date of the claimed invention to combine the teaching of all to encode the first point input using Gaussian Random Fourier features for providing spatial detail.
Regarding claim 4, the rationale applied to the rejection of claim 2 has been incorporated herein. Cheng et al further teaches: The data processing system of claim 2, wherein during an evaluation phase of the single object tracking pipeline, the first point input comprises a user-specified point selected on a user interface of a tracking application [page 2: p03].
Regarding claim 13, the rationale applied to the rejection of claim 12 has been incorporated herein. Claim 13 has been analyzed and rejected with regard to claim 2.
Regarding claim 15, the rationale applied to the rejection of claim 13 has been incorporated herein. Claim 15 has been analyzed and rejected with regard to claim 4.
Regarding claim 18, the rationale applied to the rejection of claim 17 has been incorporated herein. Claim 18 has been analyzed and rejected with regard to claim 2.
Regarding claim 20, the rationale applied to the rejection of claim 18 has been incorporated herein. Claim 20 has been analyzed and rejected with regard to claim 4.
5. Claims 3, 14, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al (Joint Visual Grounding and Tracking with Natural Language Specification, March 21, 2023) (Applicant disclosed reference), Zhang et al (All in One: Exploring Unified Vision-Language Tracking with Multi-Modal Alignment, Feb. 28th, 2025), and Cheng et al (Segment and Track Anything, May 11, 2023) (Applicant disclosed reference), Kirillov et al (Segment Anything, Apr. 05, 2023) (Applicant disclosed reference), and Tancik et al (Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, Jun. 18th, 2020) (Applicant disclosed reference); and in further view of Yue et al (SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation, Dec. 21, 2023).
Regarding claim 3, the rationale applied to the rejection of claim 2 has been incorporated herein. Kirillov et al further teaches: The data processing system of claim 2, wherein during a training phase of the single object tracking pipeline, the first point input comprises a center of a ground truth bounding box of the target object [page 8: 7.1: p02]. Kirillov et al does not specify jitter component. In the same field of endeavor, Yue et al teaches a random jitter component [page 2: p03, fig. 2]. Therefore, given Yue et al’s teaching on addressing sensitivity to imperfect prompts using jitter and Kirillov et al’s teaching on training with point prompts including center based and randomly sampled points, it would have been obvious to apply Yue’s prompt jitter concept to Kirillov’s point prompt training to obtain a training point with a ground truth object center plus a random jitter component.
Regarding claim 14, the rationale applied to the rejection of claim 12 has been incorporated herein. Claim 14 has been analyzed and rejected with regard to claim 3.
Regarding claim 19, the rationale applied to the rejection of claim 18 has been incorporated herein. Claim 19 has been analyzed and rejected with regard to claim 3.
6. Claims 5 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al (Joint Visual Grounding and Tracking with Natural Language Specification, March 21, 2023) (Applicant disclosed reference), Zhang et al (All in One: Exploring Unified Vision-Language Tracking with Multi-Modal Alignment, Feb. 28th, 2025), Cheng et al (Segment and Track Anything, May 11, 2023) (Applicant disclosed reference), and Kirillov et al (Segment Anything, Apr. 05, 2023) (Applicant disclosed reference) and Tancik et al (Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, Jun. 18th, 2020) (Applicant disclosed reference); and in further view of Luo et al (SAM-PD: How Far Can SAM Take Us in Tracking and Segmenting Anything in Videos by Prompt Denoising, March 07, 2024).
Regarding claim 5, the rationale applied to the rejection of claim 4 has been incorporated herein. For using predicted bounding box in the previous frame to determine a second point input in a second frame, Luo et al, in the same field of endeavor teaches: The data processing system of claim 4, wherein the machine-readable medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of: obtaining a second frame of the video content [abstract]; determining a second point input denoting a second point on the first frame of the video content based on a predicted bounding box of the target object in the first frame of the video content [abstract, page 3: B]; encoding the second frame of video content, the second point input, and the natural language description of the target object to obtain second fused encoding information using the single object tracking pipeline; and tracking the target object using the second fused encoding information [fig. 2, page 3: B]. Therefore, given Luo et al’s teaching on a prior frame prediction used to generate the next frame prompt, it would have been obvious for an ordinary skilled in the art before the effective filing date of the claimed invention to combine the teaching of all to use a point derived from prior frame bounding box as the next point input for to provide spatial guidance for tracking.
Regarding claim 16, the rationale applied to the rejection of claim 15 has been incorporated herein. Claim 16 has been analyzed and rejected with regard to claim 5.
7. Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al (Joint Visual Grounding and Tracking with Natural Language Specification, March 21, 2023) (Applicant disclosed reference), Zhang et al (All in One: Exploring Unified Vision-Language Tracking with Multi-Modal Alignment, Feb. 28th, 2025), and Cheng et al (Segment and Track Anything, May 11, 2023) (Applicant disclosed reference), Kirillov et al (Segment Anything, Apr. 05, 2023) (Applicant disclosed reference), and Tancik et al (Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, Jun. 18th, 2020) (Applicant disclosed reference); and in further view of Yan et al (Learning Spatio-Temporal Transformer for Visual Tracking, March 31, 2021) (Applicant disclosed reference).
Regarding claim 10, the rationale applied to the rejection of claim 1 has been incorporated herein. Zhou et al further teaches: The data processing system of claim 1, wherein tracking the target object with the single object tracking pipeline further comprises: generating a semantic temporal cue for tracking the target object based on previously predicted bounding boxes associated with the target object [abstract, page 8: conclusion];
In Zhou et al, NL description, template and test image, and temporal historical information are encode which is used by decoder to predict target box. In the same field of endeavor, Yan et al teaches: utilizing the semantic temporal cue to build a query for a unified fusion decoder to predict a bounding box for the target object associated with a current frame of the plurality of frames [abstract]. Therefore, it would have been obvious for an ordinary skilled in the art before the effective filing date of the claimed invention to incorporate Yan et al’s decoder query bounding box prediction to Zhou et al’s semantic temporal tracking framework to guide the decoder more directly to current frame target location to improve tracking accuracy.
Allowable Subject Matter
8. Claim 9 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Contact
9. Any inquiry concerning this communication or earlier communications from the examiner should be directed to FAN ZHANG whose telephone number is (571)270-3751. The examiner can normally be reached on Mon-Fri 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Benny Tieu can be reached on 571-272-7490. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Fan Zhang/
Patent Examiner, Art Unit 2682
Prosecution Insights