DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed 02/19/2026 has been entered. Applicant has amended claims 1, 2, 13, 14, 16, 17 and 20. Claims 18 has been cancelled. No new claims have added. Claims 1-17 and 19-20 are currently pending in the instant application.
Response to Arguments
Applicant’s arguments, see page 8-12, filed 02/19/2026, with respect to the rejection(s) of claim(s) 1-20 under 35 U.S.C. 102 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in further view of Vasylyev (US 2024/0412720). Vasylyev teaches the amended limitations as seen in the current rejection below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-17 and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Huang et al (US 2025/0191369) in view of Vasylyev (US 2024/0412720).
Regarding claim 1,
Huang discloses:
1. A method of processing a query associated with a video, the method comprising ([0007] Embodiments of the present disclosure relate to language instructed temporal localization in videos. Systems and methods are disclosed that address temporal localization in videos using LLMs. In contrast to conventional systems, such as those described above, models according to the present disclosure are designed to answer “when?” questions, while simultaneously improving other relevant Vid-LLM capabilities.): generating, using at least one large language model, a set of event parsing instructions based on the query and an event parsing prompt ([0041] - The processing circuitry is further configured to pre-process the natural language input in a language pre-processing pathway to generate a plurality of language tokens, provide, to a pre-trained multimodal large language model (LLM), the plurality of visual tokens and the plurality of language tokens, and process, by the pre-trained multimodal LLM, the plurality of fast tokens, the plurality of slow tokens, and the plurality of language tokens to generate output responsive to the natural language input. ); executing the set of event parsing instructions to generate event parsing data; ([0062] The LITA model includes a large language model (LLM) module 111, a SlowFast token pooling layer 112, and a visual encoder and linear projection layer 113. In FIG. 1B, a video 114 is provided as input to the visual encoder and linear projection layer 113 in order to be first encoded into visual tokens (numbered by frame). The visual tokens are further processed, in the SlowFast token pooling layer 112, via two pathways. A fast token pathway averages all the tokens in a frame to maintain a high temporal resolution. A slow token pathway sparsely samples frames to maintain a larger number of tokens per frame to provide spatial information. Timestamps are converted to time tokens <1> to <T>. This is important for better temporal localization learning.) generating, by the at least one large language model, a set of grounding instructions for the video based on a grounding prompt and the event parsing data; executing the set of grounding instructions to generate grounding data ([0110] The process also involves, at 306, pre-processing the language component (i.e. the query) of the model input in a natural language pre-processing pathway. Pre-processing the language component of the model input involves tokenizing the query at 306 to provide a plurality of language tokens.); generating, by the at least one large language model, a set of reasoning instructions based on a reasoning prompt and the grounding data; ([0064] - . The entire LITA model is fine-tuned via reasoning temporal localization (RTL) data—as discussed herein-along with other video tasks, such as dense video captioning and event localization. The LITA model learns to use time tokens instead of absolute timestamps. For temporal localization, the LITA model can then respond to “when” questions (e.g. “When is she dancing?”) with time tokens (e.g. “She is dancing from <2> to <3>.”), which can then be converted to timestamps given the video length.) executing the set of reasoning instructions to generate reasoning data ([0045] According to a fourth aspect of the present disclosure, a computer-implemented method is provided for training a multimodal large language model (LLM) to perform language instructed temporal localization in video. The method includes providing a reasoning temporal localization (RTL) training dataset. The RTL training dataset includes a set of videos and a collection of query: answer pairs. The collection of query: answer pairs includes, for each video, at least one respective query: answer pair, each of which includes a query including language input that requires temporal reasoning and an answer including an absolute timestamp and a natural language caption responsive to the corresponding query. ); and generating, by the at least one large language model, a response to the query based on the reasoning data ([0116] - At 327, the multimodal LLM computes, during inference, output that is responsive to the multimodal input. At 328, the process 320 provides the out).
Huang does not explicitly teach generate event parsing data indicative of at least one temporal relationship to an event indicated by the query; …generate grounding data indicative of at least one spatial location of the event in the video; …s to generate reasoning data indicative of context associated with the grounding data.
Vasylyev teaches generate event parsing data indicative of at least one temporal relationship to an event indicated by the query; …generate grounding data indicative of at least one spatial location of the event in the video; …s to generate reasoning data indicative of context associated with the grounding data. ([0082] For processing images captured by the camera, according to one embodiment, the AI Assistant may use CNNs to analyze each frame individually as a static image, in accordance with the above-described techniques, ignoring temporal information (e.g., the information on how objects and features move and change over time). According to an alternative embodiment, a 3D convolution technique, being an adaptation of CNNs, can be used to process video data. Instead of 2D filters and pooling operations, the 3D convolution technique (or 3D CNNs) can be configured to use 3D filters and 3D pooling to extract features across both space and time. The third dimension in the filters may extend across the frames, enabling the network to detect motions directly by applying convolutions across the time axis as well. According to another alternative embodiment, an optical flow technique can be used alongside frame-by-frame analysis to capture motion between successive frames by calculating the motion of objects or features. This technique may be specifically configured to estimate the motion of objects between two frames of a video at different times. It may be based on the apparent motion of brightness patterns in the image, assuming that the patterns are relatively consistent between successive frames, and that their motion within the scene is smooth. According to further alternative embodiment, CNNs may be combined a recurrent neural network (RNN), a useful example of which can be Long Short-Term Memory (LSTM) networks. In this embodiment, the CNN can extract spatial features from individual frames, and the LSTM can piece together information across frames to understand temporal dynamics.)
Accordingly, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention to have modified the teachings of Haung to include enerate event parsing data indicative of at least one temporal relationship to an event indicated by the query; …generate grounding data indicative of at least one spatial location of the event in the video; …s to generate reasoning data indicative of context associated with the grounding data as taught by Vasylyev. It would be advantageous since an improved AI assistant system that can record, store, and process conversations in real-time and provide contextual understanding for reduced latency and improved accuracy and user experience as taught by Vasylyev [0009]
Regarding claim 2, Haung in view of Vasylyev teaches The method of claim 1, Vasyleyv further teaches further comprising storing the event parsing data, the grounding data, and the reasoning data at a memory that is accessible to the at least one large language model ([0034] The hardware may include a sound capturing device, such as a microphone, for recording the conversations, an audio output device, such as a speaker, for communication, a physical memory for storing relevant conversation data, one or more processors for processing the conversation and user commands, and a wireless communication device for accessing external databases and internet resources.
[0035] The software aspect of the Assistant may be grounded on a natural language processing model that is trained on a substantial amount of text data, primarily comprising a transformer-based language model. This model may be configured to enable the Assistant to generate contextually relevant responses and foster engaging and meaningful conversations.t.).
Regarding claim 3, Haung in view of Vasylyev teaches The method of claim 1, Haung further teaches wherein the at least one large language model corresponds to a single large language model with shared parameters. ([0091] Two variations of the LITA model (i.e. one incorporating a 7 billion (7B) parameter LLaVA model and one incorporating a 13 billion (13B) parameter LLaVA model) were configured to uniformly sample 100 frames from a video, and use 100 time tokens <1> to <100> to represent timestamps. CLIP-L-14 was used as the visual encoder, and Vicuna as the LLM module. )
Regarding claim 4, Haung in view of Vasylyev teaches The method of claim 1, Haung further teaches wherein the at least one large language model comprises: a first large language model that is used to generate the set of event parsing instructions, the set of grounding instructions, and the set of reasoning instructions; and a response prediction large language model that is used to generate the response([0111] The process further involves, at 307, concatenating the plurality of fast tokens, the plurality of slow tokens, and the plurality of language tokens. At 308, the process provides the tokens to a multimodal LLM. In at least one embodiment, the multimodal LLM is an image LLM. In at least one embodiment, the multimodal LLM is the LLaVA LLM. In at least one embodiment, the multimodal LLM is based on a neural network architecture that includes at least one transformer model. In at least one embodiment, the at least one transformer model includes multiple layers of attention mechanisms and feedforward neural networks.) .
Regarding claim 5, Haung in view of Vasylyev teaches The method of claim 1, Haung further teaches wherein the at least one large language model comprises: a first large language model that is used to generate the set of event parsing instructions; a second large language model that is used to generate the set of grounding instructions; a third large language model that is used to generate the set of reasoning instructions; and a response prediction large language model that is used to generate the response([0111] The process further involves, at 307, concatenating the plurality of fast tokens, the plurality of slow tokens, and the plurality of language tokens. At 308, the process provides the tokens to a multimodal LLM. In at least one embodiment, the multimodal LLM is an image LLM. In at least one embodiment, the multimodal LLM is the LLaVA LLM. In at least one embodiment, the multimodal LLM is based on a neural network architecture that includes at least one transformer model. In at least one embodiment, the at least one transformer model includes multiple layers of attention mechanisms and feedforward neural networks. ).
Regarding claim 6, Haung in view of Vasylyev teaches The method of claim 1, Haung further teaches wherein the set of event parsing instructions correspond to a first set of application programming interface (API) calls, wherein the set of grounding instructions correspond to a second set of API calls, and wherein the set of reasoning instructions correspond to a third set of API calls. ([0215] - ng embedding vector to perform a nearest neighbor search to generate one or more neighbors 816. In at least one embodiment, one or more neighbors 816 is value in retrieval database 814 corresponding to a key comprising input data 810. In at least one embodiment, one or more neighbors 816 comprise text data. In at least one embodiment, encoder 818 encodes one or more neighbors 816. In at least one embodiment, encoder 818 encodes one or more neighbors 816 into a text embedding vector. In at least one embodiment, encoder 818 encodes one or more neighbors 816 into a sentence embedding vector. In at least one embodiment, large language model 816 uses input data 810 and data generated by encoder 818 to generate output data 820. In at least one embodiment, processor 806 interfaces with application 802 using large language model (LLM) application programming interface(s) (API(s)) 804. In at least one embodiment, processor 806 accesses large language model 816 using large language model (LLM) application programming interface(s) (API(s)) 804.) Vasylyev also teaches the limitations see [0051]
Regarding claim 7, Haung in view of Vasylyev teaches The method of claim 1, Haung further teaches wherein, execution of the set of event parsing instructions by a processor, cause the processor to perform operations comprising: detecting temporal hint indicators in the query; detecting temporal relationship indicators in the query; detecting a question type of the query; or determining whether the query invokes use of one or more tools ([0066] Given the time representation utilized by the LITA model, many video tasks related to temporal localization can be transformed into language instructions and answers. For example, dense video captioning can be achieved by prompting the model with “Describe the video. Each sentence begins with start and end timestamps.” (Q3 and A3 in FIG. 1B). Standard event localization is also transformed to “When does X happen?” (Q1 and A1 in FIG. 1B). Standard video question answering can also be incorporated (Q2 and A2 in FIG. 1B). More details are discussed herein below.).
Regarding claim 8, Haung in view of Vasylyev teaches The method of claim 7, Haung further teaches wherein the one or more tools comprise an optical character recognition (OCR) tool. ([0072] The first training task is dense video captioning. In dense video captioning, each video is described by a set of sentences, and each sentence comes with the start and end timestamps of the event. Each sentence in dense video captioning can thus be represented as: <start time><end time> SENTENCE. All sentences can be sorted by start time and directly concatenated along with the timestamps. One example prompt to the model for this task is: “Provide a detailed description of the given video. Each sentence should begin with the start and end timestamps.”)
Regarding claim 9, Haung in view of Vasylyev teaches The method of claim 1, Haung further teaches wherein, execution of the set of grounding instructions by a processor, cause the processor to perform operations comprising: identifying candidate frames of the video that are associated with the query; or identifying temporal regions in the video with one or more vision-language tools for entity detection and image-text alignment. ([0067] While time in videos can be discretized into T steps in order to make Video LLMs better at reasoning about time, the visual input should still match the temporal resolution T in order to achieve effective temporal processing. Ideally, at least T frames would be needed to temporally localize events with the resolution T. However, naively feeding all T frames into the LLM module 111 could be computationally prohibitive. For example, using T=100 and M=256 (CLIP VIT-L-14) would lead to 25600 tokens per video. To simultaneously provide both sufficient temporal resolution and sufficient visual resolution without requiring that a computationally prohibitive input be provided to the LLM module 111, the LITA model utilizes two pathways to pool the T×M tokens for T frames. The first pathway utilizes densely sampled fast tokens to provide temporal information.)
Regarding claim 10, Haung in view of Vasylyev teaches The method of claim 1, Haung further teaches wherein, execution of the set of reasoning instructions by a processor, causes the processor to perform operations comprising generating responses to one or more sub-questions associated with the query ([0074] The third training task is video question answering. The question answering task is already represented as language instructions. However, answers in existing question answering datasets often consist of a single word or phrase because models for this task might not be able to generate longer text. To address this issue, the following prompt can be appended to the question: “Answer the question using a single word or phrase.” The goal is to provide the context for short answers so that it affects the model's text generation less.)
Regarding claim 11, Haung in view of Vasylyev teaches The method of claim 1, Haung further teaches wherein the set of reasoning instructions are further based on the event parsing data. ([0062] The LITA model includes a large language model (LLM) module 111, a SlowFast token pooling layer 112, and a visual encoder and linear projection layer 113. In FIG. 1B, a video 114 is provided as input to the visual encoder and linear projection layer 113 in order to be first encoded into visual tokens (numbered by frame). The visual tokens are further processed, in the SlowFast token pooling layer 112, via two pathways. A fast token pathway averages all the tokens in a frame to maintain a high temporal resolution. A slow token pathway sparsely samples frames to maintain a larger number of tokens per frame to provide spatial information. Timestamps are converted to time tokens <1> to <T>. This is important for better temporal localization learning.)
Regarding claim 12, Haung in view of Vasylyev teaches The method of claim 1, Haung further teaches wherein the query is expressed in natural language. ([0064] - The text tokens (which are derived from input in the form of a natural language prompt) are processed to convert any referenced timestamps to specialized time tokens (<1> to <T>). )
Claim 13 is rejected using similar reasoning seen in the rejection of claim 1 due to reciting similar limitations but directed towards a different statutory category (an apparatus).
Regarding claim 14, Haung in view of Vasylyev teaches The appartus of claim 13, Haung further teaches wherein the processor is further configured to store the event parsing data, the grounding data, and the reasoning data at an external memory that is accessible to the at least one large language model. ([0054] -The memory stores a reasoning temporal localization (RTL) training dataset. The RTL training dataset includes a set of videos and a collection of query: answer pairs. The collection of query: answer pairs includes, for each video, at least one respective query: answer pair. Each respective query: answer pair includes a query including language input that requires temporal reasoning and an answer including an absolute timestamp and a natural language caption responsive to the corresponding query. The processing circuitry connected to the memory is configured to train a multimodal LLM using the RTL training dataset.).
Claims 15- 19 are rejected using similar reasoning seen in the rejection of claim 3-7 due to reciting similar limitations but directed towards a different statutory category (an apparatus).
Claim 20 is rejected using similar reasoning seen in the rejection of claim 1 due to reciting similar limitations but directed towards a different statutory category (a non-transitory computer-readable medium).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SAMUEL SHARPLESS whose telephone number is (571)272-1521. The examiner can normally be reached M-F 7:30 AM- 3:30 PM (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ALEKSANDR KERZHNER can be reached at 571-270-1760. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.C.S./Examiner, Art Unit 2165
/ALEKSANDR KERZHNER/Supervisory Patent Examiner, Art Unit 2165
Prosecution Insights