Prosecution Insights
Last updated: July 17, 2026
Application No. 18/790,931

In-Vehicle Object Queries with Large Multi-Modal Models

Non-Final OA §101§103
Filed
Jul 31, 2024
Examiner
SONIFRANK, RICHA MISHRA
Art Unit
2654
Tech Center
2600 — Communications
Assignee
Nissan North America Inc.
OA Round
1 (Non-Final)
66%
Grant Probability
Favorable
1-2
OA Rounds
1y 0m
Est. Remaining
92%
With Interview

Examiner Intelligence

Grants 66% — above average
66%
Career Allowance Rate
256 granted / 386 resolved
+4.3% vs TC avg
Strong +26% interview lift
Without
With
+25.8%
Interview Lift
resolved cases with interview
Typical timeline
3y 0m
Avg Prosecution
21 currently pending
Career history
415
Total Applications
across all art units

Statute-Specific Performance

§101
3.2%
-36.8% vs TC avg
§103
90.3%
+50.3% vs TC avg
§102
2.7%
-37.3% vs TC avg
§112
3.2%
-36.8% vs TC avg
Black line = Tech Center average estimate • Based on career data from 386 resolved cases

Office Action

§101 §103
Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . DETAILED ACTION The office action sent in response to Applicant’s communication received on 7/31/2024 for the application number 18790931. The office hereby acknowledges receipt of the following placed of record in the file: Specification, Abstract, Oath/Declaration and claims. Status of the claims Claims 1-20 are presented for examination. Claim Rejections - 35 USC § 101 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1-6, 9, 12-13, 15, 17 and 19 are rejected under 101. Claim 17 includes: A system, comprising: one or more memories; and one or more processors configured to execute instructions stored in the one or more memories to: (a) detect a trigger that causes one or more in-cabin cameras to capture one or more videos of a cabin environment of a vehicle; (b) generate a first caption for a first frame of a first video of the one or more videos by a large multi-modal model (LMM); (c) receive a query concerning the cabin environment by a microphone; (d) convert the query to a text-based prompt; (e) generate a response to the prompt by the LMM based on the first caption; (f) convert the response to speech; and cause a speaker to output the speech. Step a is a data gathering activity. Step b -f recites a mental process since human can create a caption based on the video. A person can also receive a question from another person regarding a child or human inside a vehicle ( vehicle environment) and give the response. Step 1: This part of the eligibility analysis evaluates whether the claim falls within any statutory category. See MPEP 2106.03. The claim recites at least a system hence a machine. Thus, the claim is , recites a statutory categories of invention. (Step 1: YES). Step 2A, Prong One: This part of the eligibility analysis evaluates whether the claim recites a judicial exception. As explained in MPEP 2106.04, subsection II, a claim “recites” a judicial exception when the judicial exception is “set forth” or “described” in the claim. As discussed above, the broadest reasonable interpretation of steps (b)-(f) that those steps fall within the mental process groupings of abstract ideas because they cover concepts performed in the human mind, including observation, evaluation, judgment, and opinion. See MPEP 2106.04(a)(2), subsection III. As discussed a person can create a caption based on the video. A person can also receive a question from another person regarding a child or human inside a vehicle ( vehicle environment) and give the response. Hence, these steps can be performed by a human, using “observation, evaluation, judgment, [and] opinion,” because they involve making doing analysis on the given data which are mental tasks humans routinely do,' ” and thus can practically be performed in the human mind, In re Killian, 45 F.4th 1373, 1379 (Fed. Cir. 2022). Therefore, these limitations are considered together as a abstract idea for further analysis. (Step 2A, Prong One: YES). Step 2A, Prong Two: This part of the eligibility analysis evaluates whether the claim as a whole integrates the recited judicial exception into a practical application of the exception or whether the claim is “directed to” the judicial exception. This evaluation is performed by (1) identifying whether there are any additional elements recited in the claim beyond the judicial exception, and (2) evaluating those additional elements individually and in combination to determine whether the claim as a whole integrates the exception into a practical application. See MPEP 2106.04(d). Claim requires memory, processor, (a) detect a trigger that causes one or more in-cabin cameras to capture one or more videos of a cabin environment of a vehicle; steps (b)-(f) requires Large multimodal language model, microphone and speaker. The limitations “(a) detect a trigger that causes one or more in-cabin cameras to capture one or more videos of a cabin environment of a vehicle” are mere data gathering and output recited at a high level of generality, and thus are insignificant extra-solution activity. See MPEP 2106.05(g) (“whether the limitation is significant”). In addition, all uses of the recited judicial exceptions require such data gathering and output, and, as such, these limitations do not impose any meaningful limits on the claim. These limitations amount to necessary data gathering. See MPEP 2106.05. The limitations in (b), (d) and (e) reciting “using the large multimodal language mode” provide nothing more than mere instructions to implement an abstract idea on a generic computer. See MPEP 2106.05(f). MPEP 2106.05(f) provides the following considerations for determining whether a claim simply recites a judicial exception with the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer: (1) whether the claim recites only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished; (2) whether the claim invokes computers or other machinery merely as a tool to 8 perform an existing process; and (3) the particularity or generality of the application of the judicial exception. Although the additional element “using an LLM” limits the identified judicial exceptions. This type of limitation merely confines the use of the abstract idea to a particular technological environment (neural networks) and thus fails to add an inventive concept to the claims. See MPEP 2106.05(h). The limitations of using a microphone and speaker provide nothing more than mere instructions to implement an abstract idea on a generic computer. See MPEP 2106.05(f). MPEP 2106.05(f) provides the following considerations for determining whether a claim simply recites a judicial exception with the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer: (1) whether the claim recites only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished; (2) whether the claim invokes computers or other machinery merely as a tool to perform an existing process; and (3) the particularity or generality of the application of the judicial exception. Even when viewed in combination, these additional elements do not integrate the recited judicial exception into a practical application (Step 2A, Prong Two: NO), and the claim is directed to the judicial exception. (Step 2A: YES). Step 2B: This part of the eligibility analysis evaluates whether the claim as a whole amounts to significantly more than the recited exception, i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claim. See MPEP 2106.05. At Step 2A, Prong Two, the second additional element of using memory, processor, microphone and speaker was found to represent no more than mere instructions to apply the judicial exception on a computer using generic computer components. The analysis under Step 2A, Prong Two is carried through to Step 2B. Further, the first additional element in step (a) was found to be insignificant extra-solution activity. However, a conclusion that an additional element is insignificant extra-solution activity in Step 2A should be re-evaluated in Step 2B. See MPEP 2106.05, subsection I.A. At Step 2B, the re-evaluation of the insignificant extra-solution activity consideration takes into account whether or not the extra-solution activity is well understood, routine, and conventional in the field. See MPEP 2106.05(g). Here, the step of detecting a trigger to start monitor is a data gathering that is recited at a high level of generality, and as discussed in the disclosure, is well-understood (background - a variety of objects, from personal items to operational components, need to be identified, monitored, and managed to ensure safety, comfort, and efficiency. ). Therefore, this limitation remains insignificant extra solution activity even upon reconsideration and does not amount to significantly more. Even when considered in combination, these additional elements represent mere instructions to apply an exception and insignificant extra-solution activity, and therefore do not provide an inventive concept (Step 2B: NO). The claim is not eligible. Regarding claim 1, analysis analogous to claim 11, are applicable. Regarding 2-6, 9, 12-13, 15, and 12-18 recites a combination of mental process as same in claim 17 and additional element which are mere generic computer component under step 2, prong 2A and well known routine and conventional under prong 2b. Hence these claims are patent ineligible. Regarding claim 1 and 19, analysis analogous to claim 17, are applicable. Examiner’s Note: Claims 7-8, 10-11, 14, 16 , 18 and 20 are patent eligible. Claim Rejections - 35 USC § 103 The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. And KSR, 550 U.S. at 418, 82 USPQ2d at 1396. Exemplary rationales that may support a conclusion of obviousness include: (A) Combining prior art elements according to known methods to yield predictable results; (B) Simple substitution of one known element for another to obtain predictable results; (C) Use of known technique to improve similar devices (methods, or products) in the same way; (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results; (E) "Obvious to try" – choosing from a finite number of identified, predictable solutions, with a reasonable expectation of success; (F) Known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art; (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. See MPEP § 2143 for a discussion of the rationales listed above along with examples illustrating how the cited rationales may be used to support a finding of obviousness. See also MPEP § 2144 - § 2144.09 for additional guidance regarding support for obviousness determination. Claims 1-6, 12, 14, 16-17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Shin ( US 20250190488) and further in view of Yi (US 20200285870 ) Regarding claim 1, Shin teaches a method, comprising: generating a first caption for a first frame of a first video of the one or more videos by a large multi-modal model (LMM) ( natural language description using LLM of the key frames, Para 0058; S305, Fig 3) ; receiving a query concerning the environment by a microphone (receiving a query, Para 0030-0031) ; converting the query to a text-based prompt ( text based prompt, Fig 3) ; generating a response to the prompt by the LMM based on the first caption; converting the response to speech; and causing a speaker to output the speech ( generate a response based on description of the frame, Fig 3-4) Shin does not explicitly teach detecting a trigger that causes one or more in-cabin cameras to capture one or more videos of a cabin environment of a vehicle; receiving a query concerning the cabin environment by a microphone However Yi teaches detecting a trigger that causes one or more in-cabin cameras to capture one or more videos of a cabin environment of a vehicle (video stream based on one or more monitoring instruction; a monitoring instruction is received by the preset device, and then prompt information, and/or at least one image frame of the collected video stream, and/or detection results, etc. is received by the preset device, so that the monitoring information is fed back to the requester, Para 0135-0139); receiving a query concerning the cabin environment ( the monitoring information is fed back to the driver at the request of the driver, or the vehicle-mounted device is a vehicle-mounted device (a touch display, etc.) at another passenger's location, so that the monitoring information is fed back to the driver according to the requests of other passengers, Para 0138) Shin has a general concept of using the generative model to caption the frames of the video and utilizing a generative model to generate content responsive to a user query that is directed to a video. Shin differed by the claimed invention based on the concept the trigger of starting the video and query is related to the vehicle environment. Yi teaches the vehicle application. Therefore, under the KSR rationale of combining familiar elements according to known methods to yield predictable results, it would have been obvious to a person of ordinary skill in the art before the effective filing date to combine Shin’s generative video captioning with Yi’s vehicle monitoring to create a system that responds to in-vehicle queries ( Para 0002, Yi) Regarding claim 2, Yi as above in claim 1, teaches wherein detecting the trigger comprises at least one of: detecting a mobile device or key fob entering the cabin environment by a communication channel between the mobile device or the key fob and the vehicle; detecting an occupant entering the cabin environment by the one or more in-cabin cameras or by an in-cabin proximity sensor; detecting an occupant speaking by an in-cabin microphone; detecting the vehicle waking from a dormant state by a processor of the vehicle; or detecting the vehicle departing from an origin or arriving at a destination by a global navigation satellite system (GNSS) ( in the highway rest area, if you want to go to the toilet, but worry about the situation of the child in the vehicle, you may enable the smart rear-seat monitoring function by the mobile phone APP, Para 0131, 0139, 0186; wherein its known in the art that any monitoring instruction can be fed to the App or vehicle monitoring system as a trigger to monitor) Regarding claim 3, Shin as above in claim 1, teaches wherein the microphone comprises at least one of: an in-cabin microphone; or a microphone of a mobile device ( microphone, Para 0030) Regarding claim 4, Shin as above in claim 1, wherein the speaker comprises at least one of: an in-cabin speaker; or a speaker of a mobile device ( rendered audibly, Para 0081) Regarding claim 5, Shin as above in claim 1, further comprising: generating the response to the prompt by the LMM based on the first frame ( based on the image LLM model generates and displays , Fig 5) Regarding claim 6, Shin as above in claim 1, storing the first caption to a memory comprising at least one of: an in-vehicle storage device; or a cloud storage device ( display component ( memory) , Para 0013, 0032, 0053, 0058, 0078) Regarding claim 9, Shin as above in claim 1, teaches storing the first frame to a memory comprising at least one of: an in-vehicle storage device; or a cloud storage device ( store the key frame, Para 0013, 0032, 0053, 0058, 0078; cloud, Para 0034) Regarding claim 12, Shin as above in claim 1, teaches partitioning the first frame into a plurality of subframes; and generating a plurality of first captions for the plurality of subframes by the LMM ( frames, Fig 1c) Regarding claim 14, Shin as above in claim 1, teaches generating a plurality of first captions for a plurality of first frames of the first video by the LMM (utilize a vision-language model in generating a natural language description for the key frame(s) of the video, Abstract) ; storing individual ones of the plurality of first captions to a first memory at a first rate ( stored in the database 150, Para 0032) ; and storing individual ones of the plurality of first frames to either the first memory or a second memory at a second rate that differs from the first rate ( frames 15 and key frames are stored and selected ( rate n) and the captions in association of the video ( different rate), Fig 1c) Regarding claim 16, Shin modified by Yi as above in claim 1, teaches detecting the trigger that causes one or more in-cabin sensors to collect data for one or more properties of the cabin environment ( monitor a child, Para 0131, Yi; capture key frames, Shin) ) ; generating a description of the data for at least one of the one or more properties by the LMM (key frame description database, Fig 1c, Para 0050, 0052, Shin) ; and generating the response to the prompt by the LMM based on the description ( response is generated, Para 0052, and Fig 4, Shin) Regarding claim 17, arguments analogous to claim 1 are applicable. In addition, Shin teaches system, comprising: one or more memories; and one or more processors configured to execute instructions stored in the one or more memories to perform the steps as recited in claim 1 ( Fig 1 and Fig 5) Regarding claim 19, arguments analogous to claim 1 are applicable. In addition, Shin teaches A non-transitory computer-readable medium storing instructions operable to cause one or more processors to perform steps as recited in claim 1 ( fig 5) Regarding claim 20, Shin modified by Yi as above in claim 19, teaches detecting the trigger that causes one or more in-cabin sensors to collect data for one or more properties of the cabin environment ( collect the cabin environment --monitor a child, Para 0131, Yi; capture key frames, Shin) ) ; generating a description of the data for at least one of the one or more properties by the LMM (key frame description database, Fig 1c, Para 0050, 0052, Shin); storing the first caption and the description to a memory comprising at least one of: an in-vehicle storage device ( cloud, Para 0034, Shin or vehicle/phone app, Para 0139, Yi) ; or a cloud storage device; and generating the response to the prompt by the LMM based on the description ( response is generated, Para 0052, and Fig 4, Shin) Claims 7 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Shin ( US 20250190488) and further in view of Yi (US 20200285870 ) and further in view of Sarkar ( US 20220148586) Regarding claim 7, Shin as above in claim 1, teaches generating a second caption for a second frame of either the first video or of a second video of the one or more videos by the LMM ( generating captions of key frames, Fig 1c, (selection of key frames) , Para 0076) ; Shin does not teach generating a second caption for a second frame of either the first video or of a second video of the one or more videos; determining a similarity between the first caption and the second caption; in response to the similarity exceeding a predefined threshold, discarding the first caption and storing the second caption to a memory However, Sarkar teaches generating a second caption for a second frame of either the first video or of a second video of the one or more videos; determining a similarity between the first caption and the second caption; in response to the similarity exceeding a predefined threshold, discarding the first caption and storing the second caption to a memory (removing a set of relevant captions out of the one or more relevant captions, that are redundant, Para 0012; remove redundant captions based on similarity, Para 0068-0070, 0091) It would have been obvious having the teachings of Shin and Yi to further include the concept of Sarkar before effective filing date to reduce redundant storage and conserve memory. Regarding claim 10, Shin modified by Yi does not teach determining a similarity between the first frame and a second frame of either the first video or of a second video of the one or more videos; in response to the similarity exceeding a predefined threshold, discarding the first frame and storing the second frame to a memory However, Sarkar teaches determining a similarity between the first frame and a second frame of either the first video or of a second video of the one or more videos; in response to the similarity exceeding a predefined threshold, discarding the first frame and storing the second frame to a memory( one or more unique candidate objects from the one or more relevant scene objects, by removing the one or more relevant scene objects that are redundant, Para 0006, 0068-0070) It would have been obvious having the teachings of Shin and Yi to further include the concept of Sarkar before effective filing date to reduce redundant storage and conserve memory. Claims 8 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Shin ( US 20250190488) and further in view of Yi (US 20200285870 ) and further in view of Darrsell ( Describing Differences in Image Sets with Natural Language) Regarding claim 8, Shin as above in claim 1, teaches generating a second caption for a second frame of either the first video or of a second video of the one or more videos by the LMM ( generating captions, Fig 1, Para 0076) ; and generating the response to the prompt by the LMM based on the description ( generating a response, Fig 4) Shin does not explicitly teach determining a difference between the first caption and the second caption; in response to the difference exceeding a predefined threshold, generating a description of the difference by the LMM; and generating the response to the prompt by the LMM based on the description Darrell teaches determining a difference between the first caption and the second caption ( given a caption between a set A and B determine difference, Fig 2) ; in response to the difference exceeding a predefined threshold, generating a description of the difference by the LMM ( generating a description of the difference, Fig 2; Caption-based Proposer: We first use the VLM to generate captions of each image in SA and SB. Then, we prompt a pure language model to generate proposed differences between the two sets of captions.., Under 4.1 proposer) It would have been obvious having the teachings of Shin and Yi to further include the concept of Darrell before effective filing date so to correctly distinguish between images for classification and other application. Regarding claim 11, Shin teaches and generating the response to the prompt by the LMM based on the description ( Fig 4) Shin modified by Yi does not teach determining a difference between the first frame and a second frame of either the first video or of a second video of the one or more videos; in response to the difference exceeding a predefined threshold, generating a description of the difference by the LMM However, Darrell teaches determining a difference between the first frame and a second frame of either the first video or of a second video of the one or more videos ( Feature-based Proposer: We embed images from SA and SB into the VLM’s visual representation space, then subtract the mean embeddings of SA and SB. This sub tracted embedding is fed into VLM’s language model to generate a natural language description of the difference. Weuse BLIP-2 [23] for this proposer. Or the image based proposer, B.1. Details for Proposer) ; in response to the difference exceeding a predefined threshold, generating a description of the difference by the LMM ( generate the difference, e.g. in Page 16; also Under 4.1 Proposer; 4.2 Ranker) It would have been obvious having the teachings of Shin and Yi to further include the concept of Darrell before effective filing date so to correctly distinguish between images for classification and other applications Claims 13 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Shin ( US 20250190488) and further in view of Yi (US 20200285870 ) and further in view of He( US 20100165194 ) Regarding claim 13, Shin as above in claim 1, teaches generating a plurality of first captions for a plurality of first frames of the first video by the LMM; and storing at least one of the plurality of first captions or the plurality of first frames to a memory ( captions of the frame, Fig 1c, Para 0052, 0056, 0078) Shin modified by Yi does not teach memory configured as a circular buffer However, He teaches memory configured as a circular buffer ( The MPEG encoder 140 stores the caption data for every field/frame to handle the caption data of drop field/frame in film mode in the circular buffer 212 (step 510), buffer 0, buffer 1, . . . , buffer n,…, Para 0026) It would have been obvious having the teachings of Shin and Yi to further include the concept of He before effective filing date to improve efficiency by eliminating the need for memory reallocation as described in Para 0026 (also suggested in Para 0003, He) Regarding claim 18, Shin as above in claim 17 teach wherein the instructions include instructions to: generate a plurality of first captions for a plurality of first frames of the first video by the LMM (utilize a vision-language model in generating a natural language description for the key frame(s) of the video, Abstract); store the plurality of first captions to a first memory( stored in the database 150, Para 0032) at a first rate; and store the plurality of first frames to either the first memory or a second memory at a second rate that differs from the first rate ( frames 15 and key frames are stored and selected ( rate n) and the captions in association of the video ( different rate), Fig 1c) Shin modified by Yi does not teach memory configured as a circular buffer However, He teaches memory configured as a circular buffer ( The MPEG encoder 140 stores the caption data for every field/frame to handle the caption data of drop field/frame in film mode in the circular buffer 212 (step 510), buffer 0, buffer 1, . . . , buffer n,…, Para 0026) It would have been obvious having the teachings of Shin and Yi to further include the concept of He before effective filing date to improve efficiency by eliminating the need for memory reallocation as described in Para 0026 (also suggested in Para 0003, He) Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Shin ( US 20250190488) and further in view of Yi (US 20200285870 ) and further in view of Shetty (US 20250136134 ) Regarding claim 15, Shin modified by Yi as above in claim 1, does not explicitly teach : an individual one of the one or more in-cabin cameras comprises in infrared camera. However, Shetty in the same field of endeavor teaches an individual one of the one or more in-cabin cameras comprises in infrared camera (the interior perception camera(s) 202 may be included in a DMS, which relies on cameras and sensors positioned strategically within a vehicle cabin. In some embodiments, the interior perception camera(s) 202 are located on the dashboard, rearview mirror, or other suitable locations. In some embodiments, the interior perception camera(s) 202 includes an infrared camera for nighttime operation, Para 0076) It would have been obvious having the teachings of Shin to further include the concept of Yi before effective filing date so to capture the nighttime operation within the vehicle environment ( Para 0076, Shetty) Conclusion The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Joao (A Study on the Effectiveness of GPT-4V in Classifying Driver Behavior Captured on Video Using Just a Few Frames per Video)teaches detecting a trigger that causes one or more in-cabin cameras to capture one or more videos of a cabin environment of a vehicle (ADAS records while in operation, Method and Materials; in vehicle subsystem) ; generating a first caption for a first frame of a first video of the one or more videos by a large multi-modal model (LMM) (OCC classify the image, Table III-V) ; receiving a query concerning the cabin environment by a microphone ( user ask a question, Table V); converting the query to a text-based prompt ( prompting GPT 4v) ; generating a response to the prompt by the LMM based on the first caption (GPT-4V showed a very good performance in identifying "Yawning" and "Smoking" events with a 98.9% and 98.4% accuracy, respectively. For "Using Cellphone" events, it achieved a 95.7% accuracy, while for "Face Not Visible" events, a 94.1% accuracy., Under Experiments); and converting the response to speech; and causing a speaker to output the speech ( gpt can output using the speaker of the device) Chen( Less Is More: Picking Informative Frames for Video Captioning) storing individual ones of the plurality of first captions to a first memory at a first rate ( we sample equally-spaced 30 frames for every video, and resize them into 224×224 resolution., Under 4.3 Video processing) ; and storing individual ones of the plurality of first frames to either the first memory or a second memory at a second rate that differs from the first rate (, a compact frame subset can be selected to rep resent the visual information and perform video captioning without performance degradation, Introduction; we only use the appearance features in our model, because extracting motion features is very time consuming, which deviates from our purpose that cutting down the computation cost for video captioning, and the appearance feature is enough to represent video content when the redundant or noisy frames are filtered by our PickNet, Under 4.3; One of the advantage of our method is that it can be applied to streaming video. Different from offline video captioning, captioning for streaming video requires the model to tackle with unbounded video and generate descriptions immediately when the visual information has changed, which meets the demand of practical applications. For this online setting, we first sample frames at 1fps, and then sequentially feed the sampled frames to PickNet. ….. Figure 8 demonstrates an example of online video captioning with the picked frames and corresponding descriptions. As it is shown, the descriptions will be more appropriate and more determined as the informative frames are picked, , Under 5.3 Captioning for streaming video) Any inquiry concerning this communication or earlier communications from the examiner should be directed to Richa Sonifrank whose telephone number is (571)272-5357. The examiner can normally be reached M-T 7AM - 5:30PM. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Phan Hai can be reached at (571)272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /Richa Sonifrank/Primary Examiner, Art Unit 2654
Read full office action

Prosecution Timeline

Jul 31, 2024
Application Filed
May 13, 2026
Non-Final Rejection mailed — §101, §103 (current)

Precedent Cases

Applications granted by this same examiner with similar technology

Patent 12676156
ENCODING DEVICE AND ENCODING METHOD, DECODING DEVICE AND DECODING METHOD, AND PROGRAM
2y 8m to grant Granted Jul 07, 2026
Patent 12664183
ONLINE QUESTION ANSWERING, USING READING COMPREHENSION WITH AN ENSEMBLE OF MODELS
4y 11m to grant Granted Jun 23, 2026
Patent 12664973
VOICE DIALOGUE PROCESSING METHOD AND APPARATUS
4y 4m to grant Granted Jun 23, 2026
Patent 12645879
ENTITY RECOGNITION METHODS AND APPARATUSES, ELECTRONIC DEVICES AND STORAGE MEDIA
3y 4m to grant Granted Jun 02, 2026
Patent 12602552
Machine-Learning-Based OKR Generation
3y 0m to grant Granted Apr 14, 2026
Study what changed to get past this examiner. Based on 5 most recent grants.

Strategy Recommendation AI-generated — please review before filing

Get a prosecution strategy drawn from examiner precedents, rejection analysis, and claim mapping.
Typically takes 5-10 seconds — AI-generated, attorney review required before filing

Prosecution Projections

1-2
Expected OA Rounds
66%
Grant Probability
92%
With Interview (+25.8%)
3y 0m (~1y 0m remaining)
Median Time to Grant
Low
PTA Risk
Based on 386 resolved cases by this examiner. Grant probability derived from career allowance rate.

Sign in with your work email

Enter your email to receive a magic link. No password needed.

Personal email addresses (Gmail, Yahoo, etc.) are not accepted.

Free tier: 3 strategy analyses per month