In-Vehicle Object Queries with Large Multi-Modal Models

Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Response to Amendment Claims 1, 5, 7, 10-11, 15-17 and 19-20 are amended. Claim 8 is cancelled. Claims 1-7 and 9-20 are presented for examination. Response to Arguments Applicant argues “The rejection assigns two separate limitations of claim 1 to Yi. The limitation of detecting a trigger that causes one or more in-cabin cameras to capture one or more videos is supported by Yi, 0135-0139, and the limitation of receiving a query concerning the cabin environment by a microphone is supported by Yi, 0138. Office Action at 8.The passages relied upon describe a single user input, and that input cannot satisfy both limitations, because claim 1 requires the two events to differ in kind and in sequence. Separately recited claim elements are presumed distinct, Engel Indus., Inc. v. Lockformer Co., 96 F.3d 1398, 1404 (Fed. Cir. 1996); Becton, Dickinson & Co. v. Tyco Healthcare Grp. LP, 616 F.3d 1249, 1254 (Fed. Cir. 2010), and every limitation must be given weight, MPEP 2143.03. The trigger is what causes the cameras to capture the video. The query is a semantic request that is converted to a text-based prompt and answered by the LMM based on the first caption, which is generated from a frame of the video the trigger caused to be captured. An instruction that enables a monitoring function precedes and produces the video; it carries no semantic content to convert into a text-based prompt, and it cannot simultaneously be a question answered from the content of the video it produced. That the two Yi citations refer to the same input is apparent from Yi itself. Yi discloses receiving a monitoring instruction by a mobile application terminal or a vehicle-mounted device. Yi, 0136. That step occurs before the video stream is obtained, and the video stream is obtained according to the monitoring instruction. Yi, 0135, 0137. The request described at paragraph 0138 is that same monitoring instruction viewed from the receiving end; the paragraph opens by identifying the mobile application terminal as the preset device of the foregoing embodiment, then describes the monitoring instruction being received and monitoring information being fed back to the requester. Yi, 0138. Yi confirms that a single enablement step governs, where therear-seat monitoring function is enabled by the HMI touch screen or the mobile phone APP. Yi 0139 The passages relied upon also designate the interfaces through which the monitoring instruction is received, and none of the designated interfaces is a microphone. Yi identifies a mobile phone, a tablet computer, and a vehicle-mounted touch display. Yi, 0138. Yi identifies a mobile phone APP and a front HMI touch screen. Yi, 0139. The relied-upon disclosure does not merely omit the input interface; it specifies interfaces that do not include a microphone.” However, the examiner did not rely on the same element for two different limitations. The trigger is user pressing a button on a preset device. The query is the prompt which is based on the analysis for e.g. refer to fig 5 whether the instruction/prompt is based on the age of the rear seat passenger. Additionally, the concept of receiving a query concerning the environment is taught by Shin ( Para 0030-0031). Yi is merely relied upon the idea of detecting a trigger and that the system can receive a query. As far as the court case Engel Indus., Inc. v. Lockformer Co., 96 F.3d 1398, 1404 (Fed. Cir. 1996); here based on the plain language the two portion were different. For e.g. the return portion was clearly recited because the return portion rolled inward and was clearly distinct from the second portion. If applicant cites for e.g. “after capturing the video… receive a query”. At that point the claim construction cannot read on the query and trigger in the same way. But Examiner relies on the primary reference to teach the concept of receiving the query and performing the function afterwards. Regarding “Becton, Dickinson & Co. v. Tyco Healthcare Grp. LP, 616 F.3d 1249, 1254 (Fed. Cir. 2010)” again the plain language implies they are distinct components. Both cases are based on a device structure. b. The retrieval architecture of Shin cannot operate on the live cabin video stream of Yi Applicant argues “Shin retrieves a stored natural language description by video identity. Shin discloses a key frame description database storing entries that each identify a video and associate an identifier of the video with a natural language description for the key frames of that video. Shin, 0032. Shin described creating entry 151, which contains the natural language description and identifier 154 of the video. Shin, 0058. Retrieval proceeds by determining that the user utterance is directed to a particular identified video and then accessing the entry for that video. Shin, 0056. The video stream of Yi is a live feed of the rear seat area obtained only after the monitoring instruction is received. Yi, 0137. The passages relied upon do not describe an identifier for that stream, a database entry for it, or its selection by the user for rendering. The Office Action does not explain how the combined system would determine that a query is directed to an unbounded live stream, and there is no reasonable expectation that the identifier- keyed retrieval of Shin would operate on such a stream. MPEP 2143.02.” However, Shin discloses the generating a natural language description of the feed and after the system receives the query based on the stored information it generates an answer. Yi is only relied for the concept of trigger and query corresponds to a cabin environment. One of the ordinary skill in the art will be implement a query corresponding to a vehicle environment of Yi to any environment as suggested in Shin c. The proposed combination changes the principle of operation of Yi Applicant argues “Yi is an unprompted monitoring and alarm system. Yi outputs prompt information in response to the state information of the passenger satisfies a predetermines condition. Yi 0130. The resulting flow is set out at paragraph 0139: after the monitoring function is enabled, a software system determines whether the age of the person in the rear seat exceeds a set age, performs child state monitoring and child seat suggestion tips below that age, and monitors and alarms on whether the child is asleep, cries, is awake, or leaves the child seat, while adult monitoring displays age, gender, and other monitoring information. No query appears in that flow, and the feedback described is displayed rather than spoken. The Office Action identifies no disclosure in Yi of a speaker or of speech output. Yi also assigns the interpretive step to a person, describing the display of image frames and detection results so that the driver or other personnel may analyze and obtain the information in order to understand passenger state. Yi, 0134. Substituting the query-driven pipeline of Shin for the detection and alarm logic of Yi eliminates the unsolicited alarm that Yi exists to provide, and transfers to the LMM the interpretive role that Yi assigns to the viewer. A modification that changes the principle of operation of a reference, or that renders the reference unsatisfactory for its intended purpose, does not support a conclusion of obviousness. MPEP 2143.01(V), (VI). The articulated motivation does not cure this. The Office Action cites Yi, 0002 as the reason to combine. Office Action at 8. That paragraph addresses monitoring the driving state of the driver for driving safety. It does not concern answering occupant queries, and it does not support the combination for which it is offered. MPEP 2143.01. Applicant further notes that the limitations of converting the response to speech and causing a speaker to output the speech are supported in the rejection only by a parenthetical addressing generation of a response” However, applicant is reminded that Shin is a primary reference and teaches the claimed pipeline. Yi idea if detecting a trigger in a vehicle to start a camera can be incorporated to Shin’s pipeline so that it can be utilized in the vehicle environment. The test for obviousness is whether the combined teachings would have suggested the claimed invention, not whether the Yi can be physically modified or substituted into the Shin. d. Claim 17 and claim 19 Claim 17 and claim 19 recite corresponding limitations and are patentable over Shin in view of Yi for the same reasons. See the response to claim 1, same response applies. 2. Claim 1 as Amended is Patentable over Shin, Yi, and Darrell Claim 1 as amended incorporates the limitations of canceled claim 8. Applicant addresses the rejection of claim 8 as it would apply to claim 1 as amended. a. No reference is identified as teaching the limitation of generating the response based on the description Applicant argues “The rejection first credits Shin with generating the response to the prompt by the LMM based on the description, citing FIG. 4. Office Action at 12. The rejection then states that Shin does not explicitly teach that same limitation. Office Action at 12. The paragraph that follows maps only two limitations to Darrell, namely determining a difference between the first caption and the second caption, and generating a description of the difference in response to the difference exceeding a predefined threshold. Office Action at 12. The limitation of generating the response to the prompt by the LMM based on the description is never assigned to Darrell. The Office Action thus attributes the limitation to Shin, concedes that Shin does not teach it, and then does not locate it in the secondary reference. The limitation is left unaccounted for, and no prima facie case has been established. MPEP 2143.03. Nor could the limitation be supplied by Darrell. The Office Action identifies no disclosure in Darrell of a user query, a prompt, or a response to a query. Darrell takes two image sets as input and outputs a difference description” However, this claim was rejected based on the combination of Shin and Yi and Darrell. The concept of prompt by LLM based on description is taught by Shin and the idea of generation of description abased on difference is taught by Darrel. Applicants argue that there are no motivation. See the rejection below for the motivation to combine which is in line with KSR. b. Darrell is directed to differences between image sets, not between two captions Applicant argues “Darrell defines set difference captioning as taking two image datasets and generating a natural language description that is more true in one dataset than the other. Darrell, § 3.1. Darrell expressly distinguishes prior difference captioning and change captioning work directed to a single pair of images, and identifies scaling to sets containing thousands of images as the open problem it addresses. Darrell, § 2. The caption-based proposer relied upon in the rejection operates accordingly. It takes two subsets of twenty images each, generates a caption for every image, and prompts a language model to propose differences between the two sets of captions. Darrell, § 4.1. The supplementary material confirms twenty samples per set and ten hypotheses per round. Darrell, App. B.1. FIG. 2 of Darrell shows the same structure, listing multiple captions for set A and multiple captions for set B beneath the instruction to identify concepts more true for set A than set B. Claim 1 as amended requires determining a difference between the first caption and the second caption, each generated for a single frame. The passages relied upon determine differences between two sets of captions rather than between two individual captions.” However, Fig 2 of Darrell clearly shows the difference between two captions. c. The cited disclosure contains no threshold, and the threshold Darrell does disclose is not the recited threshold Applicant argues “The rejection supports the limitation of generating the description in response to the difference exceeding a predefined threshold by citing the caption-based proposer passage of section 4.1. Office Action at 12. That passage recites generating captions of each image and prompting a language model to propose differences. It contains no threshold, no comparison, and no condition on generation. Darrell discloses a threshold elsewhere, but it is not the recited threshold and it does not condition generation of any description. Darrell filters out proposed differences that are not statistically significant by running a t-test on the two score distributions with a significance threshold of 0.05. Darrell, § 4.2. That threshold fails to meet the limitation for two independent reasons. First, it operates on differences that have already been generated. The proposer generates candidate difference descriptions from the sampled captions, and only then does the ranker evaluate those candidates across the image sets and discard the ones that are not significant. Darrell, §§ 4, 4.2. Claim 1 as amended requires the opposite order: the difference between the first caption and the second caption is determined, that difference is compared against the predefined threshold, and the description is generated only in response to the difference exceeding the threshold. The recited order is functional rather than a rearrangement of steps. Because generation is conditioned on the threshold, caption pairs falling below it never invoke the LMM at all, which preserves storage memory. Spec, 0074. In Darrell the language model is invoked for each sampled set, and the significance filter is applied only afterward. Second, the quantity compared against the 0.05 threshold is not a difference between two captions. The Darrell threshold is applied to the statistical significance of two distributions of scores measuring how well individual images satisfy a proposed hypothesis. Darrell, § 4.2. Claim 1 as amended requires the threshold to be compared against a difference determined between the first caption and the second caption. The two comparisons are directed to different quantities.” However, examiner relied on the combination of Shin modified by Yi and Darrell to teach this concept. d. The articulated motivation is untethered to the claims Applicant argues “The stated reason to combine is to correctly distinguish between images for classification and other application. Office Action at 13. Claim 1 as amended concerns generating a response to a query about a cabin environment. Classification appears nowhere in the claim. The Office Action articulates a benefit belonging to the field of Darrell rather than a reason a person of ordinary skill would have modified Shin as modified by Yi to arrive at the claimed invention. MPEP 2143.01.” See the rejection below for the motivation to combine which is in line with KSR. e. Claim 17 and claim 19 Claim 17 and claim 19 as amended recite corresponding limitations and are patentable for the same reasons. Refer to the response to arguments for claim 1, same reasons apply here. 3. The Dependent Claims Claims 2-7, 9-16, 18, and 20 depend from claim 1, claim 17, or claim 19 and are patentable at least by virtue of that dependence. Applicant addresses certain claims separately below and does not waive argument as to the remainder. a. Claim 2 Applicant argues “Claim 2 recites five alternative triggers, each detected automatically by a vehicle system: a mobile device or key fob entering the cabin environment detected by a communication channel; an occupant entering the cabin environment detected by the in-cabin cameras or an in- cabin proximity sensor; an occupant speaking detected by an in-cabin microphone; the vehicle waking from a dormant state detected by a processor; and the vehicle departing an origin or arriving at a destination detected by GNSS. The disclosure relied upon in Yi is a person manually enabling rear-seat monitoring through a mobile phone APP or an HMI touch screen. Yi, 1 0131, 0139. Manual enablement through an application or a touch screen corresponds to none of the five recited alternatives. The rejection bridges this gap with the assertion that "its known in the art that any monitoring instruction can be fed to the App or vehicle monitoring system as a trigger to monitor." Office Action at 9. No reference is cited for that assertion. Applicant traverses the taking of Official Notice and requests that the Examiner provide documentary evidence in support. MPEP 2144.03(C). Applicant further notes that the asserted proposition, even if supported, is not directed to any alternative recited in claim 2.” However, the manual enablement would read on vehicle waking up since the vehicle has to be started for this manual enablement. b. Claim 9 Applicant argues “Claim 9 requires storing the first frame to a memory. Each paragraph relied upon describes storing the natural language description rather than a frame. Shin, 1 0013, 0032, 0058, 0078. The Office Action identifies no disclosure in Shin of storing a key frame.” However, Fig 1c, clearly shows stored key frames c. Claim 12 Applicant argues “Claim 12 requires partitioning the first frame into a plurality of subframes and generating a plurality of first captions for the subframes. The rejection cites FIG. 1C of Shin. Office Action at 9. FIG. 1C shows key frames KF1 through KFn selected from video 15 with corresponding descriptions D1 through Dn, which is temporal selection of whole frames. The Office Action identifies no disclosure in Shin of partitioning a frame into subframes.” However, refer to Fig 1c element 15 of Shin showing partitions. d. Claim 14 Applicant argues “Claim 14 requires storing captions to a first memory at a first rate and storing frames to the first memory or a second memory at a second rate that differs from the first rate. The rejection cites FIG. 1C of Shin for both limitations. Office Action at 9-10. The passages relied upon identify no storage rate for either captions or frames, and, as set forth above with respect to claim 9, the passages relied upon describe storing the natural language description rather than a frame.” However, that’s not true. See rejection for claim 14 and specifically -- frames 15 and key frames are stored and selected ( rate n) and the captions in association of the video ( different rate), Fig 1c e. Claims 7 and 10 Applicant argues “Sarkar addresses redundancy among region proposals within a single acquired scene image. Sarkar explains that overlapping region proposals result in redundant caption generation, and identifies two types of redundancy: object redundancy, where multiple bounding boxes are proposed for the same object, and caption redundancy, where multiple captions are generated for the same object. Sarkar, 0068. The disclosure relied upon for claim 10 concerns determining unique candidate objects by removing redundant relevant scene objects. Sarkar, 0006. The passages relied upon are directed to objects and captions within a single acquired scene image rather than to frames of a video, and they describe removing redundant items from a candidate set rather than discarding a first frame or a first caption and storing a second frame or a second caption to a memory.” However, the claim language does not distinguish from Sarkar, claim does not specify why one frame is being discarded or being stored. f. Claim 11 Applicant argues “claim 11 requires determining a difference between the first frame and a second frame, and generating a description of the difference by the LMM in response to the difference exceeding a predefined threshold. The passage relied upon describes embedding images from two sets into a visual representation space and subtracting the mean embeddings of the two sets, then feeding the subtracted embedding into a language model to generate a description of the difference. Darrell, § 4.1; App. B.1. That operation is performed on mean embeddings of two sets of images rather than on a first frame and a second frame, and the passages relied upon describe no threshold conditioning generation of the description. For the reasons set forth in Section 2.c above, the significance filter of section 4.2 is applied to already-generated proposals and is directed to a different quantity.” However, under broadest reasonable interpretation image reads on frame and a presence of difference reads on predetermined threshold, if there is a difference the description of difference is generated in Darrell. g. Claims 13 and 18 Applicant argues “Claim 18 recites storing captions to a first memory at a first rate and storing frames to a second memory at a second rate that differs from the first rate. As set forth above with respect to claim 14, the passages relied upon identify no storage rate for either captions or frames, and He is not relied upon for that limitation. As to both claims, He is directed to an MPEG encoder that stores caption data for every field and frame in a circular buffer in order to handle caption data of dropped fields and frames in film mode. He, 0026. Applicant respectfully submits that a broadcast encoder addressing caption timing during film-mode frame dropping is neither in the field of endeavor of the claimed invention nor reasonably pertinent to the problem with which the inventors were concerned. MPEP 2141.01(a).” However, refer to the rejection of claims 13 and 18. Examiner clearly cites how He reads on the claims and motivation to combine. Examiner’s Note: The concept of “generating a response to the prompt by the LMM based on the first caption and the description” changes the scope of invention, which necessitated claims to be read on different portions of the same prior art used. Claim Rejections - 35 USC § 103 The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. And KSR, 550 U.S. at 418, 82 USPQ2d at 1396. Exemplary rationales that may support a conclusion of obviousness include: (A) Combining prior art elements according to known methods to yield predictable results; (B) Simple substitution of one known element for another to obtain predictable results; (C) Use of known technique to improve similar devices (methods, or products) in the same way; (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results; (E) "Obvious to try" – choosing from a finite number of identified, predictable solutions, with a reasonable expectation of success; (F) Known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art; (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. See MPEP § 2143 for a discussion of the rationales listed above along with examples illustrating how the cited rationales may be used to support a finding of obviousness. See also MPEP § 2144 - § 2144.09 for additional guidance regarding support for obviousness determination. Claims 1-6, 11-12, 14, 16-17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Shin ( US 20250190488) and further in view of Yi (US 20200285870 ) and further in view of Darrsell ( Describing Differences in Image Sets with Natural Language) Regarding claim 1, Shin teaches a method, comprising: generating a first caption for a first frame of a first video of the one or more videos by a large multi-modal model (LMM) (natural language description 157, Fig 1B) ; generating a second caption for a second frame of either the first video or of a second video of the one or more videos by the LMM (generating captions of key frames, Fig 1c, (selection of key frames) , Para 0076 Fig 1, Para 0076); generating a description by the LMM (combined description which is stored in associated with the video entry, Fig 1C element 153, Para 0058, 0061) ; receiving a query concerning the environment by a microphone (receiving a query, Para 0030-0031) ; converting the query to a text-based prompt ( text based prompt, Fig 3) ; generating a response to the prompt by the LMM based on the first caption and the description ( based on element 157 and 153, Fig 1b) ; converting the response to speech; and causing a speaker to output the speech ( generate a response based on description of the frame, Fig 1b, 3-4) Shin does not teach determining a difference between the first caption and the second caption; in response to the difference exceeding a predefined threshold, generating a description of the difference by the LMM However, Darrell teaches determining a difference between the first caption and the second caption ( given a caption between a set A and B determine difference, Fig 2) ; in response to the difference exceeding a predefined threshold, generating a description of the difference by the LMM ( generating a description of the difference, Fig 2; Caption-based Proposer: We first use the VLM to generate captions of each image in SA and SB. Then, we prompt a pure language model to generate proposed differences between the two sets of captions.., Under 4.1 proposer) It would have been obvious having the teachings of Shin and Yi to further include the concept of Darrell before effective filing date so to correctly distinguish between images for classification and other application. Shin does not explicitly teach detecting a trigger that causes one or more in-cabin cameras to capture one or more videos of a cabin environment of a vehicle; receiving a query concerning the cabin environment by a microphone However Yi teaches detecting a trigger that causes one or more in-cabin cameras to capture one or more videos of a cabin environment of a vehicle (video stream based on one or more monitoring instruction; a monitoring instruction is received by the preset device, and then prompt information, and/or at least one image frame of the collected video stream, and/or detection results, etc. is received by the preset device, so that the monitoring information is fed back to the requester, Para 0135-0139); receiving a query concerning the cabin environment ( the monitoring information is fed back to the driver at the request of the driver, or the vehicle-mounted device is a vehicle-mounted device (a touch display, etc.) at another passenger's location, so that the monitoring information is fed back to the driver according to the requests of other passengers, Para 0138) Shin has a general concept of using the generative model to caption the frames of the video and utilizing a generative model to generate content responsive to a user query that is directed to a video. Shin differed by the claimed invention based on the concept the trigger of starting the video and query is related to the vehicle environment. Yi teaches the vehicle application. Therefore, under the KSR rationale of combining familiar elements according to known methods to yield predictable results, it would have been obvious to a person of ordinary skill in the art before the effective filing date to combine Shin’s generative video captioning with Yi’s vehicle monitoring to create a system that responds to in-vehicle queries ( Para 0002, Yi) Regarding claim 2, Yi as above in claim 1, teaches wherein detecting the trigger comprises at least one of: detecting a mobile device or key fob entering the cabin environment by a communication channel between the mobile device or the key fob and the vehicle; detecting an occupant entering the cabin environment by the one or more in-cabin cameras or by an in-cabin proximity sensor; detecting an occupant speaking by an in-cabin microphone; detecting the vehicle waking from a dormant state by a processor of the vehicle; or detecting the vehicle departing from an origin or arriving at a destination by a global navigation satellite system (GNSS) ( in the highway rest area, if you want to go to the toilet, but worry about the situation of the child in the vehicle, you may enable the smart rear-seat monitoring function by the mobile phone APP, Para 0131, 0139, 0186; wherein its known in the art that any monitoring instruction can be fed to the App or vehicle monitoring system as a trigger to monitor) Regarding claim 3, Shin as above in claim 1, teaches wherein the microphone comprises at least one of: an in-cabin microphone; or a microphone of a mobile device ( microphone, Para 0030) Regarding claim 4, Shin as above in claim 1, wherein the speaker comprises at least one of: an in-cabin speaker; or a speaker of a mobile device ( rendered audibly, Para 0081) Regarding claim 5, Shin as above in claim 1, further comprising: generating the response to the prompt by the LMM further based on the first frame ( based on the image LLM model generates and displays , Fig 5) Regarding claim 6, Shin as above in claim 1, storing the first caption to a memory comprising at least one of: an in-vehicle storage device; or a cloud storage device ( display component ( memory) , Para 0013, 0032, 0053, 0058, 0078) Regarding claim 9, Shin as above in claim 1, teaches storing the first frame to a memory comprising at least one of: an in-vehicle storage device; or a cloud storage device ( store the key frame, Para 0013, 0032, 0053, 0058, 0078; cloud, Para 0034) Regarding claim 11, Shin teaches and generating the response to the prompt by the LMM further based on the second description( Fig 1b, 4) Shin modified by Yi does not teach determining a second difference between the first frame and the second frame in response to the second difference exceeding a second predefined threshold, generating a second description of the second difference by the LMM; However, Darrell teaches determining a second difference between the first frame and the second frame ( Feature-based Proposer: We embed images from SA and SB into the VLM’s visual representation space, then subtract the mean embeddings of SA and SB. This sub tracted embedding is fed into VLM’s language model to generate a natural language description of the difference. Weuse BLIP-2 [23] for this proposer. Or the image based proposer, B.1. Details for Proposer) ; in response to the second difference exceeding a second predefined threshold, generating a second description of the second difference by the LMM; ( generate the difference, e.g. in Page 16; also Under 4.1 Proposer; 4.2 Ranker) It would have been obvious having the teachings of Shin and Yi to further include the concept of Darrell before effective filing date so to correctly distinguish between images for classification and other applications Regarding claim 12, Shin as above in claim 1, teaches partitioning the first frame into a plurality of subframes; and generating a plurality of first captions for the plurality of subframes by the LMM ( frames, Fig 1c) Regarding claim 14, Shin as above in claim 1, teaches generating a plurality of first captions for a plurality of first frames of the first video by the LMM (utilize a vision-language model in generating a natural language description for the key frame(s) of the video, Abstract) ; storing individual ones of the plurality of first captions to a first memory at a first rate ( stored in the database 150, Para 0032) ; and storing individual ones of the plurality of first frames to either the first memory or a second memory at a second rate that differs from the first rate ( frames 15 and key frames are stored and selected ( rate n) and the captions in association of the video ( different rate), Fig 1c) Regarding claim 16, Shin modified by Yi as above in claim 1, teaches detecting the trigger that causes one or more in-cabin sensors to collect data for one or more properties of the cabin environment ( monitor a child, Para 0131, Yi; capture key frames, Shin) ) ; generating a description of the data for at least one of the one or more properties by the LMM (key frame description database, Fig 1c, Para 0050, 0052, Shin) ; and generating the response to the prompt by the LMM based on the description ( response is generated, Para 0052, and Fig 4, Shin) Regarding claim 17, arguments analogous to claim 1 are applicable. In addition, Shin teaches system, comprising: one or more memories; and one or more processors configured to execute instructions stored in the one or more memories to perform the steps as recited in claim 1 ( Fig 1 and Fig 5) Regarding claim 19, arguments analogous to claim 1 are applicable. In addition, Shin teaches A non-transitory computer-readable medium storing instructions operable to cause one or more processors to perform steps as recited in claim 1 ( fig 5) Regarding claim 20, Shin modified by Yi as above in claim 19, teaches detecting the trigger that causes one or more in-cabin sensors to collect data for one or more properties of the cabin environment ( collect the cabin environment --monitor a child, Para 0131, Yi; capture key frames, Shin) ) ; generating a second description of the data for at least one of the one or more properties by the LMM (key frame description database, Fig 1c, Para 0050, 0052, Shin); storing the first caption and the description to a memory comprising at least one of: an in-vehicle storage device ( cloud, Para 0034, Shin or vehicle/phone app, Para 0139, Yi) ; or a cloud storage device; and generating the response to the prompt by the LMM based on the second description ( response is generated, Para 0052, and Fig 4, Shin) Claims 7 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Shin ( US 20250190488) and further in view of Yi (US 20200285870 ) and further in view of Darrsell ( Describing Differences in Image Sets with Natural Language) and further in view of Sarkar ( US 20220148586) Regarding claim 7, Shin as above in claim 1, teaches generating a second caption for a second frame of either the first video or of a second video of the one or more videos; determining a similarity between the first caption and the second caption; in response to the similarity exceeding a predefined threshold, discarding the first caption and storing the second caption to a memory However, Sarkar teaches generating a second caption for a second frame of either the first video or of a second video of the one or more videos; determining a similarity between the first caption and the second caption; in response to the similarity exceeding a second predefined threshold, discarding the first caption and storing the second caption to a memory (removing a set of relevant captions out of the one or more relevant captions, that are redundant, Para 0012; remove redundant captions based on similarity, Para 0068-0070, 0091) It would have been obvious having the teachings of Shin and Yi to further include the concept of Sarkar before effective filing date to reduce redundant storage and conserve memory. Regarding claim 10, Shin modified by Yi does not teach determining a similarity between the first frame and a second frame; in response to the similarity exceeding a second predefined threshold, discarding the first frame and storing the second frame to a memory However, Sarkar teaches determining a similarity between the first frame and a second frame; in response to the similarity exceeding a second predefined threshold, discarding the first frame and storing the second frame to a memory( one or more unique candidate objects from the one or more relevant scene objects, by removing the one or more relevant scene objects that are redundant, Para 0006, 0068-0070) It would have been obvious having the teachings of Shin and Yi to further include the concept of Sarkar before effective filing date to reduce redundant storage and conserve memory. Claims 13 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Shin ( US 20250190488) and further in view of Yi (US 20200285870 ) and further in view of Darrsell ( Describing Differences in Image Sets with Natural Language)and further in view of He( US 20100165194 ) Regarding claim 13, Shin as above in claim 1, teaches generating a plurality of first captions for a plurality of first frames of the first video by the LMM; and storing at least one of the plurality of first captions or the plurality of first frames to a memory ( captions of the frame, Fig 1c, Para 0052, 0056, 0078) Shin modified by Yi does not teach memory configured as a circular buffer However, He teaches memory configured as a circular buffer ( The MPEG encoder 140 stores the caption data for every field/frame to handle the caption data of drop field/frame in film mode in the circular buffer 212 (step 510), buffer 0, buffer 1, . . . , buffer n,…, Para 0026) It would have been obvious having the teachings of Shin and Yi to further include the concept of He before effective filing date to improve efficiency by eliminating the need for memory reallocation as described in Para 0026 (also suggested in Para 0003, He) Regarding claim 18, Shin as above in claim 17 teach wherein the instructions include instructions to: generate a plurality of first captions for a plurality of first frames of the first video by the LMM (utilize a vision-language model in generating a natural language description for the key frame(s) of the video, Abstract); store the plurality of first captions to a first memory( stored in the database 150, Para 0032) at a first rate; and store the plurality of first frames to either the first memory or a second memory at a second rate that differs from the first rate ( frames 15 and key frames are stored and selected ( rate n) and the captions in association of the video ( different rate), Fig 1c) Shin modified by Yi does not teach memory configured as a circular buffer However, He teaches memory configured as a circular buffer ( The MPEG encoder 140 stores the caption data for every field/frame to handle the caption data of drop field/frame in film mode in the circular buffer 212 (step 510), buffer 0, buffer 1, . . . , buffer n,…, Para 0026) It would have been obvious having the teachings of Shin and Yi to further include the concept of He before effective filing date to improve efficiency by eliminating the need for memory reallocation as described in Para 0026 (also suggested in Para 0003, He) Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Shin ( US 20250190488) and further in view of Yi (US 20200285870 ) and further in view of Darrsell ( Describing Differences in Image Sets with Natural Language) and further in view of Shetty (US 20250136134 ) Regarding claim 15, Shin modified by Yi as above in claim 1, does not explicitly teach : an individual one of the one or more in-cabin cameras comprises in infrared camera. However, Shetty in the same field of endeavor teaches an individual one of the one or more in-cabin cameras comprises in infrared camera (the interior perception camera(s) 202 may be included in a DMS, which relies on cameras and sensors positioned strategically within a vehicle cabin. In some embodiments, the interior perception camera(s) 202 are located on the dashboard, rearview mirror, or other suitable locations. In some embodiments, the interior perception camera(s) 202 includes an infrared camera for nighttime operation, Para 0076) It would have been obvious having the teachings of Shin to further include the concept of Yi before effective filing date so to capture the nighttime operation within the vehicle environment ( Para 0076, Shetty) Conclusion The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Joao (A Study on the Effectiveness of GPT-4V in Classifying Driver Behavior Captured on Video Using Just a Few Frames per Video)teaches detecting a trigger that causes one or more in-cabin cameras to capture one or more videos of a cabin environment of a vehicle (ADAS records while in operation, Method and Materials; in vehicle subsystem) ; generating a first caption for a first frame of a first video of the one or more videos by a large multi-modal model (LMM) (OCC classify the image, Table III-V) ; receiving a query concerning the cabin environment by a microphone ( user ask a question, Table V); converting the query to a text-based prompt ( prompting GPT 4v) ; generating a response to the prompt by the LMM based on the first caption (GPT-4V showed a very good performance in identifying "Yawning" and "Smoking" events with a 98.9% and 98.4% accuracy, respectively. For "Using Cellphone" events, it achieved a 95.7% accuracy, while for "Face Not Visible" events, a 94.1% accuracy., Under Experiments); and converting the response to speech; and causing a speaker to output the speech ( gpt can output using the speaker of the device) Chen( Less Is More: Picking Informative Frames for Video Captioning) storing individual ones of the plurality of first captions to a first memory at a first rate ( we sample equally-spaced 30 frames for every video, and resize them into 224×224 resolution., Under 4.3 Video processing) ; and storing individual ones of the plurality of first frames to either the first memory or a second memory at a second rate that differs from the first rate (, a compact frame subset can be selected to rep resent the visual information and perform video captioning without performance degradation, Introduction; we only use the appearance features in our model, because extracting motion features is very time consuming, which deviates from our purpose that cutting down the computation cost for video captioning, and the appearance feature is enough to represent video content when the redundant or noisy frames are filtered by our PickNet, Under 4.3; One of the advantage of our method is that it can be applied to streaming video. Different from offline video captioning, captioning for streaming video requires the model to tackle with unbounded video and generate descriptions immediately when the visual information has changed, which meets the demand of practical applications. For this online setting, we first sample frames at 1fps, and then sequentially feed the sampled frames to PickNet. ….. Figure 8 demonstrates an example of online video captioning with the picked frames and corresponding descriptions. As it is shown, the descriptions will be more appropriate and more determined as the informative frames are picked, , Under 5.3 Captioning for streaming video) THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to Richa Sonifrank whose telephone number is (571)272-5357. The examiner can normally be reached M-T 7AM - 5:30PM. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Phan Hai can be reached at (571)272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /Richa Sonifrank/Primary Examiner, Art Unit 2654