Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1, 14, and 18 have been amended. Claims 1-20 are pending and rejected in the application. This action is Final.
Response to Arguments
Applicant Argues:
As discussed in the interview, Lee does not appear to disclose, teach, or suggest "generating a prompt by combining the semantic information of the query request from the first output, the grounding information from the second output, and a directive to generate a response to the query request using the semantic information of the query request and the grounding information," and "providing the prompt to a lightweight large generative model to generate a query response to the query request."
Examiner Responds:
Applicant's 35 USC § 103 arguments with respect to claims 1-20 have been considered but are moot in view of the new ground(s) of rejection.
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6, 7, 12, 13, 14, 15, 16, 17, 18, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Kharbanda et al. U.S. Patent Publication (2024/0403362; hereinafter: Kharbanda) in view of Madisetti et al. U.S. Patent Publication (2025/0328560; hereinafter: Madisetti) and further in view of Lee et al. U.S. Patent Publication (2025/0190503; hereinafter: Lee)
Claims 1 and 18
As to claims 1 and 18, Kharbanda discloses a system comprising:
a processing system (paragraph[0100]); and
a computer memory comprising instructions that, when executed by the processing system, cause the processing system to perform operations of (paragraph[0100]):
obtaining a multimodal input from a client device that includes a first input portion in a first mode and a second input portion in a second mode that is different from the first mode (paragraph[0044], the reference describes inputting audio (i.e., first input portion in a first mode, as claimed) and video input (i.e., a second input portion, as claimed).);
providing the first input portion to a first lightweight large generative model to generate a first output that includes semantic information of a query request, the first lightweight large generative model being generated to process inputs in a first mode (paragraph[0068], the reference describes capturing audio data and using a model to transform the audio into a query.);
providing the second input portion to a second lightweight context model to generate a second output that includes image grounding information, the second lightweight context model being generated to process inputs in a second mode (paragraph[0037], the reference describes processing the video through a machine learning model.);
Kharbanda does not appear to explicitly disclose generating a prompt by combining the semantic information of the query request from the first output, the grounding information from the second output, and a directive to generate a response to the query request using the semantic information of the query request and the grounding information;
providing the prompt to a lightweight large generative model to generate a query response to the query request;
and providing the query response in response to the query request.
However, Madisetti discloses generating a prompt by combining the semantic information of the query request from the first output, the grounding information from the second output, and a directive to generate a response to the query request using the semantic information of the query request and the grounding information (paragraph[0323]-paragraph[0332], the reference describes generating prompts based on a sequence of prompts. The Examiner interprets a sequence of prompts being the semantic and grounding information outputs from Kharbanda. The reference describes retrieving results from the generated prompts in paragraph[0332].). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to a person having ordinary skill in the art to which said subject matter pertains to have modified the teachings of Kharbanda with the teachings of Madisetti to combine data to generate a prompt which would result in the claim invention. The skilled artisan would have been motivated to improve the teachings of Kharbanda with the teachings of Madisetti to efficiently process multi-level generative AI and large language models form generative AI applications (Madisetti: paragraph[0032]).
The combination of Kharbanda and Madisetti do not appear to explicitly disclose providing the prompt to a lightweight large generative model to generate a query response to the query request;
and providing the query response in response to the query request.
However, Lee discloses
providing the prompt to a lightweight large generative model to generate a query response to the query request (Figure 2, paragraph[0069], the reference describes providing an response.); and
providing the query response in response to the query request (Figure 2, paragraph[0164], the reference describes providing a result.). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to a person having ordinary skill in the art to which said subject matter pertains to have modified the teachings of Kharbanda with the teachings of Madisetti and Lee to combine query data to input into a model which would result in the claim invention. The skilled artisan would have been motivated to improve the teachings of Kharbanda with the teachings of Madisetti and Lee to efficiently process a queries associated with a videos using models (Lee: paragraph[0005]).
Claim 2
As to claim 2, the combination of Kharbanda, Madisetti, and Lee discloses all the elements in claim 1, as noted above, and Kharbanda further disclose wherein:
the first input portion includes captured audio and the first context model includes a speech-to-text model to generate converted text as the semantic information (paragraph[0068], the reference describes transcribing the speech to text.);
the second input portion includes a captured image and the second context model includes a lightweight image metadata model that generates descriptive text information of the captured image as image grounding information from input images (paragraph[0052], the reference describes generating data about the video using the model.).
Kharbanda does not appear to explicitly disclose the lightweight large generative model includes a text-based large language model that receives text-only versions of the grounding information and the semantic information to answer the query request.
However, Lee further discloses the lightweight large generative model includes a text-based large language model that receives text-only versions of the grounding information and the semantic information to answer the query request (Figure 2, paragraph[0068], the reference describes using a large language model.).
Claim 3
As to claim 3, the combination of Kharbanda, Madisetti, and Lee discloses all the elements in claim 2, as noted above, and Kharbanda further disclose wherein the first input portion is provided to the first context model concurrently with providing the second input portion to the second context model (Figure 1, paragraph[0047], the reference describes inputting audio and video data to the multimodal at the same time.).
Claim 4
As to claim 4, the combination of Kharbanda, Madisetti, and Lee discloses all the elements in claim 2, as noted above, and Lee further disclose providing the captured image to one or more additional lightweight image metadata models selected from an image classification model, a similar image search model, an object detection model, an image segmentation model, a text recognition model, or a visual search image model (Figure 2, paragraph[0068], the reference describes using a classification model.).
Claim 6
As to claim 6, the combination of Kharbanda, Madisetti, and Lee discloses all the elements in claim 1, as noted above, and Lee further disclose providing the second input portion to a visual-based large generative model to obtain additional image grounding information, wherein the visual-based large generative model takes longer to process images than the second context model, wherein the visual-based large generative model receives both text and image inputs (Figure 2, paragraph[0066], the reference describes inputting the video to generate additional information (i.e., image grounding information, as claimed).); and
sending the additional image grounding information to the lightweight large generative model (Figure 2, paragraph[0066], the reference describes using the additional information to input into the generative model (e.g., Figure 2, element 236) for a query response.).
Claim 7
As to claim 7, the combination of Kharbanda, Madisetti, and Lee discloses all the elements in claim 6, as noted above, and Lee further disclose sending the additional image grounding information to the lightweight large generative model with the prompt (Figure 2, paragraph[0068], the reference describes using the additional information to input into the generative model (e.g., Figure 2, element 236) in a query form.).
Claim 12
As to claim 12, the combination of Kharbanda, Madisetti, and Lee discloses all the elements in claim 1, as noted above, and Lee further disclose detecting a selection of an input element within a graphical user interface of the client device (paragraph[0082], the reference describes an selection interface to select portions of a video.); and
in response to detecting the selection of the input element, capturing the multimodal input by capturing the first input portion and the second input portion together (Figure 2, paragraph[0068], the reference describes inputting audio and video into a multimodal.).
Claim 13
As to claim 13, the combination of Kharbanda, Madisetti, and Lee discloses all the elements in claim 1, as noted above, and Lee further disclose generating the prompt to send to the lightweight large generative model by including image grounding information from the second output and the semantic information from the first output into a query prompt (Figure 2, paragraph[0071], the reference describes using a lightweight model to receive query data.).
Claim 14
As to claim 14, Kharbanda discloses a computer-implemented method for responding to multimodal input queries within a threshold time:
based on detecting a query request from a client device, capturing a multimodal input that includes an audio input portion and an image input portion(paragraph[0068], the reference describes capturing audio data and using a model to transform the audio into a query.);
providing the audio input portion to a speech-to-text model to generate a converted text string that includes semantic information of the query request (paragraph[0068], the reference using speech models to transcript to a text query.);
providing the image input portion to a lightweight image metadata model to generate image grounding information that includes a text description of an image captured by the client device(paragraph[0037], the reference describes processing the video through a machine learning model.).
Kharbanda does not appear to explicitly disclose generating, a query prompt by combining the semantic information of the query request from the converted text string, the text description of the image from the image grounding information, and a directive to generate a response to the query request using the semantic information of the query request and the text description of the image;
providing the query prompt to a lightweight text-based large generative model to generate a query response to the query request; and
providing the query response to the client device in response to the query request.
However, Madisetti discloses generating, a query prompt by combining the semantic information of the query request from the converted text string, the text description of the image from the image grounding information, and a directive to generate a response to the query request using the semantic information of the query request and the text description of the image(paragraph[0323]-paragraph[0332], the reference describes generating prompts based on a sequence of prompts. The Examiner interprets a sequence of prompts being the semantic and grounding information outputs from Kharbanda. The reference describes retrieving results from the generated prompts in paragraph[0332].). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to a person having ordinary skill in the art to which said subject matter pertains to have modified the teachings of Kharbanda with the teachings of Madisetti to combine data to generate a prompt which would result in the claim invention. The skilled artisan would have been motivated to improve the teachings of Kharbanda with the teachings of Madisetti to efficiently process multi-level generative AI and large language models form generative AI applications (Madisetti: paragraph[0032]).
The combination of Kharbanda and Madisetti do not appear to explicitly disclose
providing the query prompt to a lightweight text-based large generative model to generate a query response to the query request; and
providing the query response to the client device in response to the query request.
However, Lee discloses providing the query prompt to a lightweight text-based large generative model to generate a query response to the query request (Figure 2, paragraph[0069], the reference describes providing an response.); and
providing the query response to the client device in response to the query request (Figure 2, paragraph[0164], the reference describes providing a result.). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to a person having ordinary skill in the art to which said subject matter pertains to have modified the teachings of Kharbanda with the teachings of Madisetti and Lee to combine query data to input into a model which would result in the claim invention. The skilled artisan would have been motivated to improve the teachings of Kharbanda with the teachings of Madisetti and Lee to efficiently process a queries associated with a videos using models (Lee: paragraph[0005]).
Claim 15
As to claim 15, the combination of Kharbanda, Madisetti, and Lee discloses all the elements in claim 14, as noted above, and Kharbanda further disclose wherein the audio input portion is provided to the speech-to-text model concurrently with providing the image input portion to the lightweight image metadata model(Figure 1, paragraph[0047], the reference describes inputting audio and video data to the multimodal at the same time.).
Claim 16
As to claim 16, the combination of Kharbanda, Madisetti, and Lee discloses all the elements in claim 15, as noted above, and Kharbanda further disclose wherein providing the query response to the client device comprises:
causing a generation of a visual overlay element that includes the query response (paragraph[0110], the reference describes using an overlay in response to query results.); and
causing a display of the visual overlay element over the image captured by the client device(paragraph[0110], the reference describes using an overlay in response to query results.).
Claim 17
As to claim 17, the combination of Kharbanda, Madisetti, and Lee discloses all the elements in claim 16, as noted above, and Kharbanda further disclose wherein the visual overlay element includes text and image links that are included in the query response received from the lightweight text-based large generative model (Figure 5E, paragraph[0050], the reference describes generating overlays with product links.).
Claim 19
As to claim 19, the combination of Kharbanda, Madisetti, and Lee discloses all the elements in claim 18, as noted above, and Kharbanda further disclose wherein the query response received from the lightweight large generative model includes an answer to the query request without additional metadata not being provided to the client device (paragraph[0071], the reference describes using a lightweight model with the need for additional data.).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Kharbanda et al. U.S. Patent Publication (2024/0403362; hereinafter: Kharbanda) in view of Madisetti et al. U.S. Patent Publication (2025/0328560; hereinafter: Madisetti) and further in view of Lee et al. U.S. Patent Publication (2025/0190503; hereinafter: Lee) and further in view of Hansson et al. U.S. Patent Publication (2012/0047135; hereinafter: Hansson)
Claim 5
As to claim 5, the combination of Kharbanda, Madisetti, and Lee discloses all the elements in claim 1, as noted above, but do not appear to explicitly disclose wherein the threshold time to provide the query response to the client device in response to the query request is half.
However, Hansson discloses wherein the threshold time to provide the query response to the client device in response to the query request is half (paragraph[0085], the reference describes providing a query result in half a second (i.e., request is half, as claimed).). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to a person having ordinary skill in the art to which said subject matter pertains to have modified the teachings of Kharbanda with the teachings of Madisetti, Lee, and Hansson to have query results within a threshold amount of time which would result in the claim invention. The skilled artisan would have been motivated to improve the teachings of Kharbanda with the teachings of Madisetti, Lee, and Hansson to efficiently provide search results for query suggestions if a prediction criterion is met (Hansson: paragraph[0029]).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Kharbanda et al. U.S. Patent Publication (2024/0403362; hereinafter: Kharbanda) in view of Madisetti et al. U.S. Patent Publication (2025/0328560; hereinafter: Madisetti) and further in view of Lee et al. U.S. Patent Publication (2025/0190503; hereinafter: Lee) and further in view of Klingler et al. U.S. Patent Publication (2025/0182366; hereinafter: Klingler)
Claim 8
As to claim 8, the combination of Kharbanda, Madisetti, and Lee discloses all the elements in claim 1, as noted above, but do not appear to explicitly disclose sending the additional image grounding information to the lightweight large generative model in an additional prompt.
However, Klingler discloses ending the additional image grounding information to the lightweight large generative model in an additional prompt (paragraph[0294], the reference describes creating an additional prompt with grounding information.). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to a person having ordinary skill in the art to which said subject matter pertains to have modified the teachings of Kharbanda with the teachings of Madisetti, Lee, and Klingler to create additional prompts which would result in the claim invention. The skilled artisan would have been motivated to improve the teachings of Kharbanda with the teachings of Madisetti, Lee, and Klingler to efficiently build more complex interaction patterns with the interactive agent base on user intent (Klingler: paragraph[0009]).
Claims 9, 10, 11, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Kharbanda et al. U.S. Patent Publication (2024/0403362; hereinafter: Kharbanda) in view of Madisetti et al. U.S. Patent Publication (2025/0328560; hereinafter: Madisetti) and further in view of Lee et al. U.S. Patent Publication (2025/0190503; hereinafter: Lee) and further in view of Kundel et al. U.S. Patent Publication (2024/0354321; hereinafter: Kundel)
Claim 9
As to claim 9, the combination of Kharbanda, Madisetti, and Lee discloses all the elements in claim 8, as noted above, but do not appear to explicitly disclose sending the additional image grounding information to the lightweight large generative model in an additional prompt.
However, Kundel discloses, wherein the additional prompt is generated based on receiving an indication from the lightweight large generative model, in response to the prompt, that the grounding information is insufficient to answer the query request (paragraph[0034], the reference describes analyzing a query prompt to determine if the response is a definitive response. If the response is not definitive (i.e., insufficient to answer the query request, as claimed), then an additional query prompt is created.). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to a person having ordinary skill in the art to which said subject matter pertains to have modified the teachings of Kharbanda with the teachings of Madisetti, Lee, and Kundel to detect inefficient responses which would result in the claim invention. The skilled artisan would have been motivated to improve the teachings of Kharbanda with the teachings of Madisetti, Lee, and Kundel to efficiently facilitate interactivity with artificial intelligent platforms (Kundel: paragraph[0002]).
Claims 10 and 20
As to claims 10 and 20, the combination of Kharbanda, Madisetti, and Lee discloses all the elements in claim 1, as noted above, Lee further disclose wherein:
the lightweight large generative model provides the query response based on receiving additional image grounding information from a visual-based large generative model (Figure 2, paragraph[0069], the reference describes models providing responses based on additional data in a prompt.); and
the visual-based large generative model generates the additional image grounding information from the second input portion (Figure 2, paragraph[0069], the reference describes using video with addition data to generate a query prompt.).
Lee does not appear to explicitly disclose
the lightweight large generative model initially determines that the prompt is insufficient to answer the query request;
However, Kundel discloses the lightweight large generative model initially determines that the prompt is insufficient to answer the query request (paragraph[0034], the reference describes analyzing a query prompt to determine if the response is a definitive response. If the response is not definitive (i.e., insufficient to answer the query request, as claimed), then an additional query prompt is created.). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to a person having ordinary skill in the art to which said subject matter pertains to have modified the teachings of Kharbanda with the teachings of Madisetti, Lee, and Kundel to detect inefficient responses which would result in the claim invention. The skilled artisan would have been motivated to improve the teachings of Kharbanda with the teachings of Madisetti, Lee, and Kundel to efficiently facilitate interactivity with artificial intelligent platforms (Kundel: paragraph[0002]).
Claim 11
As to claim 11, the combination of Kharbanda, Madisetti, Lee, and Kundel discloses all the elements in claim 10, as noted above, and Kundel further disclose wherein the lightweight large generative model receives the additional image grounding information from the visual-based large generative model in response to sending the prompt to the visual-based large generative model (paragraph[0027], the reference describes using a lightweight generative model to gather a response.).
Final Rejection
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAWAUNE A CONYERS whose telephone number is (571)270-3552. The examiner can normally be reached on M-F 8:00am-4:30pm EST. EST.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Neveen Abel-Jalil can be reached on (571) 270-0474. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAWAUNE A CONYERS/Primary Examiner, Art Unit 2152 May 29, 2026
/DAWAUNE A CONYERS/Primary Examiner, Art Unit 2152 February 24, 2024
Prosecution Insights