DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Allowable Subject Matter
Claims 7-8, 13-15, and 17-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-6, 9-12, 16 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Gu (US 2025/0094456).
With respect to claim 10 (similarly claims 1 and 16), Gu teaches a system (e.g. the system of Figs 1-3 [0019], [0043] and Fig 6 [0068]) comprising:
one or more memory devices (e.g. a storage subsystem 624, Fig 6 [0069]); and
one or more processors (e.g. at least one processor 614 Fig 6 [0069]) coupled to the one or more memory devices (e.g. coupled to storage subsystem 624), configured to cause the system to:
generate (e.g. generate [0033]-[0034], Fig 5 S503 [0061]), utilizing a multimodal large language model (e.g. utilizing a GM response generation engine 128 [0034], [0061]), a hidden answer embedding from an answer obtained in response to a prompt relative to a digital document (e.g. embeddings [0034], [0061] from an answer, see Fig 3 A-B obtained in response to query 250 Fig 2, Fig 5 S501 [0060] relative to a document 251 [0044]), the digital document comprising text and image elements (e.g. the document 251 of [0044] comprising text and image elements, [0023], [0043]-[0044]);
generate, utilizing the multimodal large language model, hidden text embeddings from the text of the digital document and hidden image embeddings from the image elements of the digital document (e.g. Fig 5 S509 A-C [0063]-[0064] suggest generate, utilizing the multimodal large language model, hidden text embeddings from the text of the digital document and hidden image embeddings from the image elements of the digital document);
based on comparing the hidden text embeddings with the hidden answer embedding and comparing the hidden image embeddings with the hidden answer embedding (e.g. based on comparing the hidden text embeddings with the hidden answer embedding and comparing the hidden image embeddings with the hidden answer embedding as suggested in [0052], [0056]-[0058]), determine at least one of a text attribution or an image attribution responsive to the prompt to query the digital document (e.g. determine at least one of a text attribution or an image attribution responsive to the prompt to query the digital document as suggested in [0036], Fig 4 [0056]-[0058]); and
based on at least one of the text attribution or the image attribution (e.g. based on at least one of the text attribution or the image attribution as suggested in [0036], [0056]-[0058]), provide, for display in the digital document of a client device, at least one of the text attribution within the digital document or the image attribution within the digital document (e.g. render [0024] at least one of the text attribution within the digital document or the image attribution within the digital document as suggested in [0041], [0049], [0052]).
With respect to claim 11 (similarly claim 6), Gu teaches the system of claim 10, wherein the one or more processors are configured to cause the system to:
receive, from a client device, a selection of at least a portion of the answer obtained in response to the prompt relative to the digital document (e.g. receive a selection of textual fragment 376 Fig 3B [0055] i.e. at least a portion of the answer obtained in response to the prompt relative to the digital document); and
utilize the selection of the at least a portion of the answer as an anchor to identify a subset of hidden state embeddings from a plurality of hidden state embeddings generated from intermediate layers of the multimodal large language model, wherein the subset of hidden state embeddings corresponds to tokens of the selection of the at least a portion of the answer (e.g. the user may select textual fragment 376 to cause a pop-up window 378 or other similar auxiliary interface (e.g., a new browser tab that opens, and in some instances scrolls to, the contradicting content) to be rendered at client device 310. The window 378 may depict information about content such as a search result document or snippet(s)/fragment(s) from the document that tend to contradict textual fragment 376. [0055] suggest utilize the selection of the at least a portion of the answer as an anchor to identify a subset of hidden state embeddings from a plurality of hidden state embeddings generated from intermediate layers of the multimodal large language model, wherein the subset of hidden state embeddings corresponds to tokens of the selection of the at least a portion of the answer).
With respect to claim 12, Gu teaches the system of claim 11, wherein the one or more processors are configured to cause the system to generate a hidden answer embedding for the anchor by averaging the subset of hidden state embeddings (e.g. the annotations made in Fig 3 A-B [0065] suggest to generate a hidden answer embedding for the anchor by averaging the subset of hidden state embeddings).
With respect to claim 2, Gu teaches the computer-implemented method of claim 1, wherein generating the answer to the prompt comprises determining, in the digital document, one or more text spans and one or more regions of a digital image that provide support to the answer (e.g. The window 378 may depict information about content such as a search result document or snippet(s)/fragment(s) from the document that tend to contradict textual fragment 376. [0055]).
With respect to claim 3, Gu teaches the computer-implemented method of claim 1, wherein generating the image attribution of the image element in the digital document comprises generating the image attribution that indicates a portion of the digital document for one of a natural image, a chart, an infographic, a scanned digital document, or an image with multilingual text (e.g. an image with multilingual text, see Fig 3 A-B).
With respect to claim 4, Gu teaches the computer-implemented method of claim 1, wherein generating the answer to the prompt relative to the digital document occurs simultaneously with generating the image attribution of the image element and the text attribution of the text (e.g. [0051], [0064] suggest generating the answer to the prompt relative to the digital document occurs simultaneously with generating the image attribution of the image element and the text attribution of the text).
With respect to claim 5, Gu teaches the computer-implemented method of claim 1, wherein receiving the prompt relative to the digital document comprises providing, for display on a graphical user interface of a client device, the digital document in tandem with a prompt panel for the client device to submit a question about the digital document (e.g. Fig 3A item 372 [0053] suggest receiving the prompt relative to the digital document comprises providing, for display on a graphical user interface of a client device, the digital document in tandem with a prompt panel for the client device to submit a question about the digital document).
With respect to claim 9, Gu teaches the computer-implemented method of claim 1, wherein providing the image attribution and the text attribution for display in the digital document of the client device comprises: highlighting a relevant text span in the digital document that is responsive to the selection of the selection of the at least a portion of the answer (e.g. For instance, a sentence that is corroborated may be visually emphasized (e.g., highlighted) using one color, another sentence that is contradicted may be visually emphasized using another color, and so forth [0041], see also [0052], [0054], claim 8); and
outlining a relevant image region in the digital document that is responsive to the selection of the selection of the at least a portion of the answer (e.g. [0041], [0052], [0054], claim 8 suggest outlining a relevant image region in the digital document that is responsive to the selection of the selection of the at least a portion of the answer).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IBRAHIM SIDDO whose telephone number is (571)272-4508. The examiner can normally be reached 9:00-5:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Akwasi Sarpong can be reached at 5712703438. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/IBRAHIM SIDDO/Primary Examiner, Art Unit 2681
Prosecution Insights