DETAILED ACTION
Receipt of Applicant’s Amendment, filed May 12, 2026 is acknowledged.
Claims 1, 2, 12, 13, 17, and 18 were amended.
Claim 21 was newly added.
Claims 1-21 are pending in this office action.
Claim Objections
Claims 1-21 are objected to because of the following informalities. Appropriate correction is required.
With regard to claims 1, 12, and 17, claim 1 recites “generating, using a first generative artificial intelligence (AI) model, a first text corpus comprising an AI-generated summary of the non-text portion… generating a cross-modal embedding of the document as a function of a combination of at least the AI-generated summary of the non-text portion and the non-text portion, the cross-modal embedding comprising a vector representation of the first text corpus and the non-text portion”. Claims 12 and 17 appear to recite substantially similar language and are rejected based upon similar reasoning and rational.
The distinction between the ‘first text corpus’ and the ‘summary of the non-text portion’ is unclear. The claim appears to recite two distinct labels (e.g. ‘first text corpus’ and ‘summary of non-text portion’) to refer to the same claim element. The claim explicitly recites that the first text corpus comprises the summary. Yet the use of two distinct labels suggests that the terms are expected to refer to distinct elements.
For examination purposes the claimed first text corpus has ben identified as being the summary of the non-text portion. It is suggested that the claim be amended to use a single claim label for each unique claim element (e.g. amend the claim to use ‘first text corpus’ for all references to this claim element). Should applicant be intending to recite two distinct elements, it is suggested that the claim be amended to clarify the distinction between the ‘first text corpus’ and the ‘summary of the non-text portion’.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-5, 7-9, 11-15, 17-19, and 21 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Morato [2025/0371901].
With regard to claim 1 Morato teaches A method, comprising:
accessing a document as multi-modal document 402 (Morato, ¶90 “By way of example, a multi-modal document 402 may include one or more text segments 404 and/or one or more embedded media segments 406 that complement the one or more text segments 404.”; Figure 4, 406) containing a text portion as text segments 404 (Id) and a non-text portion as media segments 406 (Id);
generating, using a first generative artificial intelligence (AI) model as the first channel, e.g. generative model 424 including a LLM (Morato, ¶83 “For instance, a first channel of the multi-modal embedding model architecture may be configured to generate an embedding from an embedded media segment”; ¶113 “A generative model 424 may include any type of model configured, trained, and/or the like to generate natural language text from media, such as an embedded media segment, as described herein. For example, the generative model 424 may include an LLM, such as a GPT model. In some examples, the generative model 424 may include a GPT-4 with vision model and/or any other machine learning model with multimodal capabilities.”; Figure 4, 424), a first text corpus as a textual media description 426 (Morato, ¶109 “A textual media description 426 may be generated, using a generative model 424, for the embedded media segment 406 based on the embedded media segment 406 and the one or more media attributes.”; Figure 4, 426) comprising an AI-generated summary as a textual description of the media segment generated by the generative model(Morato, ¶70 “In some embodiments, the term "textual media description" refers to generative text for an embedding media segment. A textual media description, for example, may include one or more media text segments that describe, in text, an embedding media segment. In some examples, a textual media description may be generated by inputting the embedded media segment to a generative model that is configured to transfer the embedded media segment to a text description.”; ¶76 “The natural language text, for example, may summarize one or more portions of a context field”) of the non-text portion as based on embedded media segment 406 (Morato, ¶109; Figure 4 412); and
generating a cross-modal embedding of the document (Morato, ¶57 “multi-model section embedding … a multi-modal section embedding may be generated for each semantic section entity of a plurality of semantic section entities within a multi-modal document using a cross-modal retrieval approach”) as a function of a combination as a co-learned embedding representation (¶61 “For example, a multi-modal document embedding may describe a common embedding space for each of the segments of a multi-modal document. The multi-modal document embedding, for example, may include a co-learned embedding representation for each text segment and/ or embedded media segment within the multi-modal document. In some examples, the multi-modal document embedding may be generated by applying a multi-modal embedding model to a plurality of text and media features of the plurality of text and embedded media segments, such that: multi-modal document embedding=multi-modal embedding modal (text features, media features).”) of at least the AI-generated summary of the non-text portion as media features including media attributes (¶61 “such that: multi-modal document embedding=multi-modal embedding modal (text features, media features).”; ¶62; ¶68 “A media feature, for example, may include one or more media attributes and/or a textual media description for embedded media segments.”) and the non-text portion as the media features including textural media descriptions (¶61 “such that: multi-modal document embedding=multi-modal embedding modal (text features, media features).”; ¶68 “A media feature, for example, may include one or more media attributes and/or a textual media description for embedded media segments.”), the cross-modal embedding comprising a vector representation as multi-modal section embedding (Morato, ¶98 “a multi-modal section embedding is an encoded vector that describes one or more of the text segments 404 and embedded media segments 406 within a semantic section entity”) of the first text corpus as the text segments 404 (Id) and the non-text portion as the embedded media segments 406, e.g. media features (Id; ¶61; ¶62); and
storing the cross-modal embedding as using cross-modal retrieval requires the data to be stored in order for it to be retrievable (Morato, ¶57) such as via an index (Morato, ¶141 “At a second, image preprocessing stage 912, images from the multi-modal document 402 may be extracted, preprocessed to remove noise, blur, and/or other image defects, and stored with an image index for later retrieval.”) in a vector database (Morato, ¶29; ¶40 “the volatile storage or memory media may be used to store at least portions of the databases, database instances, database management systems, data, applications, programs, program modules, code (source code, object code, byte code, compiled code, interpreted code, machine code)”);
wherein the method is performed by at least one device including a hardware processor as one or more processors (Morato, ¶4).
With regard to claim 2 Morato further teaches accessing a second as a second of the plurality of documents (Morato, ¶63 “a plurality of training multi-modal documents”; ¶67 “unstructured healthcare documents”) document containing a second text portion as text segment 404 for the second document (Morato, ¶90 “By way of example, a multi-modal document 402 may include one or more text segments 404 and/or one or more embedded media segments 406 that complement the one or more text segments 404.”; Figure 4, 406; ¶63), a graphic data representation of a first type as a table (Morato, ¶90 “the variety of modalities may contain text, images, tables, and/or other graphical elements”), and an image as images (Id);
generating, using a first language model (Morato, ¶66 “an enriched text segment is generated using a natural language technique, such as a named entity recognition (NER) model”), a second text corpus corresponding to the second text portion as a modified text segment to emphasize terms within the text segment (Morato, ¶66 “the term "enriched text segment" refers to a modified text segment that is modified to standardize and emphasize one or more terms within the text segment that are predictive within a prediction domain.”);
generating, using a second multimodal model as OCRFeeder, Teseract, layout parse (Morato, ¶93 “An embedded media segment may include any type of media content, including images, audio, tables, flowcharts, and/or the like. In some examples, an embedded media segment may be extracted from the multi-modal document 402 using one or more media extraction techniques, such as Optical Character Recognition Feeder (OCRFeeder), Tesseract, layout parse, and/or the like.”) configured for the first type of graphic data representation as the table (Id), a third text corpus describing information contained in the graphic data representation as a textual media description 426 for the second document (Morato, ¶109 “A textual media description 426 may be generated, using a generative model 424, for the embedded media segment 406 based on the embedded media segment 406 and the one or more media attributes.”; ¶63);
generating, using a third multimodal model configured for images, a fourth text corpus as the media attribute for the second document (Morato, ¶111 “a media attribute is a data entity that describes an observation for an embedded media segment. A media attribute, for example, may include the content ( e.g., an image representation, etc.) of an embedded media segment and/or one or more contextual observations, such as a page index, a location, a title, a document title, and/or the like.”; ¶63) corresponding to contents of the image as image representation (Id); and
generating a cross-modal embedding (Morato, ¶57 “multi-model section embedding … a multi-modal section embedding may be generated for each semantic section entity of a plurality of semantic section entities within a multi-modal document using a cross-modal retrieval approach”) of the second text corpus, the third text corpus, the graphic data representation, the fourth text corpus, and the image as the multi-modal section embedding describes one or more of the identified segments for the second document (Morato, ¶98 “a multi-modal section embedding is an encoded vector that describes one or more of the text segments 404 and embedded media segments 406 within a semantic section entity”; ¶63).
With regard to claim 3 Morato further teaches the first type of graphic data representation (Morato, ¶90 “the variety of modalities may contain text, images, tables, and/or other graphical elements”)comprises at least one of: a graph, a chart as flowcharts (Morato, ¶93), a table as table (Morato, ¶90), a plot, a diagram, a frequency distribution, a histogram, a pictograph, and a knowledge graph.
With regard to claims 4, 14, and 19, Morato further teaches wherein:
the non-text portion comprises a graphic data representation (Morato, ¶93 “An embedded media segment may include any type of media content, including images, audio, tables, flowcharts, and/or the like.”);
the method further comprising:
generating the first text corpus by extracting as extracting using OCR (Morato, ¶56), from the graphic data representation (Morato, ¶111 “a media attribute is a data entity that describes an observation for an embedded media segment. A media attribute, for example, may include the content ( e.g., an image representation, etc.) of an embedded media segment and/or one or more contextual observations, such as a page index, a location, a title, a document title, and/or the like.”; ¶112 “A textual media description, for example, may include one or more media text segments that describe, in text, an embedding media segment.”), at least one of: a title text as a title (Id), an axis label text, a body text, a caption text, and a data point value.
With regard to claims 5 and 15 Morato further teaches extracting a data point from the non-text portion using a data point extraction model as extracting a table from the media content region using OCR (Morato, ¶56 “As one example, an image, table, and/or the like may be extracted by applying layout parse to identify the media content region in the multi-modal document, extracting the media content region, and then storing the extracted content as an embedded media segment.”);
collecting feedback as feedback (Morato, ¶32 “In some examples, feedback ( e.g., evaluation data, ground truth data, etc.) from the use of the machine learning model may be recorded by the predictive computing entity 106. In some examples, the feedback may be provided to the one or more external computing entities 108 to continuously train the machine learning model over time. In some examples, the feedback may be leveraged by the predictive computing entity 106 to continuously train the machine learning model over time.”) for an answer generated based on the document being retrieved (Morato, ¶25 “The architecture 100 includes a computing system 101 configured to receive request, such as generative text requests, from client computing entities 102, process the requests to generate generative text outputs, and provide the generated text outputs to the client computing entities 102.”) from the vector database using a cross-modal embedding (Morato, ¶57 “multi-model section embedding … a multi-modal section embedding may be generated for each semantic section entity of a plurality of semantic section entities within a multi-modal document using a cross-modal retrieval approach”) of the data point as extracting a table from the media content region using OCR (Morato, ¶56 “As one example, an image, table, and/or the like may be extracted by applying layout parse to identify the media content region in the multi-modal document, extracting the media content region, and then storing the extracted content as an embedded media segment.”); and
fine-tuning the data point extraction model by providing the feedback as training data to the data point extraction model (Morato, ¶32 “In some examples, feedback ( e.g., evaluation data, ground truth data, etc.) from the use of the machine learning model may be recorded by the predictive computing entity 106. In some examples, the feedback may be provided to the one or more external computing entities 108 to continuously train the machine learning model over time. In some examples, the feedback may be leveraged by the predictive computing entity 106 to continuously train the machine learning model over time.”).
With regard to claim 7 Morato further teaches the non-text portion comprises a graphic data representation (Morato, ¶90 “the variety of modalities may contain text, images, tables, and/or other graphical elements”);
the method further comprising:
generating the first text corpus by:
extracting as extracting using OCR (Morato, ¶56) a data point from the graphic data representation as extracting a table from the media content region using OCR (Morato, ¶56 “As one example, an image, table, and/or the like may be extracted by applying layout parse to identify the media content region in the multi-modal document, extracting the media content region, and then storing the extracted content as an embedded media segment.”); and
summarizing the data point using one or more models as generating the textual media description (Morato, ¶109 “A textual media description 426 may be generated, using a generative model 424, for the embedded media segment 406 based on the embedded media segment 406 and the one or more media attributes.”) configured to:
extract one or more values as the URL content itself (Morato, ¶111 “A media attribute, for example, may include the content ( e.g., an image representation, etc.) of an embedded media segment and/or one or more contextual observations, such as a page index, a location, a title, a document title, and/or the like. By way of example, a media attribute for an embedded media segment may include (i) a content index attribute that identifies a location (e.g., URL, etc.) of an embedded media segment extracted from the multi-modal document 402,”) corresponding to the data point as the media segment extracted (Id); and
extract one or more labels corresponding to the values as the media attribute, e.g. the ‘location’ label (Id); and including one or more descriptions of the one or more values and the one or more labels in the first text corpus (Morato, ¶134 “The media attributes 504 may be input to the generative model 424 to generate a textual media description”).
With regard to claim 8 Morato further teaches generating the first text corpus comprises:
extracting a text component corresponding to the non-text portion from the text portion based on the text component referencing (Morato, ¶51 “the variety of modalities may contain text, images, tables, and/or other graphical elements that may cross-reference, build upon, and/or otherwise relate to one another to provide complementary information for a predictive domain”) the non-text portion (Morato, ¶136 “As shown, the embedded media segment 406 may include a plurality of graphical elements ( e.g., arrows, boxes, etc.) that characterize text within the embedded media segment 406”); and
generating the cross-modal embedding (Morato, ¶57 “multi-model section embedding … a multi-modal section embedding may be generated for each semantic section entity of a plurality of semantic section entities within a multi-modal document using a cross-modal retrieval approach”) comprises generating a vector representation (Morato, ¶98 “a multi-modal section embedding is an encoded vector that describes one or more of the text segments 404 and embedded media segments 406 within a semantic section entity”) of the text component, the first text corpus and the non-text portion as text features, and media features including the textual media description and media attributes (¶61 “such that: multi-modal document embedding=multi-modal embedding modal (text features, media features).”; ¶68 “A media feature, for example, may include one or more media attributes and/or a textual media description for embedded media segments.”).
With regard to claim 9 Morato further teaches extracting the first text corpus from an image contained in the non-text portion as extracting text within an embedded image (Morato, ¶136 “As shown, the embedded media segment 406 may include a plurality of graphical elements ( e.g., arrows, boxes, etc.) that characterize text within the embedded media segment 406”); ¶93 “An embedded media segment may include any type of media content, including images, audio, tables, flowcharts, and/or the like. In some examples, an embedded media segment may be extracted from the multi-modal document 402 using one or more media extraction techniques, such as Optical Character Recognition Feeder (OCRFeeder), Tesseract, layout parse, and/or the like.”);
extracting a second text corpus from a graphic data representation contained in the non-text portion as extracting text from within an embedded table or flowchart (Morato, ¶136 “As shown, the embedded media segment 406 may include a plurality of graphical elements ( e.g., arrows, boxes, etc.) that characterize text within the embedded media segment 406”); ¶93 “An embedded media segment may include any type of media content, including images, audio, tables, flowcharts, and/or the like. In some examples, an embedded media segment may be extracted from the multi-modal document 402 using one or more media extraction techniques, such as Optical Character Recognition Feeder (OCRFeeder), Tesseract, layout parse, and/or the like.”); and
applying a first weighting to the first text corpus as the similarity score of the particular segment to the co-learned embedding(Morato, ¶119 “a topic similarity score is a metric that is reflective of a semantic similarity between a semantic section entity and a segment (e.g., text segment, embedded media segment, etc.) of the multi-modal document 402. In some examples, the topic similarity score for a particular segment may be based on a comparison between a co-learned embedding representation for the particular segment and a topic embedding for a semantic section entity.”) and a second weighting to the second text corpus as the similarity score of the particular segment to the co-learned embedding (Morato, ¶119 “a topic similarity score is a metric that is reflective of a semantic similarity between a semantic section entity and a segment (e.g., text segment, embedded media segment, etc.) of the multi-modal document 402. In some examples, the topic similarity score for a particular segment may be based on a comparison between a co-learned embedding representation for the particular segment and a topic embedding for a semantic section entity.”);
wherein the cross-modal embedding comprises a vector representation of the first text corpus, the second text corpus, the image, and the graphic data representation as fusing the segments across modalities when the similarity score is above a threshold similarity score (Morato, ¶119 “As described herein, the plurality of segments may be leveraged to generate a multi-modal unstructured rule 434 corresponding to the semantic section entity that fuses knowledge across multiple information modalities.”; ¶57 “(ii) one or more co-learned embedding representations are identified based on a comparison between the topic similarity scores and a similarity threshold”).
With regard to claim 11 Morato teaches the document comprises a text portion as text (Morato, ¶90 “the variety of modalities may contain text, images, tables, and/or other graphical elements”), a graphic data representation as a table or other graphical elements (Id), and an image as images (Id);
the method further comprising:
chunking the document into a first chunk comprising the text portion as text (Morato, ¶90 “the variety of modalities may contain text, images, tables, and/or other graphical elements”), a second chunk comprising the graphic data representation as a table or other graphical elements (Id), and a third chunk comprising the image as images (Id).
With regard to claim 12 Morato teaches One or more non-transitory computer readable media comprising instructions (Morato, ¶21) which, when executed by one or more hardware processors (Morato, ¶4), cause performance of operations comprising:
accessing a document as multi-modal document 402 (Morato, ¶90 “By way of example, a multi-modal document 402 may include one or more text segments 404 and/or one or more embedded media segments 406 that complement the one or more text segments 404.”; Figure 4, 406) containing a text portion as text segments 404 (Id) and a non-text portion as media segments 406 (Id);
generating, using a first generative artificial intelligence (AI) model as the first channel, e.g. generative model 424 including a LLM (Morato, ¶83 “For instance, a first channel of the multi-modal embedding model architecture may be configured to generate an embedding from an embedded media segment”; ¶113 “A generative model 424 may include any type of model configured, trained, and/or the like to generate natural language text from media, such as an embedded media segment, as described herein. For example, the generative model 424 may include an LLM, such as a GPT model. In some examples, the generative model 424 may include a GPT-4 with vision model and/or any other machine learning model with multimodal capabilities.”; Figure 4, 424), a first text corpus as a textual media description 426 (Morato, ¶109 “A textual media description 426 may be generated, using a generative model 424, for the embedded media segment 406 based on the embedded media segment 406 and the one or more media attributes.”; Figure 4, 426) comprising an AI-generated summary as a textual description of the media segment generated by the generative model(Morato, ¶70 “In some embodiments, the term "textual media description" refers to generative text for an embedding media segment. A textual media description, for example, may include one or more media text segments that describe, in text, an embedding media segment. In some examples, a textual media description may be generated by inputting the embedded media segment to a generative model that is configured to transfer the embedded media segment to a text description.”; ¶76 “The natural language text, for example, may summarize one or more portions of a context field”) of the non-text portion as based on embedded media segment 406 (Morato, ¶109; Figure 4 412); and
generating a cross-modal embedding of the document (Morato, ¶57 “multi-model section embedding … a multi-modal section embedding may be generated for each semantic section entity of a plurality of semantic section entities within a multi-modal document using a cross-modal retrieval approach”) as a function of a combination as a co-learned embedding representation (¶61 “For example, a multi-modal document embedding may describe a common embedding space for each of the segments of a multi-modal document. The multi-modal document embedding, for example, may include a co-learned embedding representation for each text segment and/ or embedded media segment within the multi-modal document. In some examples, the multi-modal document embedding may be generated by applying a multi-modal embedding model to a plurality of text and media features of the plurality of text and embedded media segments, such that: multi-modal document embedding=multi-modal embedding modal (text features, media features).”) of at least the AI-generated summary of the non-text portion as media features including media attributes (¶61 “such that: multi-modal document embedding=multi-modal embedding modal (text features, media features).”; ¶62; ¶68 “A media feature, for example, may include one or more media attributes and/or a textual media description for embedded media segments.”) and the non-text portion as the media features including textural media descriptions (¶61 “such that: multi-modal document embedding=multi-modal embedding modal (text features, media features).”; ¶68 “A media feature, for example, may include one or more media attributes and/or a textual media description for embedded media segments.”), the cross-modal embedding comprising a vector representation as multi-modal section embedding (Morato, ¶98 “a multi-modal section embedding is an encoded vector that describes one or more of the text segments 404 and embedded media segments 406 within a semantic section entity”) of the first text corpus as the text segments 404 (Id) and the non-text portion as the embedded media segments 406, e.g. media features (Id; ¶61; ¶62); and
storing the cross-modal embedding as using cross-modal retrieval requires the data to be stored in order for it to be retrievable (Morato, ¶57) such as via an index (Morato, ¶141 “At a second, image preprocessing stage 912, images from the multi-modal document 402 may be extracted, preprocessed to remove noise, blur, and/or other image defects, and stored with an image index for later retrieval.”) in a vector database (Morato, ¶29; ¶40 “the volatile storage or memory media may be used to store at least portions of the databases, database instances, database management systems, data, applications, programs, program modules, code (source code, object code, byte code, compiled code, interpreted code, machine code)”).
With regard to claims 13 and 18 Morato further teaches
accessing a second as a second of the plurality of documents (Morato, ¶63 “a plurality of training multi-modal documents”; ¶67 “unstructured healthcare documents”) document containing a second text portion as text segment 404 (Morato, ¶90 “By way of example, a multi-modal document 402 may include one or more text segments 404 and/or one or more embedded media segments 406 that complement the one or more text segments 404.”; Figure 4, 406), a graphic data representation of a first type as a table (Morato, ¶90 “the variety of modalities may contain text, images, tables, and/or other graphical elements”), and an image as images (Id);
generating, using a first language model (Morato, ¶66 “an enriched text segment is generated using a natural language technique, such as a named entity recognition (NER) model”), a second text corpus corresponding to the second text portion as a modified text segment to emphasize terms within the text segment (Morato, ¶66 “the term "enriched text segment" refers to a modified text segment that is modified to standardize and emphasize one or more terms within the text segment that are predictive within a prediction domain.”);
generating, using a second multimodal model as OCRFeeder, Teseract, layout parse (Morato, ¶93 “An embedded media segment may include any type of media content, including images, audio, tables, flowcharts, and/or the like. In some examples, an embedded media segment may be extracted from the multi-modal document 402 using one or more media extraction techniques, such as Optical Character Recognition Feeder (OCRFeeder), Tesseract, layout parse, and/or the like.”) configured for the first type of graphic data representation as the table (Id), a third text corpus describing information contained in the graphic data representation as a textual media description 426 for the second document (Morato, ¶109 “A textual media description 426 may be generated, using a generative model 424, for the embedded media segment 406 based on the embedded media segment 406 and the one or more media attributes.”; ¶63);
generating, using a third multimodal model configured for images, a fourth text corpus as the media attribute for the second document (Morato, ¶111 “a media attribute is a data entity that describes an observation for an embedded media segment. A media attribute, for example, may include the content ( e.g., an image representation, etc.) of an embedded media segment and/or one or more contextual observations, such as a page index, a location, a title, a document title, and/or the like.”; ¶63) corresponding to contents of the image as image representation (Id); and
generating a cross-modal embedding (Morato, ¶57 “multi-model section embedding … a multi-modal section embedding may be generated for each semantic section entity of a plurality of semantic section entities within a multi-modal document using a cross-modal retrieval approach”) of the second text corpus, the third text corpus, the graphic data representation, the fourth text corpus, and the image as the multi-modal section embedding describes one or more of the identified segments for the second document (Morato, ¶98 “a multi-modal section embedding is an encoded vector that describes one or more of the text segments 404 and embedded media segments 406 within a semantic section entity”; ¶63).
With regard to claim 17 Morato teaches A system, comprising:
at least one device including a hardware processor (Morato, ¶4);
the system being configured to perform operations (Morato, ¶20) comprising:
accessing a document as multi-modal document 402 (Morato, ¶90 “By way of example, a multi-modal document 402 may include one or more text segments 404 and/or one or more embedded media segments 406 that complement the one or more text segments 404.”; Figure 4, 406) containing a text portion as text segments 404 (Id) and a non-text portion as media segments 406 (Id);
generating, using a first generative artificial intelligence (AI) model as the first channel, e.g. generative model 424 including a LLM (Morato, ¶83 “For instance, a first channel of the multi-modal embedding model architecture may be configured to generate an embedding from an embedded media segment”; ¶113 “A generative model 424 may include any type of model configured, trained, and/or the like to generate natural language text from media, such as an embedded media segment, as described herein. For example, the generative model 424 may include an LLM, such as a GPT model. In some examples, the generative model 424 may include a GPT-4 with vision model and/or any other machine learning model with multimodal capabilities.”; Figure 4, 424), a first text corpus as a textual media description 426 (Morato, ¶109 “A textual media description 426 may be generated, using a generative model 424, for the embedded media segment 406 based on the embedded media segment 406 and the one or more media attributes.”; Figure 4, 426) comprising an AI-generated summary as a textual description of the media segment generated by the generative model(Morato, ¶70 “In some embodiments, the term "textual media description" refers to generative text for an embedding media segment. A textual media description, for example, may include one or more media text segments that describe, in text, an embedding media segment. In some examples, a textual media description may be generated by inputting the embedded media segment to a generative model that is configured to transfer the embedded media segment to a text description.”; ¶76 “The natural language text, for example, may summarize one or more portions of a context field”) of the non-text portion as based on embedded media segment 406 (Morato, ¶109; Figure 4 412); and
generating a cross-modal embedding of the document (Morato, ¶57 “multi-model section embedding … a multi-modal section embedding may be generated for each semantic section entity of a plurality of semantic section entities within a multi-modal document using a cross-modal retrieval approach”) as a function of a combination as a co-learned embedding representation (¶61 “For example, a multi-modal document embedding may describe a common embedding space for each of the segments of a multi-modal document. The multi-modal document embedding, for example, may include a co-learned embedding representation for each text segment and/ or embedded media segment within the multi-modal document. In some examples, the multi-modal document embedding may be generated by applying a multi-modal embedding model to a plurality of text and media features of the plurality of text and embedded media segments, such that: multi-modal document embedding=multi-modal embedding modal (text features, media features).”) of at least the AI-generated summary of the non-text portion as media features including media attributes (¶61 “such that: multi-modal document embedding=multi-modal embedding modal (text features, media features).”; ¶62; ¶68 “A media feature, for example, may include one or more media attributes and/or a textual media description for embedded media segments.”) and the non-text portion as the media features including textural media descriptions (¶61 “such that: multi-modal document embedding=multi-modal embedding modal (text features, media features).”; ¶68 “A media feature, for example, may include one or more media attributes and/or a textual media description for embedded media segments.”), the cross-modal embedding comprising a vector representation as multi-modal section embedding (Morato, ¶98 “a multi-modal section embedding is an encoded vector that describes one or more of the text segments 404 and embedded media segments 406 within a semantic section entity”) of the first text corpus as the text segments 404 (Id) and the non-text portion as the embedded media segments 406, e.g. media features (Id; ¶61; ¶62); and
storing the cross-modal embedding as using cross-modal retrieval requires the data to be stored in order for it to be retrievable (Morato, ¶57) such as via an index (Morato, ¶141 “At a second, image preprocessing stage 912, images from the multi-modal document 402 may be extracted, preprocessed to remove noise, blur, and/or other image defects, and stored with an image index for later retrieval.”) in a vector database (Morato, ¶29; ¶40 “the volatile storage or memory media may be used to store at least portions of the databases, database instances, database management systems, data, applications, programs, program modules, code (source code, object code, byte code, compiled code, interpreted code, machine code)”.
With regard to claim 21 Marato further teaches wherein the first generative AI model as the first channel, e.g. generative model 424 including a LLM (Morato, ¶83 “For instance, a first channel of the multi-modal embedding model architecture may be configured to generate an embedding from an embedded media segment”; ¶113 “A generative model 424 may include any type of model configured, trained, and/or the like to generate natural language text from media, such as an embedded media segment, as described herein. For example, the generative model 424 may include an LLM, such as a GPT model. In some examples, the generative model 424 may include a GPT-4 with vision model and/or any other machine learning model with multimodal capabilities.”; Figure 4, 424) generates the first text corpus as a textual media description 426 (Morato, ¶109 “A textual media description 426 may be generated, using a generative model 424, for the embedded media segment 406 based on the embedded media segment 406 and the one or more media attributes.”; Figure 4, 426) responsive to an input prompt (¶74 “the term "unstructured rule model prompt" refers to a generative model prompt for instructing an LLM to generate a multi-modal unstructured rule. … the unstructured rule model prompt may include one or more prompt fields”) that includes the non-text portion as inputting the embedded media segment into a generative model (¶61; ¶70 “generated by inputting the embedded media segment to a generative model that is configured to transfer the embedded media segment to a text description.”) and one or more text elements extracted from the non-text portion as the prompt field including the context field which includes the textual media description (¶74 “The second field may include a context field that provides content for generating the multi-modal unstructured rule in accordance with the set of instructions. In some examples, the context field may include an enriched text segment and a textual media description for each segment of the multi-modal document that corresponds to a particular semantic section entity.”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 6, 16, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Morato in view of Abramson ]2023/0178076].
With regard to claims 6 and 16 Morato further teaches collecting feedback as feedback (Morato, ¶32 “In some examples, feedback ( e.g., evaluation data, ground truth data, etc.) from the use of the machine learning model may be recorded by the predictive computing entity 106. In some examples, the feedback may be provided to the one or more external computing entities 108 to continuously train the machine learning model over time. In some examples, the feedback may be leveraged by the predictive computing entity 106 to continuously train the machine learning model over time.”) for an answer generated based on the document being retrieved (Morato, ¶25 “The architecture 100 includes a computing system 101 configured to receive request, such as generative text requests, from client computing entities 102, process the requests to generate generative text outputs, and provide the generated text outputs to the client computing entities 102.”) from the vector database using a cross-modal embedding (Morato, ¶57 “multi-model section embedding … a multi-modal section embedding may be generated for each semantic section entity of a plurality of semantic section entities within a multi-modal document using a cross-modal retrieval approach”);
fine-tuning as continuously training (Morato, ¶32) the first multimodal model using [[ as shared attention layer (Morato, ¶84 “embedding and transformer architecture with a shared attention layer”) by providing the feedback as training data to the first multimodal model (Morato, ¶32 “In some examples, feedback ( e.g., evaluation data, ground truth data, etc.) from the use of the machine learning model may be recorded by the predictive computing entity 106. In some examples, the feedback may be provided to the one or more external computing entities 108 to continuously train the machine learning model over time. In some examples, the feedback may be leveraged by the predictive computing entity 106 to continuously train the machine learning model over time.”).
Morato does not explicitly teach a multi-head attention layer.
Abramson teaches the first multimodal modal as the multi-modal transformer (Abramson, ¶122 “the multi-modal Transformer includes one or more self-attention layers that each have one or more self-attention heads, e.g., so that “applying self-attention” includes processing the input through the one or more self-attention layers.”) using multi-head attention (¶122 “That is, each self-attention layer can perform either single-head self-attention and therefore have only one attention head or can perform multi-head attention and therefore have multiple heads that each perform self-attention in parallel”).
It would have been obvious to one of ordinary skill to which said subject matter pertains at the time the invention was filed to have implemented the attention layer taught by Morato (¶84) using multiple heads as it enables a means of combining output of multiple outputs (Abramson ¶122 “The self-attention layer can then combine the outputs of the multiple heads to generate an output of the attention mechanism for the self-attention layer, e.g., by summing, averaging, or concatenating the outputs and then optionally applying a linear transformation to the result. Each self-attention layer can also perform any of a variety of other operations, e.g., layer normalization, position-wise feed-forward neural network computations, residual connection operations, and so on.”). The proposed combination enables a simple modification of the self-attention layer to able the image embeddings and the text embeddings to be combined (Abramson, ¶121, ¶122).
With regard to claim 20 Morato further teaches
extracting a data point from the non-text portion using a data point extraction model as extracting a table from the media content region using OCR (Morato, ¶56 “As one example, an image, table, and/or the like may be extracted by applying layout parse to identify the media content region in the multi-modal document, extracting the media content region, and then storing the extracted content as an embedded media segment.”);
collecting first feedback as feedback (Morato, ¶32 “In some examples, feedback ( e.g., evaluation data, ground truth data, etc.) from the use of the machine learning model may be recorded by the predictive computing entity 106. In some examples, the feedback may be provided to the one or more external computing entities 108 to continuously train the machine learning model over time. In some examples, the feedback may be leveraged by the predictive computing entity 106 to continuously train the machine learning model over time.”) for an answer generated based on the document being retrieved (Morato, ¶25 “The architecture 100 includes a computing system 101 configured to receive request, such as generative text requests, from client computing entities 102, process the requests to generate generative text outputs, and provide the generated text outputs to the client computing entities 102.”) from the vector database using a cross-modal embedding (Morato, ¶57 “multi-model section embedding … a multi-modal section embedding may be generated for each semantic section entity of a plurality of semantic section entities within a multi-modal document using a cross-modal retrieval approach”) of the data point as extracting a table from the media content region using OCR (Morato, ¶56 “As one example, an image, table, and/or the like may be extracted by applying layout parse to identify the media content region in the multi-modal document, extracting the media content region, and then storing the extracted content as an embedded media segment.”);
fine-tuning the data point extraction model by providing the first feedback as training data to the data point extraction model (Morato, ¶32 “In some examples, feedback ( e.g., evaluation data, ground truth data, etc.) from the use of the machine learning model may be recorded by the predictive computing entity 106. In some examples, the feedback may be provided to the one or more external computing entities 108 to continuously train the machine learning model over time. In some examples, the feedback may be leveraged by the predictive computing entity 106 to continuously train the machine learning model over time.”) collecting second feedback for an answer generated based on the document being retrieved from the vector database using the cross-modal embedding as continuous training over time (Id); and
fine-tuning as continuously training (Morato, ¶32) the first multimodal model using [[as shared attention layer (Morato, ¶84 “embedding and transformer architecture with a shared attention layer”) by providing the second feedback as training data to the first multimodal model (Morato, ¶32 “In some examples, feedback ( e.g., evaluation data, ground truth data, etc.) from the use of the machine learning model may be recorded by the predictive computing entity 106. In some examples, the feedback may be provided to the one or more external computing entities 108 to continuously train the machine learning model over time. In some examples, the feedback may be leveraged by the predictive computing entity 106 to continuously train the machine learning model over time.”).
Morato does not explicitly teach a multi-head attention layer.
Abramson teaches the first multimodal modal as the multi-modal transformer (Abramson, ¶122 “the multi-modal Transformer includes one or more self-attention layers that each have one or more self-attention heads, e.g., so that “applying self-attention” includes processing the input through the one or more self-attention layers.”) using multi-head attention (¶122 “That is, each self-attention layer can perform either single-head self-attention and therefore have only one attention head or can perform multi-head attention and therefore have multiple heads that each perform self-attention in parallel”).
It would have been obvious to one of ordinary skill to which said subject matter pertains at the time the invention was filed to have implemented the attention layer taught by Morato (¶84) using multiple heads as it enables a means of combining output of multiple outputs (Abramson ¶122 “The self-attention layer can then combine the outputs of the multiple heads to generate an output of the attention mechanism for the self-attention layer, e.g., by summing, averaging, or concatenating the outputs and then optionally applying a linear transformation to the result. Each self-attention layer can also perform any of a variety of other operations, e.g., layer normalization, position-wise feed-forward neural network computations, residual connection operations, and so on.”). The proposed combination enables a simple modification of the self-attention layer to able the image embeddings and the text embeddings to be combined (Abramson, ¶121, ¶122).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Morato in view of PonjouTasse [2020/0104318]].
With regard to claim 10 Morato further teaches the document as multi-modal document 402 (Morato, ¶90 “By way of example, a multi-modal document 402 may include one or more text segments 404 and/or one or more embedded media segments 406 that complement the one or more text segments 404.”; Figure 4, 406) comprises a first document chunk including the text portion as words, sentences (Morato, ¶53 “A text segment, for example, may include one or more characters, words, sentences, and/or the like that are arranged in a textual format. A”) and a second document chunk including the non-text portion as images, audio, tables, flowcharts (Morato, ¶56 “An embedded media segment may include any type of media content, including images, audio, tables, flowcharts, and/or the like”); and
first text corpus comprises a textual description of the non-text portion (Morato, ¶90 “By way of example, a multi-modal document 402 may include one or more text segments 404 and/or one or more embedded media segments 406 that complement the one or more text segments 404.);
the method further comprising:
accessing a query (Morato, ¶19 “a database query or search language”; ¶25 “The architecture 100 includes a computing system 101 configured to receive request, such as generative text requests, from client computing entities 102, process the requests to generate generative text outputs, and provide the generated text outputs to the client computing entities 102.”);
[[
(Morato, ¶98 “multi-modal section embedding may be generated for each semantic section entity of a plurality of semantic section entities 408 within the multi-modal document 402 using a cross-modal retrieval approach”); and
generating a response to the query as generative text outputs to be provided to the user (Morato, ¶25 “The architecture 100 includes a computing system 101 configured to receive request, such as generative text requests, from client computing entities 102, process the requests to generate generative text outputs, and provide the generated text outputs to the client computing entities 102.”) based on the first document chunk and the second document chunk (Morato, ¶98 “multi-modal section embedding may be generated for each semantic section entity of a plurality of semantic section entities 408 within the multi-modal document 402 using a cross-modal retrieval approach”).
Morato does not explicitly teach identifying the first document chunk based on a semantic similarity between the text portion and the query; identifying the second document chunk based on a vector search using the query.
PonjouTasse teaches identifying the first document chunk based on a semantic similarity (PonjouTasse, ¶73 “To retrieve the top K objects in a collection based on the multimodal query, the method of this embodiment returns objects or images Ik such that the smallest Euclidean distance between E(Ik) and E(I;npu,, ta, ... , tn_1).”; Claim 20 “A method of searching a collection of objects based on visual and semantic similarity of the unified representations of the collection of objects”) between the text portion (Claim 20 “determining one or more objects in the collection of objects having a spatially close vector representation to the unified descriptor for the search query.”; Claim 22 “determining which one or more of the images, three dimensional objects, text and combinations have a spatially close vector representation to the vector representation for the query”) and the query (Claim 20 “determining a unified descriptor for the search query, where the search query comprises both image or object data and word or tag data;”);
identifying the second document chunk as a second of the top K objects (PonjouTasse, ¶73 “To retrieve the top K objects in a collection based on the multimodal query, the method of this embodiment returns objects or images Ik such that the smallest Euclidean distance between E(Ik) and E(I;npu,, ta, ... , tn_1).”; Claim 20 “A method of searching a collection of objects based on visual and semantic similarity of the unified representations of the collection of objects”) based on a vector search as the search of the unified representations (Id) using the query (Claim 20 “determining a unified descriptor for the search query, where the search query comprises both image or object data and word or tag data;”)
It would have been obvious to one of ordinary skill to which said subject matter pertains at the time the invention was filed to have implemented the multi-model embeddings created by Morato (¶141) to facilitate a search of multi-modal data as taught by PonjouTasse as it yields the predictable results of enabling a search engine to make use of the vectors to for later retrieval (Morato, ¶141). One of ordinary skill in the art would recognize that the search system taught by PonjouTasse would be expected to be able to search the multi-modal vectors generated by Morato’s system as both are making use of a unified representation of multi0modal data within a common vector space (PonjouTasse, ¶11; Monato, ¶61).
Response to Arguments
Applicant's arguments filed May 12, 2026 have been fully considered but they are not persuasive. All the arguments regarding the newly added limitations are addressed in the above rejections.
With regard to the 112 rejections, these rejections have been withdrawn in view of the claim amendment.
Applicant argues that the generated textual description taught by Marato is applied after the media segment has been embedded, and as such Marato does not suggest incorporating this textual segment into the embedding for the media segment.
In response, it should be noted that Marato details that the media feature is evaluated in step 424, and the textual media description 426 (e.g. the claimed summary) is fed into the Multi-Model Embedding models 420, 422, 428, in which the cross-model embedding is generated (See, ¶57).
Marato explicitly recites “multi-modal document embedding=multi-modal embedding modal (text features, media features)” (See ¶61). Where these media features are defined as including both the media attributes themselves and the textual media description (e.g. the summary of the non-text portion) (See ¶68). As such, one of ordinary skill in the art would recognize that the vector embeddings generated are generated as a function of all of the attributes/features (e.g. text features, the media attributes, and the textual description of the media).
Applicant’s arguments do not appear to apply to the claim mapping put forth by the office. The “Embedded Media segment” discussed in 406 is not the embedding that is generating the cross-modal embedding as applicant appears to have argued.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMANDA WILLIS whose telephone number is (571)270-7691. The examiner can normally be reached Monday-Friday 8am-2pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ajay Bhatia can be reached at 571-272-3906. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AMANDA L WILLIS/ Primary Examiner, Art Unit 2156
Prosecution Insights