DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Amendment filed on 07/21/2026 has been entered. Claims 1-20 are pending.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Huang et al. (Huang), US Patent Application Publication No. US 2023/0112710 A1, in view of Morariu et al. (Morariu), US Patent Application Publication No. US 2023/0376687 A1, and further in view of Akbari et al. (Akbari), US Patent Application Publication No. US 2024/0037335.
As independent claim 1, Huang discloses a method, comprising:
encoding a text sample with a text encoder, wherein the encoding creates an embedded text sample in a conjoined embedding space (Figure 3 and paragraphs [0085]-[0087], [0090]: machine learning model 350 that is trained on training dataset that includes training sample, wherein the training sample includes a textual description of a GUI, for example, first encoder (text encoder) using a bidirectional encoder representations from transformers (BERT) to generate embedding vectors to represent textual descriptions, and embedding text description into a common embedding space (conjoined embedding space) that is populated with both text embedding);
encoding an electronic data sample with a multi-modal encoder, wherein the encoding the electronic data sample creates an embedded electronic data sample in the conjoined embedding space, and wherein the multi-modal encoder includes a plurality of expert networks (Figure 3 and paragraphs [0085]-[0088]: machine learning model 350 that is trained on training dataset that includes training sample, wherein the training sample includes an image of a GUI (electronic data sample), for example, second encoder (a multi-modal encoder) to generate embedding vectors to represent the image of the GUI (electronic data sample), and embedding the image of the GUI into the common embedding space (conjoined embedding space) that is populated with both text embedding and UI embedding);
training the multi-modal encoder to determine a similarity score between a target text and an electronic data segment, wherein the multi-modal encoder is trained by machine learning based on comparing labeled training samples and corresponding predictions generated by the multi-modal encoder (Figures 3 and paragraphs [0088]-[0091]: the training sample includes the image of the GUI that represents different graphical elements (electronic data segments) that make up the GUI; paragraph [0055]: the training process uses a validation set to evaluate the performance of the machine learning model, wherein the validation set includes multiple samples similar to the training samples of the training dataset; paragraphs [0076]-[0077]: generating prediction data for the predicted GUI); and
outputting the multi-modal encoder configured to query a data repository of documents for one or more selected documents that match a natural language text input (paragraph [0078]: if the user wants a GUI with a login page with two buttons, the textual description can be “login page with two buttons”.
Huang discloses encoding an electronic data sample with the second encoder, wherein the electronic data sample includes an image of a GUI, and the image of the GUI includes different segments (Figures 1 & 3 and paragraph [0033]). Thus, one of ordinary skill in the art would interpret that the image of the GUI can be an EDI document, and the second encoder is a multi-modal EDI model (paragraphs [0084]-[0085]. To support Examiner’s interpretation, Morariu discloses generating and using a multi-modal multi-granular model to analyze document regions of multiple sizes (e.g., granularities) and generate data (e.g., feature vectors) suitable for use in performing multiple tasks, and the multi-modal multi-granular model (multi-modal EDI encoder) can be used in connection with one or more other machine learning models to perform various tasks such as the page-level document extraction, region-level entity recognition, and/or token-level token classification (Morariu, paragraphs [0018], [0021] and Figure 1). Morariu further discloses the document can be an invoice or receipt, which is an EDI document (paragraph [0022]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system of Huang to include a multi-modal multi-granular model (multi-modal EDI encoder) to transform the input includes features extracted from a page-level, region-level, and word-level of the document (EDI document), as taught by Morariu. Morariu suggests that the multi-modal multi-granular model, advantageously generates data that can be used to perform multiple distinct task (e.g., entity recognition, document classification, etc.) at multiple granularities which reduces model storage cost and maintenance as well as improves performance over conventional system as a result of the model obtaining information from regions at different granularities (Morariu, paragraph [0022]).
Huang and Morariu, however, do not wherein training the multi-modal EDI encoder comprises training the text encoder and the multi-modal EDI encoder in parallel to train the multi-modal EDI encoder to encode text and EDI data comprising similar informational content with similar embeddings.
In the same field of endeavor, Akbari discloses methods, systems, and computer-readable media for bi-modal generation of natural language (NL) and artificial neural network architectures (NA) (Abstract). Akbari further discloses in Figure 2A and paragraphs [0102] and [0104] an example of Generative Bi-Modal NL+NA Model 200 includes an encoder 206, which comprises Text Encoder 210 and Architecture Encoder (multi-modal EDI encoder). Akbari further discloses in Figure 2B and paragraph [0106] that the text encoder 210 processes natural language information to generate word embeddings, and the neural network architecture encoder 220 processes neural network architecture information to generate graph encodings, and a cross transformer encoder 240 process word embeddings and graph encodings to generate joint embeddings 242. Akbari further discloses the cross transformer encoder 240 enables joint learning of NL (e.g., textual) and NA (i.e., architectural ) embeddings, in this example represented as word embeddings and graph embeddings respectively, and sharing of learning signals between both modalities, wherein the word and graph embeddings are processed simultaneously to create their corresponding joint embeddings 242 (paragraph [0113]). Akbari further discloses the joint embedding 242 include two types of cross encoded embeddings: word embeddings cross encoded with architecture information, and graph embedding cross encoded with natural language information, such that both cross encoded embedding are vectors of the same length, or in some examples, the two types of cross encoded embeddings of the joint embeddings may be concatenated together to form the joint embedding (paragraph [0113]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the systems of Huang and Morariu to include wherein training the multi-modal EDI encoder comprises training the text encoder and the multi-modal EDI encoder in parallel to train the multi-modal EDI encoder to encode text and EDI data comprising similar informational content with similar embeddings, as taught by Akbari for the purpose of supporting learning and inference related to neural network architectures designed for learning any type of data (image, video, audio, text, etc.) (Akbari, paragraph [0024]).
As to dependent claim 2, Huang, Morariu and Akbari disclose determining an EDI type of the EDI sample (Huang, paragraph [0040]; Morariu, Figure 1, paragraphs [0018], [0021]-[0022]); and
routing the EDI sample to one of the plurality of expert networks based on the EDI type (Huang, paragraph [0090]; Morariu, paragraph [0035]).
As to dependent claim 3, Huang, Morariu and Akbari disclose wherein the routing is performed by a router layer of the multi-modal EDI encoder, wherein the router layer is configured to route an EDI segment to one of the plurality of expert networks in response to the determining the EDI type of the EDI sample (Morariu, paragraph [0035]).
As to dependent claim 4, Huang, Morariu and Akbari disclose wherein the multi-modal EDI encoder includes a self-attention layer configured to weight each element of the EDI segment based on a context, the weight corresponding to a relative importance of each element with respect to each other element (Huang, paragraph [0090]; Morariu, paragraphs [0019], [0036]).
As to dependent claim 5, Huang, Morariu and Akbari disclose wherein the training includes iteratively adjusting parameters of the multi-modal EDI encoder based on a cross-entropy between the labeled training samples and corresponding predictions generated by the multi-modal EDI encoder (Huang, paragraphs [0011], [0053]; Morariu, paragraph [0060]).
As to dependent claim 6, Huang, Morariu and Akbari disclose inputting a target text to the multi-modal EDI encoder; and identifying, by the multi-modal EDI encoder, one or more EDI segments embedded in the embedding space in response to determining a match between the target text and one or more EDI segments (Huang, paragraph [0115]).
As to dependent claim 7, Huang, Morariu and Akbari disclose further comprising: inputting a plurality of new EDI segments to the multi-modal EDI encoder (Huang, paragraphs [0064], [0069]); and
generating by the multi-modal EDI encoder one or more clusters based on determining similarities between pairs of the plurality of new EDI segments (Huang, paragraphs [0064], [0069]).
As to dependent claim 8, Huang, Morariu and Akbari disclose wherein the text encoder is a transformer-based encoder (Huang, paragraph [0090]).
As to dependent claim 11, Huang, Morariu and Akbari disclose wherein the training includes computing a cross-entropy based on a prediction generated by the multi-modal EDI encoder for each of the plurality of EDI samples (Huang, paragraphs [0043], [0046]; Morariu, paragraph [0060]).
Claims 9-10 and 12-13 are system claims that contain similar limitations of claims 1-2 and 6-7, respectively. Therefore, claims 9-10 and 12-13 are rejected under the same rationale.
Claims 14-20 are computer program product claims that contain similar limitations of claims 1-7, respectively. Therefore, claims 14-20 are rejected under the same rationale.
Response to Arguments
Applicant’s arguments and amendments filed on 07/21/2026 have been fully considered but they are not deemed fully persuasive. Applicant’s arguments with respect to claims 1-20 have been considered but are moot in view of the new ground(s) of rejection as explained here below, necessitated by Applicant’s substantial amendment (i.e., wherein training the multi-modal EDI encoder comprises training the text encoder and the multi-modal EDI encoder in parallel to train the multi-modal EDI encoder to encode text and EDI data comprising similar informational content with similar embeddings) to the claims which significantly affected the scope thereof. Please see the rejection above with newly cited prior art Akbari.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHAU T NGUYEN whose telephone number is (571)272-4092. The examiner can normally be reached on Monday-Friday from 8am to 5pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula, can be reached at telephone number 5712724128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated-interview-request-air-form.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/CHAU T NGUYEN/Primary Examiner, Art Unit 2145
