Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on December 18, 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement has been considered by the examiner.
Election/Restrictions
Applicant’s election without traverse of Group 1, claims 12 – 20 in the reply filed on 04/29/2026 is acknowledged. Non-elected Group 2, claims 1-11 is canceled via amendment.
Examiner acknowledges that Applicant stands correct that the present application is a 371 National Stage application and should have been evaluated under the “Lack of Unity” test. Nevertheless, lack of unity exists as demonstrated below because Groups 1 and 2 do not fall under one of the five combination categories noted below. The two groups are processes that share special technical feature of using a large language model for training. However, one process is training for text and the other process is for training multi-modal text and image. Each process together with said special technical feature exhibits a lack of unity.
LACK OF UNITY OF INVENTION
As provided in 37 CFR 1.475(a), a national stage application shall relate to one invention only or to a group of inventions so linked as to form a single general inventive concept (“requirement of unity of invention”). Where a group of inventions is claimed in a national stage application, the requirement of unity of invention shall be fulfilled only when there is a technical relationship among those inventions involving one or more of the same or corresponding special technical features. The expression “special technical features” shall mean those technical features that define a contribution which each of the claimed inventions, considered as a whole, makes over the prior art.
The determination whether a group of inventions is so linked as to form a single general inventive concept shall be made without regard to whether the inventions are claimed in separate claims or as alternatives within a single claim. See 37 CFR 1.475(e).
When Claims Are Directed to Multiple Categories of Inventions:
As provided in 37 CFR 1.475 (b), a national stage application containing claims to different categories of invention will be considered to have unity of invention if the claims are drawn only to one of the following combinations of categories:
(1) A product and a process specially adapted for the manufacture of said product; or
(2) A product and a process of use of said product; or
(3) A product, a process specially adapted for the manufacture of the said product, and a use of the said product; or
(4) A process and an apparatus or means specifically designed for carrying out the said process; or
(5) A product, a process specially adapted for the manufacture of the said product, and an apparatus or means specifically designed for carrying out the said process.
Otherwise, unity of invention might not be present. See 37 CFR 1.475 (c).
Accordingly, restriction is required under 35 U.S.C. 121 and 372 as the groupings outlined below do not fall under the aforementioned categories 1-5.
This application contains the following inventions or groups of inventions which are not so linked as to form a single general inventive concept under PCT Rule 13.1.
I. Claim 1 – 11, drawn to machine learning model using only text input, classified in G06N 20/00.
II. Claim 12 – 20, drawn to different input data image and text, e.g. multi-modal recognition, classified in G06V10/811
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 12, 13 and 15 – 20 are rejected under 35 U.S.C 103 as being unpatentable over Jiang et al. Patent Application Publication No. CN-116627637-A (hereinafter Jiang) in view of Park et al. US Patent Application Publication No. US-20240419727-A1 (hereinafter Park) and further in view of Li Patent Application Publication No. US-20240161520-A1 (hereinafter Li).
Regarding claim 12, Jiang discloses method for running an edge-cloud fusion-aware visual prompt large language model (Jiang discloses in [0005]), comprising: training a visual-prompt image encoder, a large language model feature encoder and a small feature extraction model (Jiang in [0020] discloses, “The training process of the BERT model is completed in the cloud, and the cloud transmits the trained model to the edge end for deployment and use”. Jiang in [0076 – 0079] discloses, “The model training process by the fast-RCNN is exemplified as follows: (214) The collected preprocessed data is partitioned into 70% training data set and 30% validation data set ... inputting a picture into a network, and extracting features through a convolution network to obtain a feature map” wherein fast RCNN equates to small feature extraction model); inputting knowledge-based image prompts (Jiang in [0057] discloses, “Generating task information data to be executed and feature data related to a user and a scene by the edge through a machine learning model and a knowledge graph”) and knowledge-based text prompts to the visual-prompt image encoder (Jiang in [0058] discloses, “specifically, the voice text data collected by the robot is used as the input of a deep learning model, and the intention classification calculation is carried out through a BERT model”); inputting an image prompt to the visual-prompt image encoder in the edge device to generate a visual representation (Jiang in [0073] discloses, “And carrying out person detection on the image data through a fast-RCNN model to obtain the pixel coordinate value of the person in the picture”); comparing the (Jiang in [0055] discloses about comparing with knowledge base, “The purpose of text conversion in the initial stage is to reduce the storage and processing pressure of the server, and simultaneously facilitate the subsequent processing of NLP by the edge terminal based on characters, help the system understand the true intention of the user, contrast the knowledge base, make reasoning decision, generate task and issue”); and if the third similarity score is larger than a third threshold, then inputting the (Jiang in [0068] discloses, “And if the answer is judged to be a question and answer type, generating an answer according to the content in the knowledge graph. If no answer exists in the knowledge graph, generating a question answer through LLM”. Generating question answer with LLM if no answer exists in the knowledge graph equates to the if the threshold is larger it would be generated by small feature extraction model (RCNN)).
Jiang doesn’t disclose about the following limitation as further recited in the claim.
Park discloses the large language model feature encoder respectively in an edge device to generate a plurality of knowledge-based image embeddings and a plurality of knowledge-based text embeddings (Park in [0089] discloses, “Exemplary embodiments of the present disclosure enable user-specific embedding vectors external to underlying LLM machination. Here, the edge devices capture instantaneous user context in a variety of modalities (e.g., images, text, speech, etc.); the aggregator converts the edge information into a persistent history of user context with user-specific embeddings (e.g., “my keys”, etc.)”); concatenating the plurality of knowledge-based image embeddings and the plurality of knowledge-based text embeddings to generate a plurality of concatenated knowledge-based embeddings; building a large language model database in the edge device according to the plurality of concatenated knowledge-based embeddings (Park in [0239] discloses, “Within the context of the present disclosure, the aggregator device combines data from multiple data sources into an input sequence that can be processed within a transformer model. In one specific implementation, the input sequence is based on information gathered across different modalities of data ... While the described implementations are presented in the context of a large language models (LLMs) transformer, the concepts could be readily adapted to large multi-modal models (LMMs) and/or other fundamental models”);
It would have been obvious to one of ordinary skill in art before the effective filling date of the claimed invention to integrate the technique of Park into the system of Jiang because it would allow the system to create stored knowledge entries that contain both visual and text meaning together which would make the LLM model perform better.
Jiang and Park in the combination do not disclose about the following limitation as further recited in the claim.
Li discloses a fully connected linear projector, inputting the visual representation to the fully connected linear projector in the edge device to generate an image query embedding (Li in [0050] discloses, “As shown in both FIGS. 4 A- 4 B, a fully-connected (FC) layer 410 is connected to linearly project the output query embeddings Z 408 into embeddings 412 having the same dimension as the word embedding of the LLM 130”); inputting a text prompt to the large language model feature encoder in the edge device to generate a text query embedding (Li in [0052] discloses, “FIG. 4 B adopts an encoder-decoder-based LLM comprising the LLM encoder 130 a and LLM decoder 130 b . For encoder-decoder LLMs 130 b , the LLM encoder 130 a may receive both a prefix text 414 and the projected embedding 412 and encode a concatenation of 412 and 414 into an encoded representation”); concatenating the image query embedding and the text query embedding to generate a concatenated query embedding (Li in [0052] discloses, “the LLM encoder 130 a may receive both a prefix text 414 and the projected embedding 412 and encode a concatenation of 412 and 414 into an encoded representation”. Furthermore, Li in [0100] and [0100] discloses about concatenated query embedding, “The text encoder encodes a combination of the transformed representation, the input text, and a prefix text into the combined representation”); the concatenated query embedding (Li discloses about concatenated query embedding in [0052]).
It would have been obvious to one of ordinary skill in art before the effective filling date of the claimed invention to integrate the technique of Li into the system of Jiang in view of Park because it would allow the system to align visual information with text information.
Summary of Citations (Li)
Paragraph [0043]; “For example, the ITC module 231 may compute an image-text similarity based on the query representation Z from the image transformer 210 and the text representation t from the text transformer 220”.
Paragraph [0050]; “As shown in both FIGS. 4 A- 4 B, a fully-connected (FC) layer 410 is connected to linearly project the output query embeddings Z 408 into embeddings 412 having the same dimension as the word embedding of the LLM 130”.
Paragraph [0052]; “FIG. 4 B adopts an encoder-decoder-based LLM comprising the LLM encoder 130 a and LLM decoder 130 b . For encoder-decoder LLMs 130 b , the LLM encoder 130 a may receive both a prefix text 414 and the projected embedding 412 and encode a concatenation of 412 and 414 into an encoded representation”.
Paragraph [0087]; “An image-text similarity may be computed based on the query embeddings and the text representation”.
Summary of Citations (Park)
Paragraph [0089]; “Exemplary embodiments of the present disclosure enable user-specific embedding vectors external to underlying LLM machination. Here, the edge devices capture instantaneous user context in a variety of modalities (e.g., images, text, speech, etc.); the aggregator converts the edge information into a persistent history of user context with user-specific embeddings (e.g., “my keys”, etc.)”.
Paragraph [0239]; “Within the context of the present disclosure, the aggregator device combines data from multiple data sources into an input sequence that can be processed within a transformer model. In one specific implementation, the input sequence is based on information gathered across different modalities of data ... While the described implementations are presented in the context of a large language models (LLMs) transformer, the concepts could be readily adapted to large multi-modal models (LMMs) and/or other fundamental models”.
Summary of Citations (Jiang)
Paragraph [0005]; “deep learning is used for training a model at a cloud end, knowledge maps are produced and managed at the cloud end, each deep learning algorithm model is operated at an edge end, and character feature data, scene feature data and task instruction data are obtained through calculation. And taking various characteristic data and knowledge graph data as input, calculating reasonable output through a cloud LLM, returning to the edge end, and finally sending the reasonable output to the navigation robot”.
Paragraph [0020]; “ “The training process of the BERT model is completed in the cloud, and the cloud transmits the trained model to the edge end for deployment and use”.
Paragraph [0055]; “The purpose of text conversion in the initial stage is to reduce the storage and processing pressure of the server, and simultaneously facilitate the subsequent processing of NLP by the edge terminal based on characters, help the system understand the true intention of the user, contrast the knowledge base, make reasoning decision, generate task and issue”.
Paragraph [0057]; “Generating task information data to be executed and feature data related to a user and a scene by the edge through a machine learning model and a knowledge graph”.
Paragraph [0058]; “specifically, the voice text data collected by the robot is used as the input of a deep learning model, and the intention classification calculation is carried out through a BERT model”.
Paragraph [0068]; “And if the answer is judged to be a question and answer type, generating an answer according to the content in the knowledge graph. If no answer exists in the knowledge graph, generating a question answer through LLM”.
Paragraph [0073]; “And carrying out person detection on the image data through a fast-RCNN model to obtain the pixel coordinate value of the person in the picture”.
Paragraph [0076 – 0079]; “The model training process by the fast-RCNN is exemplified as follows: (214) The collected preprocessed data is partitioned into 70% training data set and 30% validation data set ... inputting a picture into a network, and extracting features through a convolution network to obtain a feature map”.
Regarding claim 13, Jiang in the combination discloses the method of claim 12, and the small feature extraction model is training the visual-prompt image encoder, and the small feature extraction model in a cloud (Jiang in [0076 – 0079] discloses about small feature extraction model fast R-CNN, “The model training process by the fast-RCNN is exemplified as follows: (214) The collected preprocessed data is partitioned into 70% training data set and 30% validation data set ... inputting a picture into a network, and extracting features through a convolution network to obtain a feature map”. Additionally, Jiang in [0102] discloses about small feature extraction model in a cloud, “The edge server pulls the trained deep learning algorithm models and corresponding versions of the corresponding virtual images from the cloud”).
Li further discloses training the visual-prompt image encoder (Li in [0026] discloses, “The multi-modal vision-language model that comprises an image encoder 110 , a query Transformer 120 and a (large) language model (LLM) 130 may be trained by the vision-language pretraining framework 100”), the fully connected linear projector(Li discloses fully-connected (FC) layer 410 is connected to linearly project in [0050]), the large language model feature encoder (Li discloses in [0026]), the fully connected linear projector (Li discloses fully-connected (FC) layer 410 is connected to linearly project in [0050]), the large language model feature encoder (Li discloses in [0026]).
Summary of Citations (Jiang)
Paragraph [0076 – 0079]; “The model training process by the fast-RCNN is exemplified as follows: (214) The collected preprocessed data is partitioned into 70% training data set and 30% validation data set ... inputting a picture into a network, and extracting features through a convolution network to obtain a feature map”.
Paragraph [0102]; “The edge server pulls the trained deep learning algorithm models and corresponding versions of the corresponding virtual images from the cloud”.
Summary of Citations (Li)
Paragraph [0026]; “The multi-modal vision-language model that comprises an image encoder 110 , a query Transformer 120 and a (large) language model (LLM) 130 may be trained by the vision-language pretraining framework 100”.
Paragraph [0050]; “As shown in both FIGS. 4 A- 4 B, a fully-connected (FC) layer 410 is connected to linearly project the output query embeddings Z 408 into embeddings 412 having the same dimension as the word embedding of the LLM 130”.
Regarding claim 15, Li in the combination discloses the method of claim 12, wherein the visual-prompt image encoder comprises a vision transformer (Li in [0033] discloses, “In one implementation, the image encoder 119 may be pre-trained vision transformer models”).
Summary of Citations (Li)
Paragraph [0033]; “In one implementation, the image encoder 119 may be pre-trained vision transformer models”.
Regarding claim 16, Li in the combination disclose the method of claim 12, further comprising: training the vision transformer to use a transformer architecture and a self-attention mechanism for extracting visual features from the image prompts (Li in [0033] discloses, “In one implementation, the image encoder 119 may be pre-trained vision transformer models”. Additionally, Li in [0031] discloses, “FIG. 2 , the query Transformer 120 consists of two transformer submodules 210 and 220 that share the same self-attention layers 211 and 221 : (1) an image transformer 210 that interacts with the frozen image encoder 110 for visual feature extraction”).
Summary of Citations (Li)
Paragraph [0031]; “As shown in FIG. 2 , the query Transformer 120 consists of two transformer submodules 210 and 220 that share the same self-attention layers 211 and 221 : (1) an image transformer 210 that interacts with the frozen image encoder 110 for visual feature extraction”.
Paragraph [0033]; “In one implementation, the image encoder 119 may be pre-trained vision transformer models”.
Regarding claim 17, Li in the combination discloses the method of claim 16, wherein the visual-prompt image encoder comprises a visual abstractor module (Li in [0027] discloses about query transformer 120 (abstractor module)).
Summary of Citations (Li)
Paragraph [0027]; “The query Transformer 120 is a lightweight transformer which employs a set of learnable query vectors 106 to extract visual features from the frozen image encoder 110 . In other words, the query Transformer 120 acts as an information bottleneck between the frozen image encoder 110 and the frozen LLM 130 , where it feeds the most useful visual feature from an input image 105 a for the LLM 130 to output the desired text”.
Regarding claim 18, Li in the combination discloses the method of claim 17, wherein the visual abstractor module is a Q-Former(query transformer) (Li in [0027] discloses about query transformer 120).
Summary of Citations (Li)
Paragraph [0027]; “The query Transformer 120 is a lightweight transformer which employs a set of learnable query vectors 106 to extract visual features from the frozen image encoder 110”.
Regarding claim 19, Li in the combination discloses the method of claim 18, further comprising: inputting the visual features into the Q-Former to extract useful language-informative visual representation while removing irrelevant visual information ((Li in [0027] discloses, “The query Transformer 120 is a lightweight transformer which employs a set of learnable query vectors 106 to extract visual features from the frozen image encoder 110 . In other words, the query Transformer 120 acts as an information bottleneck between the frozen image encoder 110 and the frozen LLM 130 , where it feeds the most useful visual feature from an input image 105 a for the LLM 130 to output the desired text”).
Summary of Citations (Li)
Paragraph [0027]; “The query Transformer 120 is a lightweight transformer which employs a set of learnable query vectors 106 to extract visual features from the frozen image encoder 110 . In other words, the query Transformer 120 acts as an information bottleneck between the frozen image encoder 110 and the frozen LLM 130 , where it feeds the most useful visual feature from an input image 105 a for the LLM 130 to output the desired text”.
Regarding claim 20, Jiang in the combination discloses the method of claim 12, further comprising if the third similarity score is smaller than a third threshold, then inputting the (Jiang in [0068] discloses, “And if the answer is judged to be a question and answer type, generating an answer according to the content in the knowledge graph. If no answer exists in the knowledge graph, generating a question answer through LLM”).
Jiang doesn’t disclose about the following limitation as further recited in the claim.
Li discloses concatenated query embedding (Li discloses about concatenated query embedding in [0052]).
Summary of Citations (Li)
Paragraph [0052]; “FIG. 4 B adopts an encoder-decoder-based LLM comprising the LLM encoder 130 a and LLM decoder 130 b . For encoder-decoder LLMs 130 b , the LLM encoder 130 a may receive both a prefix text 414 and the projected embedding 412 and encode a concatenation of 412 and 414 into an encoded representation”.
Claim 14 is rejected under 35 U.S.C 103 as being unpatentable over Jiang in view of Park and Li and further in view of Aswin US Patent Application Publication No. US-20250069356-A1 (hereinafter Aswin).
Regarding claim 14, Jiang in the combination discloses the method of claim 12, large language model database (Jiang discloses about LLM in [0068]).
Jiang, Park and Li in the combination doesn’t disclose about the following limitation as further recited in the claim.
Aswin discloses comprising applying a third library to the (Aswin in [0027] discloses, “an optional hyperdimensional (HD) database 140 for implementing a Retrieval Augmented Generation (RAG) technique of the present principles (described in greater detail below)” wherein HD database 140 equates to third library); wherein comparing the concatenated query embedding with the (Aswin in [0044] discloses, “a similarity measure between at least one the projected HD query vector representation of at least one of the image of the query and the text of the query and at least one of the respective exemplars can be determined by, for example the similarity/confidence module 138 to identify an exemplar having a highest degree of similarity measure to the projected hyperdimensional query text vector representation in the hyperdimensional embedding space to identify at least one object”. Additionally, Aswin in [0078] discloses, “HD database 140 searches for examples (e.g., datapoints) in the HD database 140 relevant to the received query”. Returning multiple search result that is relevant equates finding K most similar embedding); selecting an embedding best matching the concatenated query embedding from the k most similar embeddings (Aswin in [0044] discloses, “a similarity measure between ... query vector representation of at least one of the image of the query and the text of the query ... the similarity/confidence module 138 to identify an exemplar having a highest degree of similarity measure to the projected hyperdimensional query text vector representation ...”); and comparing the concatenated query embedding with the embedding best matching the concatenated query embedding to generate the third similarity score (Jiang in [0046] discloses, the similarity/confidence module 138 can determine a similarity measure, such as a cosine similarity, between at least one of a projected hyperdimensional query text and/or image vector representation determined for the text and/or image of a query and at least one embedded exemplar to determine an exemplar that is best representative of/can best respond to a query request”).
It would have been obvious to one of ordinary skill in art before the effective filling date of the claimed invention to integrate the technique of Aswin into the system of Jiang in view Park and Li because it would allow the system to compare the query with the closest available multimodal knowledge entry instead of basing the decision based on weaker or less relevant candidate.
Summary of Citations (Jiang)
Paragraph [0068]; “And if the answer is judged to be a question and answer type, generating an answer according to the content in the knowledge graph. If no answer exists in the knowledge graph, generating a question answer through LLM”.
Summary of Citations (Aswin)
Paragraph [0027]; “an edge object detection and visual grounding system of the present principles can further include an optional hyperdimensional (HD) database 140 for implementing a Retrieval Augmented Generation (RAG) technique of the present principles (described in greater detail below)”.
Paragraph [0044]; “a similarity measure between at least one the projected HD query vector representation of at least one of the image of the query and the text of the query and at least one of the respective exemplars can be determined by, for example the similarity/confidence module 138 to identify an exemplar having a highest degree of similarity measure to the projected hyperdimensional query text vector representation in the hyperdimensional embedding space to identify at least one object in the received image and a location of the at least one object in the received image”.
Paragraph [0046]; the similarity/confidence module 138 can determine a similarity measure, such as a cosine similarity, between at least one of a projected hyperdimensional query text and/or image vector representation determined for the text and/or image of a query and at least one embedded exemplar to determine an exemplar that is best representative of/can best respond to a query request”.
Paragraph [0078]; “HD database 140 searches for examples (e.g., datapoints) in the HD database 140 relevant to the received query”.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to ZAID MUHAMMAD SALEH whose telephone number is (703)756-1684.
The examiner can normally be reached M-F 8 am - 5 pm ET. Examiner interviews are available
via telephone, in-person, and video conferencing using a USPTO supplied web-based
collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO
Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.If attempts to
reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be
reached on (571)272-7332. The fax phone number for the organization where this application or
proceeding is assigned is 571-273-8300. Information regarding the status of published or
unpublished applications may be obtained from Patent Center. Unpublished application
information in Patent Center is available to registered users. To file and manage patent
submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit
https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center
and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For
additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
If you would like assistance from a USPTO Customer Service Representative, call 800-786
9199 (IN USA OR CANADA) or 571-272-1000.
/ZAID MUHAMMAD SALEH/
Examiner, Art Unit 2668
05/13/2026
/VU LE/Supervisory Patent Examiner, Art Unit 2668
Prosecution Insights