Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103 is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Objections
Claims 6 and 16 are objected to for being dependent on claims 1 and 11 instead of claims 5 and 15. In particular, claims 1 and 11 do not provide the antecedent basis for “aggregation agent” recited in claims 6 and 16, only claims 5 and 15 do.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
(a) NOVELTY; PRIOR ART.—A person shall be entitled to a patent unless—
(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention; or
(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
PNG
media_image1.png
18
19
media_image1.png
Greyscale
(b) EXCEPTIONS.—
(1) DISCLOSURES MADE 1 YEAR OR LESS BEFORE THE EFFECTIVE FILING DATE OF THE CLAIMED INVENTION.—A disclosure made 1 year or less before the effective filing date of a claimed invention shall not be prior art to the claimed invention under subsection (a)(1) if—
(A) the disclosure was made by the inventor or joint inventor or by another who obtained the subject matter disclosed directly or indirectly from the inventor or a joint inventor; or
(B) the subject matter disclosed had, before such disclosure, been publicly disclosed by the inventor or a joint inventor or another who obtained the subject matter disclosed directly or indirectly from the inventor or a joint inventor.
(2) DISCLOSURES APPEARING IN APPLICATIONS AND PATENTS.—A disclosure shall not be prior art to a claimed invention under subsection (a)(2) if—
(A) the subject matter disclosed was obtained directly or indirectly from the inventor or a joint inventor;
(B) the subject matter disclosed had, before such subject matter was effectively filed under subsection (a)(2), been publicly disclosed by the inventor or a joint inventor or another who obtained the subject matter disclosed directly or indirectly from the inventor or a joint inventor; or
(C) the subject matter disclosed and the claimed invention, not later than the effective filing date of the claimed invention, were owned by the same person or subject to an obligation of assignment to the same person.
Claims 1-2, 4-8, 10-12, 14-18, and 20 are rejected under 35 USC 102(a)(2) as being anticipated by Zhao et al. (WO 2024/120385 A1; see attached translation).
Regarding Claims 1 and 11, Zhao discloses a system (p. 5, ¶46 and Figs. 6-7) comprising:
a computing platform including a hardware processor and a system memory (p. 5, ¶46, server or terminal device implemented by a processor calling a computer readable program instruction stored in a memory);
an entity tracing engine stored in the system memory, the entity tracing engine including a first machine learning (ML) model trained as a mapping agent and a second ML model trained as a scoring agent (Fig. 6, device 600 includes an acquisition module 601 using TransE model, ViT model, or a BERT model per p. 15, ¶151 and prediction module 602 using pre-trained prediction model per p. 15, ¶152);
the hardware processor configured to execute the entity tracing engine to:
receive content including at least one of an image, a video, an audio, or a text (Fig. 4, receiving at least text data and image data);
identify, using a feature analyzer, one or more entities referenced in the content (p. 5, ¶63, use BERT model to extract first entity; p. 5, ¶64, use ViT model to extract first entity);
map, using the first ML model trained as the mapping agent, each of the one or more entities to respective one or more entries in a knowledge base to provide one or more entity mappings (p. 7, ¶64, image data uses ViT model to (1) extract the first entity and unimodal relationship with first entity, and (2) obtain relationship between the entity with image features and image modality; p. 7, ¶63, text data uses BERT model to (1) extract first entity and unimodal relationship with the first entity, and (2) obtain relationship between the entity with text features and the text modality; per p. 5, ¶52, the text modality is used to represent entity text information in the knowledge graph and image modality is used to represent the entity image information in the knowledge graph);
determine, using the second ML model trained as the scoring agent, a relevance score for each of the one or more entity mappings relative to the content (p. 6, ¶55 and p. 8, ¶71, different forms of decoders use different models where text modality trains a text prediction model and image modality trains an image prediction model; pp. 6-7, ¶60, decoding is performed based on single modal relationship to obtain embedding score of each modality; p. 7, ¶61, predict at least one candidate entity corresponding to the first entity based on the first entity in each modality and the unimodal relationship of the first entity and obtain an embedding score of the at least one candidate entity); and
provide an output identifying the content, at least one of the one or more entity mappings and the relevance score for the at least one of the one or more entity mappings (pp. 10-11, ¶109, obtain the final prediction result in combination with the modal score; p. 15, ¶149, determination module 604 determines second entity from at least one candidate entity based on the embedding scores to obtain triples in the knowledge graph comprising the first entity, the unimodal relationship, and the second entity; e.g., p. 13, ¶¶133-34, based on the text, structural, and image, determine that the entity is an athlete, the unimodal relationship is nationality, and predict the second entity to be country A by increasing the weight of the text modality and reducing the weight of image modality).
Regarding Claims 2 and 12, Zhao discloses wherein the mapping agent is implemented using a first large-language model (LLM) or a first multimodal foundation model (p. 6, ¶56, p. 7, ¶63, and p. 15, ¶151, acquisition module 601 comprising TransE model for structural modality, ViT (Vision Transformer) model in image modality, and BERT model for text modality; Fig. 4 showing an encoder comprising the TransE model, BERT model, and ViT model), and wherein the scoring agent is implemented using a second LLM or a second multimodal foundation model (p. 7, ¶62 and p. 7, ¶71, Fig. 4, decoder comprising prediction models for respective structural modality, text modality, and image modality).
Regarding Claims 4 and 14, Zhao discloses wherein the hardware processor is further configured to execute the entity tracing engine to:
identify, based on the content, a context for tracing the one or more entities (p. 16, ¶156, determination module 604 perform weighted summation on each candidate entity based on the weights of each modality and the embedding scores of the candidate entities under each modality to obtain a weighted summation result of each candidate entity);
wherein each of the mapping and the determining uses the context (p. 13, ¶133, Fig. 5, text modality introduces entity (a certain athlete), structural modality contains athlete’s unimodal relationships, and image modality shows the scene of the athlete playing where text modality is helpful, image modality is of no help, and structural modality has both useful and useless information; p. 13, ¶134, adjust the weights of the text modality, image modality, and structural modality such as increasing the weight of the text modality and reducing the weight of the image modality to improve prediction accuracy where based on entity is an athlete and unimodal relationship is nationality in the text modality, predict the second entity as country A; i.e., p. 15, ¶149, weights characterize the importance of each modality being the context, which are used to improve prediction accuracy).
Regarding Claims 5 and 15, Zhao discloses wherein the one or more entities include a plurality of entities, the one or more entity mappings include a plurality of entity mappings (p. 7, ¶64, image data uses ViT model to (1) extract the first entity and unimodal relationship with first entity, and (2) obtain relationship between the entity with image features and image modality; p. 7, ¶63, text data uses BERT model to (1) extract first entity and unimodal relationship with the first entity, and (2) obtain relationship between the entity with text features and the text modality), and wherein the hardware processor is further configured to execute the entity tracing engine to:
before the determining
aggregate, using a third ML model trained as an aggregation agent (p. 16, ¶157, weights under each modality are determined based on a trained weight model), all entity mappings of the plurality of entity mappings referencing a same entity of the plurality of entities to identify a set of aggregated entity mappings referencing the same entity (p. 16, ¶156, determination module 604 perform weighted summation on each candidate entity based on the weights of each modality and the embedding scores of the candidate entities under each modality to obtain a weighted summation result of each candidate entity);
wherein the output further identifies the set of aggregated entity mappings (p. 16, ¶156, perform weighted summation on each candidate entity based on the weights of each modality and the embedding scores of the candidate entities under each modality to obtain a weighted summation result of each candidate entity and determine the second entity corresponding to the first entity based on the weighted summation result of each candidate entity).
Regarding Claims 6 and 16, Zhao discloses wherein the aggregation agent is implemented using a third LLM or a third multimodal foundation model (p. 16, ¶158, training step of the weight model includes an inner loop and an outer loop, the inner loop is a loop between multiple modalities).
Regarding Claims 7 and 17, Zhao discloses wherein each of the one or more entity mappings includes an identity of an entity mapped by the entity mapping (p. 13, ¶133, text modality introduces the entity as a certain athlete; p. 13, ¶134, in the exemplary entity-relationship-entity mapping of Fig. 5, the identity of the missing entity was correctly predicted to be country A), an entity type of the entity (p. 13, ¶134, in the exemplary entity-relationship-entity mapping of Fig. 5, the type of the missing entity is nationality according to the unimodal relationship; per p. 1, ¶10, nodes and edges are used to represent triples consisting of entity-relationship (or attribute)-entity), and a knowledge base address of a knowledge base entry referencing the entity (p. 13, ¶134, complete the knowledge graph based on the triplet athlete-nationality-country A; i.e., p. 4, ¶40, knowledge graph / base are constructed (i.e., addressed with entries or nodes per p. 1, ¶10 where nodes represent entities and edge between two nodes represent relationship between two entities) based on entities and relationship triples).
Regarding Claims 8 and 18, Zhao discloses wherein the feature analyzer (Fig. 4 showing an encoder and a decoder, the encoder comprising Vision Transformer for image data, BERT model for text, and TransE model for structural data) includes at least one of a facial recognition module (Fig. 4, Vision Transformer in the encoder analyzes image data to predict entity ev; Fig. 5 and p. 13, ¶133, Vision Transformer analyzes image data showing scene of the athlete playing), an object recognition module, an activity recognition module (Fig. 4, Vision Transformer in the encoder analyzes image data to predict entity ev; Fig. 5 and p. 13, ¶133, Vision Transformer analyzes image data showing scene of the athlete playing), or a text analysis module (Fig. 4, BERT in the encoder analyzes text data to predict entity et; p. 13, ¶133, BERT analyzes text data introducing the entity as a certain athlete) configured to analyze text and speech included in the content (p. 14, ¶140, different decoders are used to obtain candidate entities in text and audio modality and determine the second entity based on the text embedding score and the audio embedding score).
Regarding Claims 10 and 20, Zhao discloses wherein the content comprises at least one of sports content (p. 13, ¶133, text modality introduces the entity as a certain athlete while image modality shows the scene of the athlete playing), television programming content, movie content, advertising content, or video game content.
Claim Rejections - 35 USC § 103
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 103 that form the basis for the rejections under this section made in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3 and 13 are rejected under 35 USC 103 for being unpatentable over Zhao et al. (WO 2024/120385 A1; see attached translation) in view of Zhang et al. (CN 116955646 A; see attached translation).
Regarding Claims 3 and 13, Zhao does not disclose wherein at least the first LLM or the first multimodal foundation model is configured to perform one or more of zero-shot learning or few-shot learning.
Zhang discloses constructing (i.e., training) a natural language processing model for entity identification in text modality to generate a knowledge map (Abstract) where the natural language processing model is a LLM (p. 6, ¶50) configured to perform one or more of zero-shot learning or few-shot learning (p. 7, ¶57, the LLM training generates an entity recognition model with strong zero-shot or one-shot capability).
It would’ve been obvious to one ordinarily skilled in the art before the effective filing date of the invention to implement the first multimodal foundation model in Liu’s encoder (Fig. 4, model for text modality) to perform zero-shot learning or few-shot learning in order to implement a strong zero-shot or one-shot capability to simply enter text into the LLM / multimodal foundation model to return output data (Zhang, p. 7, ¶57) such as to extract entities and relationship between entities (Zhang, p. 7, ¶58).
Claims 9 and 19 are rejected under 35 USC 103 for being unpatentable over Zhao et al. (WO 2024/120385 A1; see attached translation) in view of Farre Guiu et al. (US 11354894 B2).
Regarding Claims 9 and 19, Zhao does not disclose wherein the feature analyzer includes at least one of an organization recognition module or a venue recognition module.
Farre Guiu discloses a feature analyzer to apply labels to features detected in the content, access one or more knowledge bases to validate the applied labels, and obtain descriptive data linked to the validated labels from the knowledge base (Abstract and Col 2, Rows 3-14), the feature analyzer includes an organization recognition module or a venue recognition module (Col 5, Rows 25-31).
It would’ve been obvious to one ordinarily skilled in the art before the effective filing date of the invention to implement the feature analyzer / encoder of Zhao (Fig. 4) to include an organization recognition module or a venue recognition module to perform text analysis of the content to identify venue, brands, logo or organization of the content (Farre Guiu, Col 5, Rows 25-31).
Conclusion
Prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US 12373643 B2 discloses zero shot event extraction model to identify event mentions from documents and classifying then into pre-defined event types.
US 10198491 B1 discloses extracting and storing information regarding entities from documents comprising using entity models to detect a known entity, detects a context proximate to the known entities and new entities having a lexical relationship to the known entity, and generate an entry reflecting an association between the new entity and an entity class.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to examiner Richard Z. Zhu whose telephone number is 571-270-1587 or examiner’s supervisor Hai Phan whose telephone number is 571-272-6338. Examiner Richard Zhu can normally be reached on M-Th, 0730:1700.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RICHARD Z ZHU/Primary Examiner, Art Unit 2654 06/12/2026
Prosecution Insights