DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-6, 8-18 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Krishna U.S. PAP 2025/0103642 A1 in view of Feng U.S. PAP 2022/0391426 A1.
Regarding claim 1 Krishna teaches a data processing system (Visual search in an operating system of a computing device can process and provide additional information on the content being provided for display, see abstract) comprising:
a processor (one or more processors, see par. [0008]);
and a machine-readable medium storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations (one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing device to perform operations, see par. [0008]) comprising:
obtaining an image of a document and an indication of one or more content items to generate based on content of the document (FIG. 4B depicts an illustration of object selection and processing within the visual search interface. For example, a circling gesture 414 can be received that selects a particular object depicted in the displayed content, see par. [0091]);
analyzing the image of the document using a first machine learning model trained to generate a textual representation of contents of the document in the image ( Pixels descriptive of the object may be segmented and searched. In some implementations, the image segment can be processed with a generative model (e.g., a vision language model and/or a large language model) to generate a model-generated response 418 to the query. , see par. [0092]);
constructing a query based on the textual representation of the contents of the document using a query processing unit, the query processing unit extracting information from the textual representation of the content and formatting the information according to a query format (follow-up query suggestions may be determined and provided for display in a suggestion panel adjacent to the query input box, see par. [0093]; the machine-learned suggestion model can process an output of at least one of the object detection model or the segmentation model to generate the one or more query suggestions. The one or more query suggestions can include a query to transmit to the server computing system. The query can include a multimodal query that includes a portion of the display data and a text segment, see par. [0063]);
analyzing the query using a second machine learning model to obtain embeddings representing one or more categories of information represented in the query (follow-up visual search results can include a model-generated response 432 to the processed follow-up query 426, see par. [0094]);
providing the query results to a content generation unit to generate the one or more content items based on the results of the query (The input can be obtained and provided as an updated query 446, which can include the text of the input and a thumbnail depicting the image segment, see par. [0096]);
and obtaining the one or more content items from the content generate unit (The updated query 446 can be processed to determine a plurality of updated search results 448, see par. [0096]).
However Krishna does not teach searching a knowledge graph based on the query embeddings to obtain results of the query, the knowledge graph comprising embeddings representing one or more categories of information associated with each of a plurality of content items, the results of the query comprising content related to the one or more categories of information represented in the query.
In the same field of endeavor Feng teaches a multi-system-based intelligent question answering method and apparatus, and a device, relating to the field of artificial intelligence, in particular to the field of knowledge graph, see abstract. Determining a question category of question information in response to a question answering instruction of a user, where the question answering instruction is used to indicate the question information; determining a query engine corresponding to the question category, and invoking multiple question analysis systems corresponding to the query engine according to the query engine, see par. [0008-0009]. The question analysis system obtains data information from various databases in the basic data layer 104, where the data types in the basic data layer 104 include a knowledge graph, see par. [0031] (searching a knowledge graph based on the query embeddings to obtain results of the query). The advantage of this setting is that the data required by the question category can be queried according to different question categories, thereby improving the query efficiency., see par. [0034]. Determining the answer information corresponding to the question information according to the current question analysis system when the current question analysis system is the knowledge graph system in a process of processing the question information by sequentially using the multiple question analysis systems according to system priorities of the question analysis system. In an example, when the current question analysis system is the knowledge graph system, determining the answer information corresponding to the question information according to the current question analysis system includes: extracting an entity in the question information (categories of information); and identifying, according to a preset knowledge graph in the knowledge graph system and the third ERNIE model, the entity in the question information to obtain the answer information corresponding to the question information (results of the query), where the preset knowledge graph includes multiple entities, there is a connection relationship between the entities in the preset knowledge graph, and the third ERNIE model is used to process the entity in the question information, see par. [0097-0100] (The knowledge graph comprising embeddings representing one or more categories of information associated with each of a plurality of content items, the results of the query comprising content related to the one or more categories of information represented in the query).
It would have been obvious to one of ordinary skill in the art to combine the Krishna invention with the teachings of Feng for the benefit of improving query efficiency, see par. [0034].
Regarding claim 2 Krishna teaches the data processing system of claim 1, wherein the second machine learning model is a Large Language Model (LLM) or Small Language Model (SLM), the second machine learning model having an encoder-decoder architecture (a large language model, see par. [0094]; he one or more machine-learned models 120 can process image data, text data, audio data, and/or latent encoding data to generate output data that can include image data, text data, audio data, and/or latent encoding data, see par. [0165]).
Regarding claim 3 Feng teaches the data processing system of claim 1, further comprising:
Feng teaches constructing one or more first prompts to a third machine learning model using the content generation unit ( identifying, according to a preset knowledge graph in the knowledge graph system and the third ERNIE model, the entity in the question information to obtain the answer information corresponding to the question information, see par. [0100]);
providing the one or more first prompts to the third machine learning model to obtain first generated textual content (extracting an entity in the question information, see par. [0099]);
obtaining the first generated textual content at the content generation unit (after the entity in the question information is extracted, the entity is determined in the preset knowledge graph through the preset knowledge graph in the knowledge graph system and the third ERNIE model, and the answer information related to the entity is determined, see par. [0101]);
and generating the one or more content based on the first generated textual content (determine multiple initial answers, then the multiple initial answers is scored through the third ERNIE model, see par. [0106]).
Regarding claim 4 Krishna teaches the data processing system of claim 3, wherein the third machine learning model is a Large Language Model (LLM) or Small Language Model (SLM), the third machine learning model having an encoder-decoder architecture (Content items (e.g., articles, images, videos, audio, blogs, and/or social media posts; one or more self-attention models, one or more embedding models, one or more encoders, one or more decoders, see par. [0204]) associated with the one or more visual search results can be processed with a generative language model (e.g., an autoregressive language model, which may include a large language model), see par. [0124]; he one or more machine-learned models 120 can process image data, text data, audio data, and/or latent encoding data to generate output data that can include image data, text data, audio data, and/or latent encoding data, see par. [0165]).
Regarding claim 5 Krishna teaches the data processing system of claim 3, wherein the machine-readable storage medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of: causing a user interface of an application on a client device to present the one or more content items ( The search results interface can include search results of a plurality of different types and may be displayed in a plurality of different formats in a plurality of different panels, see par. [0092]).
Regarding claim 6 Krishna teaches the data processing system of claim 5, wherein the machine-readable storage medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:
receiving a natural language query from the application on the client device, the natural language query requesting that a specified content item of the one or more content items be further refined (receive a follow-up text input 442 (e.g., “Are there other shapes available?”). The input can be obtained and provided as an updated query, see par. [0096]);
constructing a second prompt to the third machine learning model to refine the specified content item (he input can be obtained and provided as an updated query 446, which can include the text of the input and a thumbnail depicting the image segment. The updated query 446 can be processed to determine a plurality of updated search results 448, see par. [0096]);
providing the one or more first prompts to the third machine learning model to obtain second generated textual content (The plurality of updated search results 448 may be formatted by processing the web resource search results with a generative model to include natural language sentences, see par. [0096]);
obtaining the second generated textual content at the content generation unit (he visual search interface may include an action suggestion and one or more query suggestions in a suggestion panel. The action suggestion can be determined and provided based on the plurality of updated search results 448, see par. [0097]);
and generating a refined version of the specified content item based on the second generated textual content (The action suggestion can include utilizing an augmented-reality experience to view one or more products in a user environment. The one or more products can be associated with a search result, see par. [0097]).
Regarding claim 8 Krishna teaches the data processing system of claim 5, wherein the user interface is a dashboard user interface that presents the one or more content items and includes controls for viewing each of the one or more content items (see 448 and 450 in figure 4D; The web page 518 may be provided for display with a plurality of application suggestions 520, see par. [0100]).
Regarding claim 9 Krishna teaches the data processing system of claim 3, wherein the second machine learning model and the third machine learning model are the same machine learning model ( The visual search data may include an output of the one or more classification models, one or more augmentation models, and/or one or more generative vision language models, see par. [0123]).
Regarding claim 10 Krishna teaches the data processing system of claim 3, wherein the second machine learning model and the third machine learning model are different machine learning models ( The visual search data may include an output of the one or more classification models, one or more augmentation models, and/or one or more generative vision language models, see par. [0123]).
Regarding claim 11 Krishna teaches the data processing system of claim 1, wherein searching the knowledge graph based on the query embeddings to obtain the results of the query comprises searching the knowledge graph using a vector search (The search engine may determine one or more visual search results based on detected features in the image segments, an embedding search (e.g., embedding neighbor determination), see par. [0122]).
Regarding claim 12 Krishna teaches the data processing system of claim 1, wherein the document is a slide, poster, or paper ( a user may be viewing a web page (poster) 502 in a browser application. The user may provide an input to utilize the visual search interface, see par. [0099] Feng teaches a database with multiple documents, see par. [0070]).
Regarding claim 13 Krishna teaches a method implemented in a data processing system for generating electronic content, the method comprising:
obtaining an image of a document and an indication of one or more content items to generate based on content of the document (FIG. 4B depicts an illustration of object selection and processing within the visual search interface. For example, a circling gesture 414 can be received that selects a particular object depicted in the displayed content, see par. [0091]);
analyzing the image of the document using a first machine learning model trained to generate a textual representation of contents of the document in the image ( Pixels descriptive of the object may be segmented and searched. In some implementations, the image segment can be processed with a generative model (e.g., a vision language model and/or a large language model) to generate a model-generated response 418 to the query. , see par. [0092]);
constructing a query based on the textual representation of the contents of the document using a query processing unit, the query processing unit extracting information from the textual representation of the content and formatting the information according to a query format (follow-up query suggestions may be determined and provided for display in a suggestion panel adjacent to the query input box, see par. [0093]);
analyzing the query using a second machine learning model to obtain embeddings representing one or more categories of information represented in the query (follow-up visual search results can include a model-generated response 432 to the processed follow-up query 426, see par. [0094]);
providing the query results to a content generation unit to generate the one or more content items based on the results of the query (The input can be obtained and provided as an updated query 446, which can include the text of the input and a thumbnail depicting the image segment, see par. [0096]);
and obtaining the one or more content items from the content generate unit (The updated query 446 can be processed to determine a plurality of updated search results 448, see par. [0096]).
However Krishna does not teach searching a knowledge graph based on the query embeddings to obtain results of the query, the knowledge graph comprising embeddings representing one or more categories of information associated with each of a plurality of content items, the results of the query comprising content related to the one or more categories of information represented in the query.
In the same field of endeavor Feng teaches a multi-system-based intelligent question answering method and apparatus, and a device, relating to the field of artificial intelligence, in particular to the field of knowledge graph, see abstract. Determining a question category of question information in response to a question answering instruction of a user, where the question answering instruction is used to indicate the question information; determining a query engine corresponding to the question category, and invoking multiple question analysis systems corresponding to the query engine according to the query engine, see par. [0008-0009]. The question analysis system obtains data information from various databases in the basic data layer 104, where the data types in the basic data layer 104 include a knowledge graph, see par. [0031] (searching a knowledge graph based on the query embeddings to obtain results of the query). The advantage of this setting is that the data required by the question category can be queried according to different question categories, thereby improving the query efficiency., see par. [0034]. Determining the answer information corresponding to the question information according to the current question analysis system when the current question analysis system is the knowledge graph system in a process of processing the question information by sequentially using the multiple question analysis systems according to system priorities of the question analysis system. In an example, when the current question analysis system is the knowledge graph system, determining the answer information corresponding to the question information according to the current question analysis system includes: extracting an entity in the question information (categories of information); and identifying, according to a preset knowledge graph in the knowledge graph system and the third ERNIE model, the entity in the question information to obtain the answer information corresponding to the question information (results of the query), where the preset knowledge graph includes multiple entities, there is a connection relationship between the entities in the preset knowledge graph, and the third ERNIE model is used to process the entity in the question information, see par. [0097-0100] (The knowledge graph comprising embeddings representing one or more categories of information associated with each of a plurality of content items, the results of the query comprising content related to the one or more categories of information represented in the query).
It would have been obvious to one of ordinary skill in the art to combine the Krishna invention with the teachings of Feng for the benefit of improving query efficiency, see par. [0034].
Regarding claim 14 Krishna teaches the method of claim 13, wherein the second machine learning model is a Large Language Model (LLM) or Small Language Model (SLM), the second machine learning model having an encoder-decoder architecture (a large language model, see par. [0094]; he one or more machine-learned models 120 can process image data, text data, audio data, and/or latent encoding data to generate output data that can include image data, text data, audio data, and/or latent encoding data, see par. [0165]).
Regarding claim 15 Krishna teaches the method of claim 13, further comprising:
constructing one or more first prompts to a third machine learning model using the content generation unit ( identifying, according to a preset knowledge graph in the knowledge graph system and the third ERNIE model, the entity in the question information to obtain the answer information corresponding to the question information, see par. [0100]);
providing the one or more first prompts to the third machine learning model to obtain first generated textual content (extracting an entity in the question information, see par. [0099]);
obtaining the first generated textual content at the content generation unit (after the entity in the question information is extracted, the entity is determined in the preset knowledge graph through the preset knowledge graph in the knowledge graph system and the third ERNIE model, and the answer information related to the entity is determined, see par. [0101]);
and generating the one or more content based on the first generated textual content (determine multiple initial answers, then the multiple initial answers is scored through the third ERNIE model, see par. [0106]).
Regarding claim 16 Krishna teaches the method of claim 13, wherein the third machine learning model is a Large Language Model (LLM) or Small Language Model (SLM), the third machine learning model having an encoder-decoder architecture (a large language model, see par. [0094]; he one or more machine-learned models 120 can process image data, text data, audio data, and/or latent encoding data to generate output data that can include image data, text data, audio data, and/or latent encoding data, see par. [0165]).
Regarding claim 17 Krishna teaches the method of claim 13, further comprising: causing a user interface of an application on a client device to present the one or more content ( The search results interface can include search results of a plurality of different types and may be displayed in a plurality of different formats in a plurality of different panels, see par. [0092]).
Regarding claim 18 Krishna teaches the method of claim 15, further comprising: receiving a natural language query from the application on the client device, the natural language query requesting that a specified content item of the one or more content items be further refined (receive a follow-up text input 442 (e.g., “Are there other shapes available?”). The input can be obtained and provided as an updated query, see par. [0096]);
constructing a second prompt to the third machine learning model to refine the specified content item (he input can be obtained and provided as an updated query 446, which can include the text of the input and a thumbnail depicting the image segment. The updated query 446 can be processed to determine a plurality of updated search results 448, see par. [0096]);
providing the one or more first prompts to the third machine learning model to obtain second generated textual content (The plurality of updated search results 448 may be formatted by processing the web resource search results with a generative model to include natural language sentences, see par. [0096]);
obtaining the second generated textual content at the content generation unit (he visual search interface may include an action suggestion and one or more query suggestions in a suggestion panel. The action suggestion can be determined and provided based on the plurality of updated search results 448, see par. [0097]);
and generating a refined version of the specified content item based on the second generated textual content (The action suggestion can include utilizing an augmented-reality experience to view one or more products in a user environment. The one or more products can be associated with a search result, see par. [0097]).
Regarding claim 20 Krishna teaches the method of claim 11, wherein the document is a slide, poster, or paper ( a user may be viewing a web page (poster) 502 in a browser application. The user may provide an input to utilize the visual search interface, see par. [0099]).
Allowable Subject Matter
Claims 7 and 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Gusarov ‘888 teaches the artificial intelligence and machine learning system 230 may implement a visual-semantic machine learning model, also known as a vision-language model. As used herein, the term “visual-semantic machine learning model” refers to a machine learning model or a combination of machine learning models that has the ability to process both visual (e.g., image or video) and language (e.g., textual) data, see col. 13 lines 12-22.
Feng ‘426 teaches a knowledge graph system, see claims 8-9.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael Ortiz-Sanchez whose telephone number is (571)270-3711. The examiner can normally be reached Monday- Friday 9AM-6PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh Mehta can be reached at 571-272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL ORTIZ-SANCHEZ/Primary Examiner, Art Unit 2656
Prosecution Insights