DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1-23 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1 is directed to a system. The claim recites “processing the multimodal query to determine a plurality of initial search results, wherein the plurality of initial search results are determined to be responsive to the multimodal query; determining the multimodal query comprises weak search signals based on the image input and the plurality of initial search results; in response to determining the multimodal query comprises weak search signals, processing the image input to generate a prompt, wherein the prompt is associated with a query clarification request” and “processing the multimodal query and the user input to determine a plurality of second search results”. The processes of multimodal query to determine initial search results, determining multimodal query being unclear or having weak signal based on input image and initial search results, generating prompts for clarifying the query in response to determining weak signal and processing user subsequent multimodal query in prompt involve observation, judgement and evaluation. Aforementioned processes can practically be performed in the human mind. Thus, the claim is directed to an abstract idea falling within the grouping of mental steps, see MPEP 2106.04(a)(2)(III).
At step 2A, prong 2, this judicial exception is not integrated into a practical application. In particular, the claim recites additional elements – “obtaining a multimodal query, wherein the multimodal query comprises a text input and an image input” and “receiving a user input via the search results interface”. Above mentioned step of retrieving contents recites insignificant extra-solution activity of mere data gathering is “obtaining information” as identified in MPEP 2106.05 (g).The claim further recites “providing the prompt for display with the plurality of initial search results in a search results interface”. Above mentioned steps constitutes insignificant extra-solution activity of presenting data output. The claim also mentions generic computer and generic computer components such as “computing system”, “processors”, and “non-transitory computer-readable media”, which are recited at a high level of generality and is recited as performing generic computer functions routinely used in computer applications. Thus, which is use of a computer or generic computer components to execute abstract idea in the form of software constitutes use of the computer or its components as a tool. Accordingly, this additional elements do not integrate the abstract idea into a practical application. Viewing the additional limitations together and the claim as a whole, nothing provides integration into a practical application. Therefore, claim is directed to an abstract idea.
At step 2B, the claims don’t include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above the additional elements recites insignificant extra-solution activity of data gathering and outputting/transmitting data such are also well- understood, routine, and conventional. Further, sending/transmitting data is insignificant extra-solution activity of data transmission, such is also well- understood, routine, and conventional (OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network)). Also, presenting data is WURC based on OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1362-63 (Fed. Cir. 2015) (presenting offers and gathering statistics). Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept, see MPEP 2106.05 (f). Looking at the limitations in combination and the claim as a whole does not change this conclusion and the claim is ineligible.
Dependent claim 2-7 are directed to the same abstract idea as the independent claim from which they depend and further recite limitations – “processing the multimodal query and at least a subset of the plurality of initial search results ……… to generate a responsiveness score associated with how responsive the plurality of initial search results are to the multimodal query; and determining the responsiveness score is below a threshold score”, “processing the image input and the plurality of initial search results to determine an object identification for an object depicted in the image input is ambiguous based on candidate object identifications associated with the plurality of initial search results”, “processing the image input to determine the image input comprises an image quality below a quality threshold; and determining one or more similarity measures between the image input and at least a subset of the plurality of initial search results are below a similarity threshold”, “processing the multimodal query, data associated with the user input, and the plurality of second search results ……… to generate a model-generated response, wherein the …… response is generated to be responsive to the text input” and “processing the image input with an image classification model to generate a plurality of predicted classification labels and a plurality of confidence scores associated with the plurality of predicted classification labels; determining each of the plurality of confidence scores are below a threshold confidence score; and generating a prompt based on a subset of the plurality of predicted classification labels determined to have a highest probability based on the plurality of confidence scores”. The process of determining search response scores based on multimodal query and initial search results, determining if responsive score is below a threshold, determining if an object identification is ambiguous based on candidate object identification, determining input image quality score and if score is below a threshold, determining similarity measures between the image input and initial search results, processing multimodal user input and second search result for response, classifying labels for image with confidence scores, determining if confidence score is below a threshold, generating prompts based on image classification labels determined to be highest in confidence score involve observation, judgement and evaluation and can practically be performed in human mind. Accordingly, recited limitations fall into abstract idea groupings of mental process (see MPEP 2106.04(a)(2)(III)) under Step 2A, prong 1 of the 2019 PEG. Therefore, aforementioned processes can practically be performed in the human mind and directed to an abstract idea.
At step 2A, prong 2, this judicial exception is not integrated into a practical application. In particular, the claim recites additional elements – “providing the plurality of second search results for display in a search results interface, wherein the search results interface comprises a query input box, a search results panel, and a knowledge panel comprising information obtained from a curated knowledge database” and ”providing the model-generated response for display with the plurality of second search results”. Above mentioned steps constitutes insignificant extra-solution activity of presenting data output. The claim also mentions generic computer and generic computer components such as “generative model”, which is recited at a high level of generality and is recited as performing generic computer functions routinely used in computer applications. Thus, which is use of a computer or generic computer components to execute abstract idea in the form of software constitutes use of the computer or its components as a tool. Accordingly, this additional elements do not integrate the abstract idea into a practical application. Viewing the additional limitations together and the claim as a whole, nothing provides integration into a practical application. Therefore, claims are directed to an abstract idea.
At step 2B, the claims don’t include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above the additional elements recites insignificant extra-solution activity of data gathering and outputting/transmitting data such are also well- understood, routine, and conventional. Further, sending/transmitting data is insignificant extra-solution activity of data transmission, such is also well- understood, routine, and conventional (OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network)). Also, presenting data is WURC based on OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1362-63 (Fed. Cir. 2015) (presenting offers and gathering statistics). Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept, see MPEP 2106.05 (f). Looking at the limitations in combination and the claims as a whole do not change this conclusion and the claim 2-7 are ineligible.
Dependent claim 8-10 are directed to the same abstract idea as the independent claim from which they depend and further recite limitations – “processing the image input with an image………. to generate a plurality of predicted classification labels and a plurality of confidence scores associated with the plurality of predicted classification labels; determining a first score and a second score of the plurality of confidence scores are similar”, “determining a differentiating feature between the first object details and the second object details; and generating the prompt based on the differentiating feature”, “processing the multimodal query and the plurality of initial search results with a ….. overview, wherein the…….generated overview is descriptive of a model understanding of the multimodal query and the plurality of initial search results; and generating the prompt based on the ……..generated overview” and “processing the multimodal query and the plurality of initial search results with a …… to generate a……. overview, wherein the model-generated overview is descriptive of a model understanding of the multimodal query and the plurality of initial search results”. The process of generating image classification labels and confidence scores, determining similarity among scores, determining differentiating features between object details and accordingly generating prompts, processing multimodal query and search results for overview generation, generating prompts based on generated overview involve observation, judgement and evaluation and can practically be performed in human mind. Accordingly, recited limitations fall into abstract idea groupings of mental process (see MPEP 2106.04(a)(2)(III)) under Step 2A, prong 1 of the 2019 PEG. Therefore, aforementioned processes can practically be performed in the human mind and directed to an abstract idea.
At step 2A, prong 2, this judicial exception is not integrated into a practical application. In particular, the claims recite additional elements – “obtaining first object details associated with a first object classification of the plurality of predicted classification labels” and “obtaining second object details associated with a second object classification of the plurality of predicted classification labels”. Above mentioned step of retrieving contents recites insignificant extra-solution activity of mere data gathering is “obtaining information” as identified in MPEP 2106.05 (g).The claim further recites “providing the prompt for display with the plurality of initial search results in the search results interface comprises: providing the prompt and the model-generated overview for display with the plurality of initial search results in a search results interface”. Above mentioned steps constitutes insignificant extra-solution activity of presenting data output. The claim also mentions generic computer and generic computer components such as “generative model”, which is recited at a high level of generality and is recited as performing generic computer functions routinely used in computer applications. Thus, which is use of a computer or generic computer components to execute abstract idea in the form of software constitutes use of the computer or its components as a tool. Accordingly, this additional elements do not integrate the abstract idea into a practical application. Viewing the additional limitations together and the claim as a whole, nothing provides integration into a practical application. Therefore, claims 8-10 directed to an abstract idea.
At step 2B, the claims don’t include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above the additional elements recites insignificant extra-solution activity of data gathering and outputting/transmitting data such are also well- understood, routine, and conventional. Further, sending/transmitting data is insignificant extra-solution activity of data transmission, such is also well- understood, routine, and conventional (OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network)). Also, presenting data is WURC based on OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1362-63 (Fed. Cir. 2015) (presenting offers and gathering statistics). Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept, see MPEP 2106.05 (f). Looking at the limitations in combination and the claims as a whole do not change this conclusion and the claims 8-10 are ineligible.
Claim 11 is directed to a process. The claim recites “processing, …… the multimodal query to determine a plurality of initial search results, wherein the plurality of initial search results are determined to be responsive to the multimodal query; determining, …… the multimodal query comprises weak search signals based on the image input and the plurality of initial search results; in response to determining the multimodal query comprises weak search signals, processing, …………. the image input with an object detection … to generate one or more object detections; processing…………. the image input and the one or more object detections to generate a prompt, wherein the prompt is associated with a query clarification request” and “processing, …….the multimodal query and the user input to determine a plurality of second search results”. The processes of multimodal query to determine initial search results, determining multimodal query being unclear or having weak signal based on input image and initial search results, detecting objects and generating prompts for clarification requests based on object detection and generating second search results based on multimodal input involve observation, judgement and evaluation. Aforementioned processes can practically be performed in the human mind. Thus, the claim is directed to an abstract idea falling within the grouping of mental steps, see MPEP 2106.04(a)(2)(III).
At step 2A, prong 2, this judicial exception is not integrated into a practical application. In particular, the claim recites additional elements – “obtaining, by a computing system comprising one or more processors, a multimodal query, wherein the multimodal query comprises a text input and an image input” and “receiving, by the computing system, a user input via the search results interface”. Above mentioned step of retrieving contents recites insignificant extra-solution activity of mere data gathering is “obtaining information” as identified in MPEP 2106.05 (g).The claim further recites “providing, by the computing system, the prompt for display with the plurality of initial search results in a search results interface”. Above mentioned steps constitutes insignificant extra-solution activity of presenting data output. The claim also mentions generic computer and generic computer components such as “object detection model”, which is recited at a high level of generality and is recited as performing generic computer functions routinely used in computer applications. Thus, which is use of a computer or generic computer components to execute abstract idea in the form of software constitutes use of the computer or its components as a tool. Accordingly, this additional elements do not integrate the abstract idea into a practical application. Viewing the additional limitations together and the claim as a whole, nothing provides integration into a practical application. Therefore, claim is directed to an abstract idea.
At step 2B, the claims don’t include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above the additional elements recites insignificant extra-solution activity of data gathering and outputting/transmitting data such are also well- understood, routine, and conventional. Further, sending/transmitting data is insignificant extra-solution activity of data transmission, such is also well- understood, routine, and conventional (OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network)). Also, presenting data is WURC based on OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1362-63 (Fed. Cir. 2015) (presenting offers and gathering statistics). Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept, see MPEP 2106.05 (f). Looking at the limitations in combination and the claim as a whole does not change this conclusion and the claim 11 is ineligible.
Dependent claim 12-16 are directed to the same abstract idea as the independent claim from which they depend and further recite limitations – “processing the image input to determine the image input comprises an image quality below a quality threshold; or determining a responsiveness score for the plurality of initial search results is below a response threshold” and “processing at least one of the image input or the one or more object detections to determine a plurality of image search results; and generating the plurality of selectable images based on the plurality of image search results”. The process of determining if the input image quality is below a threshold, determining if responsive score is below a threshold, determining if an object identification is ambiguous based on candidate object identification, determining input image quality score and if score is below a threshold, detection of object for search result generation involve observation, judgement and evaluation and can practically be performed in human mind. Accordingly, recited limitations fall into abstract idea groupings of mental process (see MPEP 2106.04(a)(2)(III)) under Step 2A, prong 1 of the 2019 PEG. Therefore, aforementioned processes can practically be performed in the human mind and directed to an abstract idea.
At step 2A, prong 2, this judicial exception is not integrated into a practical application. In particular, the claim recites additional elements – “wherein the prompt comprises a plurality of selectable images, wherein the plurality of selectable images are obtained based on the one or more object detections”, “generating the plurality of selectable images based on the plurality of image search results” and “the user input is descriptive of a selection of a particular image of the plurality of selectable images”. Above mentioned step of retrieving contents recites insignificant extra-solution activity of mere data gathering is “obtaining information” as identified in MPEP 2106.05 (g).The claim further recites “generating the plurality of selectable images based on the plurality of image search results” and “the prompt for display with the plurality of initial search results in the search results interface comprises: providing the plurality of selectable images for display in a carousel interface”. Above mentioned steps constitutes insignificant extra-solution activity of presenting data output. The claim also mentions generic computer and generic computer components such as “object detection model”, which is recited at a high level of generality and is recited as performing generic computer functions routinely used in computer applications. Thus, which is use of a computer or generic computer components to execute abstract idea in the form of software constitutes use of the computer or its components as a tool. Accordingly, this additional elements do not integrate the abstract idea into a practical application. Viewing the additional limitations together and the claims as a whole, nothing provides integration into a practical application. Therefore, claim 12-16 are directed to an abstract idea.
At step 2B, the claims don’t include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above the additional elements recites insignificant extra-solution activity of data gathering and outputting/transmitting data such are also well- understood, routine, and conventional. Further, sending/transmitting data is insignificant extra-solution activity of data transmission, such is also well- understood, routine, and conventional (OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network)). Also, presenting data is WURC based on OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1362-63 (Fed. Cir. 2015) (presenting offers and gathering statistics). Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept, see MPEP 2106.05 (f). Looking at the limitations in combination and the claims as a whole individually do not change this conclusion and the claims 12-16 are ineligible.
Claim 17 is directed to a product. The claim recites “processing the visual query to determine a plurality of initial search results, wherein the plurality of initial search results are determined to be responsive to the visual query; determining a search intent of the visual query is ambiguous based on at least one of the image input and the plurality of initial search results; in response to determining the search intent of the visual query is ambiguous, processing the image input with an image classification …. to generate an image classification; generating a plurality of prompts based on the image classification, wherein the plurality of prompts comprise a plurality of suggested data processing actions” and “processing the multimodal query and the user input to determine a plurality of second search results”. The processes of visual query to determine initial search results, determining query being ambiguous based on input image and initial search results, classifying images and generation of prompts based on classification with prompt comprising suggested actions and generating second search results based on multimodal input involve observation, judgement and evaluation. Aforementioned processes can practically be performed in the human mind. Thus, the claim is directed to an abstract idea falling within the grouping of mental steps, see MPEP 2106.04(a)(2)(III).
At step 2A, prong 2, this judicial exception is not integrated into a practical application. In particular, the claim recites additional elements – “obtaining a visual query, wherein the visual query comprises an image input” and “receiving a user input via the search results interface”. Above mentioned step of retrieving contents recites insignificant extra-solution activity of mere data gathering is “obtaining information” as identified in MPEP 2106.05 (g).The claim further recites “providing the plurality of prompts for display with the plurality of initial search results in a search results interface”. Above mentioned steps constitutes insignificant extra-solution activity of presenting data output. The claim also mentions generic computer and generic computer components such as “classification model”, which is recited at a high level of generality and is recited as performing generic computer functions routinely used in computer applications. Thus, which is use of a computer or generic computer components to execute abstract idea in the form of software constitutes use of the computer or its components as a tool. Accordingly, this additional elements do not integrate the abstract idea into a practical application. Viewing the additional limitations together and the claim as a whole, nothing provides integration into a practical application. Therefore, claim i17 s directed to an abstract idea.
At step 2B, the claims don’t include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above the additional elements recites insignificant extra-solution activity of data gathering and outputting/transmitting data such are also well- understood, routine, and conventional. Further, sending/transmitting data is insignificant extra-solution activity of data transmission, such is also well- understood, routine, and conventional (OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network)). Also, presenting data is WURC based on OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1362-63 (Fed. Cir. 2015) (presenting offers and gathering statistics). Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept, see MPEP 2106.05 (f). Looking at the limitations in combination and the claim as a whole does not change this conclusion and the claim 17 is ineligible.
Dependent claim 18-20 are directed to the same abstract idea as the independent claim from which they depend and further recite limitations – “processing the image input with an optical character recognition … to generate text data descriptive of text within the image input; determining a plurality of text data processing actions based on the image classification being descriptive of the image input being text-focused; and generating the plurality of prompts based on the plurality of text data processing actions” and “processing the multimodal query and the user input to determine a plurality of second search results comprises: processing the text data with a search engine to determine a plurality of web search results; and processing the particular prompt and the text data with a …….. to generate a response, wherein the plurality of second search results comprises the plurality of web search results and the ……….generated response” and “ processing the image input with an optical character recognition ……l to generate text data descriptive of text within the image input; processing the text data and the plurality of initial search results with a …. to generate the plurality of prompts”. The process of classify images using optical character recognition to generate descriptive text within image input, determining data processing action based on image classification, generating prompts based on text data processing action, processing multimodal user query to determine second search results and processing text and prompt data involve observation, judgement and evaluation and can practically be performed in human mind. Accordingly, recited limitations fall into abstract idea groupings of mental process (see MPEP 2106.04(a)(2)(III)) under Step 2A, prong 1 of the 2019 PEG. Therefore, aforementioned processes can practically be performed in the human mind and directed to an abstract idea.
At step 2A, prong 2, this judicial exception is not integrated into a practical application. In particular, the claims recite additional elements – “the user input comprises a selection of a particular prompt associated with a particular text data processing action of the plurality of text data processing actions”, “generating the plurality of selectable images based on the plurality of image search results” and “the user input is descriptive of a selection of a particular image of the plurality of selectable images”. Above mentioned step of retrieving contents recites insignificant extra-solution activity of mere data gathering is “obtaining information” as identified in MPEP 2106.05 (g).The claim further recites “the plurality of second search results comprises the plurality of web search results and the model-generated response”. Above mentioned steps constitutes insignificant extra-solution activity of presenting data output. The claim also mentions generic computer and generic computer components such as “character recognition model”, which is recited at a high level of generality and is recited as performing generic computer functions routinely used in computer applications. Thus, which is use of a computer or generic computer components to execute abstract idea in the form of software constitutes use of the computer or its components as a tool. Accordingly, this additional elements do not integrate the abstract idea into a practical application. Viewing the additional limitations together and the claims as a whole, nothing provides integration into a practical application. Therefore, claim 18-20 are directed to an abstract idea.
At step 2B, the claims don’t include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above the additional elements recites insignificant extra-solution activity of data gathering and outputting/transmitting data such are also well- understood, routine, and conventional. Further, sending/transmitting data is insignificant extra-solution activity of data transmission, such is also well- understood, routine, and conventional (OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network)). Also, presenting data is WURC based on OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1362-63 (Fed. Cir. 2015) (presenting offers and gathering statistics). Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept, see MPEP 2106.05 (f). Looking at the limitations in combination and the claims as a whole individually do not change this conclusion and the claims 18-20 are ineligible.
Claim 21 is directed to a system. The claim recites “processing the multimodal query to determine a plurality of initial search results, wherein the plurality of initial search results are determined to be responsive to the multimodal query”, “determining the multimodal query comprises a particular task type of a plurality of different task types; determining the particular task type is associated with a set of task types that a search interface requests information grounding, wherein information grounding comprises details ….. conditioned to generate a response based on for responding to a query” and “ in response to determining the particular task type is associated with a set of task types that the search interface requests information grounding, processing the multimodal query and the plurality of initial search results …… to determine an additional information request is to be provided to the user …..; processing the multimodal query and the plurality of initial search results ….. to generate a preliminary model-generated response comprising a caveat statement, wherein the preliminary ……generated response comprises a natural language response to the text input, and wherein the caveat statement is descriptive of reasoning for a conclusion of the preliminary -generated response and an indication of a lack of a threshold confidence level due to a lack of additional information”. The processes of determining initial search result in response to multimodal query, determining multimodal query to have a particular task type of a plurality of task types, determining particular task type is associated with a set of task types that requests information grounding (dictating information source to obtain data), processing the query for initial search result to determine additional information request and processing the query to generate response having caveat statement with an indication of lack of threshold confidence level due to the lack of additional information involve observation, judgement and evaluation. Aforementioned processes can practically be performed in the human mind. Thus, the claim is directed to an abstract idea falling within the grouping of mental steps, see MPEP 2106.04(a)(2)(III).
At step 2A, prong 2, this judicial exception is not integrated into a practical application. In particular, the claim recites additional elements – “obtaining a multimodal query from a user computing device, wherein the multimodal query comprises a text input and an image input”. Above mentioned step of retrieving contents recites insignificant extra-solution activity of mere data gathering is “obtaining information” as identified in MPEP 2106.05 (g). The claim also mentions generic computer and generic computer components such as “generative model”, which is recited at a high level of generality and is recited as performing generic computer functions routinely used in computer applications. Thus, which is use of a computer or generic computer components to execute abstract idea in the form of software constitutes use of the computer or its components as a tool. Accordingly, this additional elements do not integrate the abstract idea into a practical application. Viewing the additional limitations together and the claim as a whole, nothing provides integration into a practical application. Therefore, claim is directed to an abstract idea.
At step 2B, the claims don’t include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above the additional elements recites insignificant extra-solution activity of data gathering and outputting/transmitting data such are also well- understood, routine, and conventional. Further, sending/transmitting data is insignificant extra-solution activity of data transmission, such is also well- understood, routine, and conventional (OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network)). Also, presenting data is WURC based on OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1362-63 (Fed. Cir. 2015) (presenting offers and gathering statistics). Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept, see MPEP 2106.05 (f). Looking at the limitations in combination and the claim as a whole does not change this conclusion and the claim is ineligible.
Dependent claim 22-23 are directed to the same abstract idea as the independent claim from which they depend and further recite limitations – “processing the multimodal query and the plurality of initial search results with the generative model to generate a prompt, wherein the prompt comprises the additional information request” , “processing the multimodal query and the user input to determine a plurality of second search results” and “processing the multimodal query, the user input, and the plurality of second search results with the generative model to generate an updated model-generated response”. The process of generating prompt by processing multimodal query and initial search results, further, processing multimodal query for determining second search results and generation of updated response by processing user input and second search results involve observation, judgement and evaluation and can practically be performed in human mind. Accordingly, recited limitations fall into abstract idea groupings of mental process (see MPEP 2106.04(a)(2)(III)) under Step 2A, prong 1 of the 2019 PEG. Therefore, aforementioned processes can practically be performed in the human mind and directed to an abstract idea.
At step 2A, prong 2, this judicial exception is not integrated into a practical application. In particular, the claims recite additional elements – “receiving a user input via the search results interface”. Above mentioned step of retrieving contents recites insignificant extra-solution activity of mere data gathering is “obtaining information” as identified in MPEP 2106.05 (g).The claim further recites “providing the prompt for display with the plurality of initial search results in a search results interface of the search interface” and “providing the updated model-generated response and the plurality of second search results for display within the search results interface”. Above mentioned steps constitutes insignificant extra-solution activity of presenting data output. The claim also mentions generic computer and generic computer components such as “generative model”, which is recited at a high level of generality and is recited as performing generic computer functions routinely used in computer applications. Thus, which is use of a computer or generic computer components to execute abstract idea in the form of software constitutes use of the computer or its components as a tool. Accordingly, this additional elements do not integrate the abstract idea into a practical application. Viewing the additional limitations together and the claims as a whole, nothing provides integration into a practical application. Therefore, claim 22-23 are directed to an abstract idea.
At step 2B, the claims don’t include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above the additional elements recites insignificant extra-solution activity of data gathering and outputting/transmitting data such are also well- understood, routine, and conventional. Further, sending/transmitting data is insignificant extra-solution activity of data transmission, such is also well- understood, routine, and conventional (OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network)). Also, presenting data is WURC based on OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1362-63 (Fed. Cir. 2015) (presenting offers and gathering statistics). Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept, see MPEP 2106.05 (f). Looking at the limitations in combination and the claims as a whole individually do not change this conclusion and the claims 22-23 are ineligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1, 3, 5-6 and 9-12 are rejected under 35 U.S.C. 103 as being unpatentable over Bhangare, Shreyas et al(PGPUB Document No. 20260105573), hereafter referred as to “Bhangare”, in view of Badjatiya, Pinkesh et al (PGPUB Document No. 20250335775), hereafter, referred to as “Badjatiya”.
Claim 1, Bhangare teaches A computing system for search interface prompting, the system comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising(Bhangare, para 0090 discloses a system with processors, memories and storages “Computing device 500 can include an interconnect system 502 that directly or indirectly couples the following devices: memory 504, one or more central processing units (CPUs) 506, one or more graphics processing units (GPUs) 508” ):
obtaining a multimodal query, wherein the multimodal query comprises a text input and an image input(Bhangare, para 0046 discloses getting multimodal input comprising text and image “In some implementations, the machine-learning model(s) can include any type of transformer-based model capable of converting visual descriptions into structured data formats based at least in part on multimodal inputs. That is, the input can be multimodal (e.g., visual and textual) such that the model can map visual features to structured representations”); processing the multimodal query to determine a plurality of initial search results(Bhangare. Fig. 2 and para 0053 disclose after obtaining the visual and text (prompt) inputs, processing or execute the query “at block 210, includes obtaining, by and/or using a visual language model (VLM) and/or at least one neural network (NN) model, visual data and a first prompt corresponding to a context of the visual data. That is, the VLM can be configured to process visual and contextual information based at least in part on one or more prompts” ), wherein the plurality of initial search results are determined to be responsive to the multimodal query(Bhangare. Element 220 of Fig. 2 and para 0054 disclose after obtaining the obtaining initial/first result “ the visual data and/or the prompt can be inputted into the VLM. In this example, the VLM can output a detailed description of detected objects, corresponding positions, and other contextual attributes. In some implementations, determining the at least one description of the visual data can include extracting, by the VLM, the one or more attributes from the visual data using multimodal feature extraction” );
determining the multimodal query comprises weak search signals based on the image input and the plurality of initial search results; in response to determining the multimodal query comprises weak search signals, processing the image input to generate a prompt, wherein the prompt is associated with a query clarification request(Bhangare, para 0058 discloses generation of subsequent prompt for validation of initial query for matching visual image when needed based on previous/initial findings (not enough information to decide the match or weak signal)“The method 200 can further include receiving, by the VLM, a subsequent prompt to validate the one or more attributes within the visual data. That is, the prompt can request a validation of details of the visual content or scenarios during testing………..the structure of the subsequent prompt can be based at least in part on the initial prompt and the outcomes of previous tests”);
and processing the multimodal query and the user input to determine a plurality of second search results(Bhangare, para 0058 discloses subsequent prompt is getting executed for matching with expected result “The method 200 can further include receiving, by the VLM, a subsequent prompt to validate the one or more attributes within the visual data. That is, the prompt can request a validation of details of the visual content or scenarios during testing. For example, the prompt can query whether detected attributes match the expected results or if specific conditions are met”).
But Bhangare does not explicitly teach providing the prompt for display with the plurality of initial search results in a search results interface; receiving a user input via the search results interface;
However, in the same field of endeavor of conducting AI based search Badjatiya teaches providing the prompt for display with the plurality of initial search results in a search results interface; receiving a user input via the search results interface(Badjatiya, Fig. 8 and para 0031 disclose receiving user input (element 804) along with initial search result interface element 806-A ~D);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the feature of receiving user input via a searching user interface of Badjatiya into generation additional prompt for search clarification of Bhangare to produce an expected result of obtaining relevant contents from search. The modification would be obvious because one of ordinary skill in the art would be motivated to use subsequent prompt which would get more detailed and precise prompt for AI search for facilitating a more targeted and efficient search experience (Badjatiya, para 0031).
Regarding claim 3, Bhangare and Badjatiya teaches all the limitations of claim 1 and Bhangare further teaches wherein determining the multimodal query comprises weak search signals based on the image input and the plurality of initial search results comprises(Bhangare, para 0058 discloses generation of subsequent prompt for validation of initial query for matching visual image when needed based on previous/initial findings (not enough information to decide the match or weak signal)“The method 200 can further include receiving, by the VLM, a subsequent prompt to validate the one or more attributes within the visual data. That is, the prompt can request a validation of details of the visual content or scenarios during testing………..the structure of the subsequent prompt can be based at least in part on the initial prompt and the outcomes of previous tests”):
processing the image input and the plurality of initial search results to determine an object identification for an object depicted in the image input (Bhangare, para 0042 discloses object identification “model 116 can verify the presence of an object, its position within the scene, and whether it matches the expected description provided in the prompt”) is ambiguous based on candidate object identifications associated with the plurality of initial search results(Bhangare, para 0042 further discloses identification of discrepancies (ambiguous) of the inputted image with respect to output of findings “the validation output can be a binary decision indicating the success or failure of the test, a probability-based confidence measure of the detected attributes, or a qualitative description of the observed discrepancies”).
Regarding claim 5, Bhangare and Badjatiya teaches all the limitations of claim 1 and Bhangare further teaches wherein the operations further comprise: providing the plurality of second search results for display in a search results interface (Bhangare, para 0058 discloses generation of subsequent prompt for validation of initial query for matching visual image when needed based on previous/initial findings (not enough information to decide the match or weak signal)“The method 200 can further include receiving, by the VLM, a subsequent prompt to validate the one or more attributes within the visual data. That is, the prompt can request a validation of details of the visual content or scenarios during testing………..the structure of the subsequent prompt can be based at least in part on the initial prompt and the outcomes of previous tests”),
Badjatiya teaches wherein the search results interface comprises a query input box, a search results panel, and a knowledge panel comprising information obtained from a curated knowledge database(Badjatiya, Fig. 8 and para 0031 disclose receiving user input (element 804) along with initial search result interface element and obtained information 806-A ~D “An example of such a specialized user interface component for the structured prompt builder 116 is depicted in FIG. 8, where the user interface element with reference number 808 represents the specialized user interface component associated with the structured prompt builder 116. This user interface component 808 allows users to interactively select various intent indicators 810, which are then used to build a more detailed and precise prompt for the AI-based search assistant 114”).
Regarding claim 6, Bhangare and Badjatiya teaches all the limitations of claim 1 and Bhangare further teaches processing the multimodal query, data associated with the user input, and the plurality of second search results with a generative model to generate a model-generated response(Bhangare, para 0058 discloses subsequent prompt is getting executed for matching with expected result “The method 200 can further include receiving, by the VLM, a subsequent prompt to validate the one or more attributes within the visual data. That is, the prompt can request a validation of details of the visual content or scenarios during testing. For example, the prompt can query whether detected attributes match the expected results or if specific conditions are met”), wherein the model-generated response is generated to be responsive to the text input; and providing the model-generated response for display with the plurality of second search results(Bhangare, claim 5 further discloses response generation by visual model in response to subsequent prompt or text “receive, by the VLM, a subsequent prompt to validate the one or more attributes within the visual data; generate, by the VLM, a corresponding validation output”).
Regarding claim 9, Bhangare and Badjatiya teaches all the limitations of claim 1 and Bhangare further teaches wherein processing the image input to generate the prompt comprises(Bhangare, para 0058 discloses generation of subsequent prompt for validation of initial query for matching visual image when needed based on previous/initial findings (not enough information to decide the match or weak signal)“The method 200 can further include receiving, by the VLM, a subsequent prompt to validate the one or more attributes within the visual data. That is, the prompt can request a validation of details of the visual content or scenarios during testing………..the structure of the subsequent prompt can be based at least in part on the initial prompt and the outcomes of previous tests”):
processing the multimodal query and the plurality of initial search results with a generative model to generate a model-generated overview, wherein the model-generated overview is descriptive of a model understanding of the multimodal query and the plurality of initial search results(Bhangare. Element 220 of Fig. 2 and para 0054 disclose after obtaining the obtaining initial/first result with descriptions or overviews “ the visual data and/or the prompt can be inputted into the VLM. In this example, the VLM can output a detailed description of detected objects, corresponding positions, and other contextual attributes. In some implementations, determining the at least one description of the visual data can include extracting, by the VLM, the one or more attributes from the visual data using multimodal feature extraction” );
and generating the prompt based on the model-generated overview(Bhangare, para 0058 discloses generation of subsequent prompt for validation of initial query for matching visual image when needed based on previous/initial findings (not enough information to decide the match or weak signal)“The method 200 can further include receiving, by the VLM, a subsequent prompt to validate the one or more attributes within the visual data. That is, the prompt can request a validation of details of the visual content or scenarios during testing………..the structure of the subsequent prompt can be based at least in part on the initial prompt and the outcomes of previous tests”).
Regarding claim 10, Bhangare and Badjatiya teaches all the limitations of claim 1 and Bhangare further teaches wherein the operations further comprise: processing the multimodal query (Bhangare. Fig. 2 and para 0053 disclose after obtaining the visual and text (prompt) inputs, processing or execute the query “at block 210, includes obtaining, by and/or using a visual language model (VLM) and/or at least one neural network (NN) model, visual data and a first prompt corresponding to a context of the visual data. That is, the VLM can be configured to process visual and contextual information based at least in part on one or more prompts” ) and the plurality of initial search results with a generative model to generate a model-generated overview, wherein the model-generated overview is descriptive of a model understanding of the multimodal query and the plurality of initial search results(Bhangare. Element 220 of Fig. 2 and para 0054 disclose after obtaining the obtaining initial/first result with descriptions or overviews “ the visual data and/or the prompt can be inputted into the VLM. In this example, the VLM can output a detailed description of detected objects, corresponding positions, and other contextual attributes. In some implementations, determining the at least one description of the visual data can include extracting, by the VLM, the one or more attributes from the visual data using multimodal feature extraction” );
Badjatiya further teaches and wherein providing the prompt for display with the plurality of initial search results in the search results interface comprises(Badjatiya, Fig. 8 and para 0031 disclose receiving user input (element 804) along with initial search result interface element 806-A ~D).
Claim 11, Bhangare teaches A computer-implemented method, the method comprising: obtaining, by a computing system comprising one or more processors, a multimodal query, wherein the multimodal query comprises a text input and an image input(Bhangare, para 0046 discloses getting multimodal input comprising text and image “In some implementations, the machine-learning model(s) can include any type of transformer-based model capable of converting visual descriptions into structured data formats based at least in part on multimodal inputs. That is, the input can be multimodal (e.g., visual and textual) such that the model can map visual features to structured representations”);
processing, by the computing system, the multimodal query to determine a plurality of initial search results(Bhangare. Fig. 2 and para 0053 disclose after obtaining the visual and text (prompt) inputs, processing or execute the query “at block 210, includes obtaining, by and/or using a visual language model (VLM) and/or at least one neural network (NN) model, visual data and a first prompt corresponding to a context of the visual data. That is, the VLM can be configured to process visual and contextual information based at least in part on one or more prompts” ), wherein the plurality of initial search results are determined to be responsive to the multimodal query(Bhangare. Element 220 of Fig. 2 and para 0054 disclose after obtaining the obtaining initial/first result “ the visual data and/or the prompt can be inputted into the VLM. In this example, the VLM can output a detailed description of detected objects, corresponding positions, and other contextual attributes. In some implementations, determining the at least one description of the visual data can include extracting, by the VLM, the one or more attributes from the visual data using multimodal feature extraction” );
determining, by the computing system, the multimodal query comprises weak search signals based on the image input and the plurality of initial search results(Bhangare, para 0058 discloses determination of pass-fail or confidence of the multimodal query matches “the prompt can query whether detected attributes match the expected results or if specific conditions are met. Additionally, the processing circuits can generate (e.g., by the VLM) a corresponding validation output. In some implementations, the corresponding validation output can include at least one of a pass-fail result, a confidence score…”);
in response to determining the multimodal query comprises weak search signals, processing, by the computing system, the image input with an object detection model to generate one or more object detections; processing, by the computing system, the image input and the one or more object detections to generate a prompt, wherein the prompt is associated with a query clarification request(Bhangare, para 0058 discloses generation of subsequent prompt for validation of initial query for matching visual image when needed based on previous/initial findings (not enough information to decide the match or weak signal)“The method 200 can further include receiving, by the VLM, a subsequent prompt to validate the one or more attributes within the visual data. That is, the prompt can request a validation of details of the visual content or scenarios during testing………..the structure of the subsequent prompt can be based at least in part on the initial prompt and the outcomes of previous tests”);
and processing, by the computing system, the multimodal query and the user input to determine a plurality of second search results(Bhangare, para 0058 discloses subsequent prompt is getting executed for matching with expected result “The method 200 can further include receiving, by the VLM, a subsequent prompt to validate the one or more attributes within the visual data. That is, the prompt can request a validation of details of the visual content or scenarios during testing. For example, the prompt can query whether detected attributes match the expected results or if specific conditions are met”).
But Bhangare does not explicitly teach providing, by the computing system, the prompt for display with the plurality of initial search results in a search results interface; receiving, by the computing system, a user input via the search results interface;
However, in the same field of endeavor of conducting AI based search Badjatiya teaches providing, by the computing system, the prompt for display with the plurality of initial search results in a search results interface; receiving, by the computing system, a user input via the search results interface(Badjatiya, Fig. 8 and para 0031 disclose receiving user input (element 804) along with initial search result interface element 806-A ~D);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the feature of receiving user input via a searching user interface of Badjatiya into generation additional prompt for search clarification of Bhangare to produce an expected result of obtaining relevant contents from search. The modification would be obvious because one of ordinary skill in the art would be motivated to use subsequent prompt which would get more detailed and precise prompt for AI search for facilitating a more targeted and efficient search experience (Badjatiya, para 0031).
Regarding claim 12, Bhangare and Badjatiya teaches all the limitations of claim 11 and Bhangare further teaches wherein determining the multimodal query comprises weak search signals comprises at least one of: processing the image input to determine the image input comprises an image quality below a quality threshold; or determining a responsiveness score for the plurality of initial search results is below a response threshold(Bhangare, para 0058 discloses generation of subsequent prompt for validation of initial query for matching visual image when needed based on previous/initial findings (not enough information to decide the match or weak signal)“The method 200 can further include receiving, by the VLM, a subsequent prompt to validate the one or more attributes within the visual data. That is, the prompt can request a validation of details of the visual content or scenarios during testing………..the structure of the subsequent prompt can be based at least in part on the initial prompt and the outcomes of previous tests”).
Claim 2, 4, 12-14 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Bhangare, Shreyas et al(PGPUB Document No. 20260105573), hereafter referred as to “Bhangare”, in view of Badjatiya, Pinkesh et al (PGPUB Document No. 20250335775), hereafter, referred to as “Badjatiya”, in further view of Guy, Ido (PGPUB Document No. 20230011114), hereafter, referred to as “Guy”.
Regarding claim 2, Bhangare and Badjatiya teaches all the limitations of claim 1 and Bhangare further teaches wherein determining the multimodal query comprises weak search signals based on the image input and the plurality of initial search results comprises (Bhangare, para 0058 discloses generation of subsequent prompt for validation of initial query for matching visual image when needed based on previous/initial findings (not enough information to decide the match or weak signal) “The method 200 can further include receiving, by the VLM, a subsequent prompt to validate the one or more attributes within the visual data. That is, the prompt can request a validation of details of the visual content or scenarios during testing………..the structure of the subsequent prompt can be based at least in part on the initial prompt and the outcomes of previous tests”):
processing the multimodal query and at least a subset of the plurality of initial search results with a vision language model to generate a responsiveness score associated with how responsive the plurality of initial search results are to the multimodal query(Bhangare, para 0058 discloses using vision language model determination of matching scores “the processing circuits can generate (e.g., by the VLM) a corresponding validation output. In some implementations, the corresponding validation output can include at least one of a pass-fail result, a confidence score”);
But Bhangare and Badjatiya don’t explicitly tach and determining the responsiveness score is below a threshold score.
However, in the same field of endeavor of image quality determination Guy teaches and determining the responsiveness score is below a threshold score(Guy, para 0068 discloses finding search query result score and threshold value “the image similarity determiner 124 can identify one or more top ranked search results. Additionally, the image similarity determiner 124 can identify a lowest ranked search result. In embodiments, top ranked search results can be identified based on the top ranked search results each having an image similarity score that is above a threshold” ).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the feature of receiving user input via a searching user interface of Guy into generation additional prompt for search clarification of Bhangare and Badjatiya to produce an expected result of obtaining relevant contents from search. The modification would be obvious because one of ordinary skill in the art would be motivated to match images by embeddings for matching improved contextual similarly (Guy, para 0034).
Regarding claim 4, Bhangare and Badjatiya teaches all the limitations of claim 1 and Bhangare further teaches wherein determining the multimodal query comprises weak search signals based on the image input and the plurality of initial search results comprises(Bhangare, para 0058 discloses generation of subsequent prompt for validation of initial query for matching visual image when needed based on previous/initial findings (not enough information to decide the match or weak signal) “The method 200 can further include receiving, by the VLM, a subsequent prompt to validate the one or more attributes within the visual data. That is, the prompt can request a validation of details of the visual content or scenarios during testing………..the structure of the subsequent prompt can be based at least in part on the initial prompt and the outcomes of previous tests”):
But Bhangare and Badjatiya don’t explicitly tach processing the image input to determine the image input comprises an image quality below a quality threshold;
and determining one or more similarity measures between the image input and at least a subset of the plurality of initial search results are below a similarity threshold.
However, in the same field of endeavor of image quality determination Guy teaches processing the image input to determine the image input comprises an image quality below a quality threshold(Guy, element 304 of Fig. 3 and para 0088 input image quality determination to a threshold value “the search image is provided as an input into an image quality indication model………the image quality indication model may provide output that may be used for determining whether an image aspect for a current image frame of an image of an image corpus satisfies an image quality threshold”); and determining one or more similarity measures between the image input and at least a subset of the plurality of initial search results are below a similarity threshold(Guy, para 0068 discloses finding search query result score and threshold value “the image similarity determiner 124 can identify one or more top ranked search results. Additionally, the image similarity determiner 124 can identify a lowest ranked search result. In embodiments, top ranked search results can be identified based on the top ranked search results each having an image similarity score that is above a threshold” ).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the feature of receiving user input via a searching user interface of Guy into generation additional prompt for search clarification of Bhangare and Badjatiya to produce an expected result of obtaining relevant contents from search. The modification would be obvious because one of ordinary skill in the art would be motivated to match images by embeddings for matching improved contextual similarly (Guy, para 0034).
Regarding claim 12, Bhangare and Badjatiya teaches all the limitations of claim 11 and Bhangare further teaches wherein determining the multimodal query comprises weak search signals comprises at least one of (Bhangare, para 0058 discloses generation of subsequent prompt for validation of initial query for matching visual image when needed based on previous/initial findings (not enough information to decide the match or weak signal) “The method 200 can further include receiving, by the VLM, a subsequent prompt to validate the one or more attributes within the visual data. That is, the prompt can request a validation of details of the visual content or scenarios during testing………..the structure of the subsequent prompt can be based at least in part on the initial prompt and the outcomes of previous tests”):
But Bhangare and Badjatiya don’t explicitly tach processing the image input to determine the image input comprises an image quality below a quality threshold; or determining a responsiveness score for the plurality of initial search results is below a response threshold.
However, in the same field of endeavor of image quality determination Guy teaches processing the image input to determine the image input comprises an image quality below a quality threshold; or determining a responsiveness score for the plurality of initial search results is below a response threshold (Guy, para 0068 discloses finding search query result score and threshold value “the image similarity determiner 124 can identify one or more top ranked search results. Additionally, the image similarity determiner 124 can identify a lowest ranked search result. In embodiments, top ranked search results can be identified based on the top ranked search results each having an image similarity score that is above a threshold” ).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the feature of receiving user input via a searching user interface of Guy into generation additional prompt for search clarification of Bhangare and Badjatiya to produce an expected result of obtaining relevant contents from search. The modification would be obvious because one of ordinary skill in the art would be motivated to match images by embeddings for matching improved contextual similarly (Guy, para 0034).
Claim 13-14 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Bhangare, Shreyas et al(PGPUB Document No. 20260105573), hereafter referred as to “Bhangare”, in view of Badjatiya, Pinkesh et al (PGPUB Document No. 20250335775), hereafter, referred to as “Badjatiya”, in further view of Guy, Ido (PGPUB Document No. 20230011114), hereafter, referred to as “Guy”.
Regarding claim 13, Bhangare and Badjatiya teaches all the limitations of claim 11 but don’t explicitly teach wherein the prompt comprises a plurality of selectable images, wherein the plurality of selectable images are obtained based on the one or more object detections.
However, in the same field of endeavor of image quality determination Guy teaches wherein the prompt comprises a plurality of selectable images, wherein the plurality of selectable images are obtained based on the one or more object detections (Guy, Fig. 2A-B and para 0078-0080 disclose finding the desired search query result based on selecting one or more initial search result images “either of the first image and the second image may be selected as the search query for identifying search results…. Receiving a Selection of One of the Selectable Options…. GUI 200 further provides for display one or more of the following: the image quality indication, the image similarity, and the search query performance for the recommended image 210” ).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the feature of receiving user input via a searching user interface of Guy into generation additional prompt for search clarification of Bhangare and Badjatiya to produce an expected result of obtaining relevant contents from search. The modification would be obvious because one of ordinary skill in the art would be motivated to match images by embeddings for matching improved contextual similarly (Guy, para 0034).
Regarding claim 14, Bhangare, Badjatiyaand Guy teach all the limitations of claim 13 and Badjatiya further teaches wherein processing the image input and the one or more object detections to generate the prompt comprises: processing at least one of the image input or the one or more object detections to determine a plurality of image search results; and generating the plurality of selectable images based on the plurality of image search results (Guy, Fig. 2A-B and para 0078-0080 disclose finding the desired search query result based on selecting one or more initial search result images “either of the first image and the second image may be selected as the search query for identifying search results…. Receiving a Selection of One of the Selectable Options…. GUI 200 further provides for display one or more of the following: the image quality indication, the image similarity, and the search query performance for the recommended image 210” ).
Regarding claim 16, Bhangare, Badjatiya and Guy teach all the limitations of claim 13 and Guy further teaches wherein the user input is descriptive of a selection of a particular image of the plurality of selectable images (Guy, Fig. 2A-B and para 0078-0080 disclose finding the desired search result by selecting one or more selectable images “either of the first image and the second image may be selected as the search query for identifying search results…. Receiving a Selection of One of the Selectable Options…. GUI 200 further provides for display one or more of the following: the image quality indication, the image similarity, and the search query performance for the recommended image 210”).
Claim 7 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Bhangare, Shreyas et al(PGPUB Document No. 20260105573), hereafter referred as to “Bhangare”, in view of Badjatiya, Pinkesh et al (PGPUB Document No. 20250335775), hereafter, referred to as “Badjatiya”, in further view of Gentleman, Aaron et al (PGPUB Document No. 20220207163), hereafter, referred to as “Gentleman”.
Regarding claim 7, Bhangare and Badjatiya teaches all the limitations of claim 1 and Bhangare further teaches wherein processing the image input to generate the prompt comprises(Bhangare, para 0058 discloses generation of subsequent prompt for validation of initial query for matching visual image when needed based on previous/initial findings (not enough information to decide the match or weak signal)“The method 200 can further include receiving, by the VLM, a subsequent prompt to validate the one or more attributes within the visual data. That is, the prompt can request a validation of details of the visual content or scenarios during testing………..the structure of the subsequent prompt can be based at least in part on the initial prompt and the outcomes of previous tests”):
But Bhangare and Badjatiya don’t explicitly teach processing the image input with an image classification model to generate a plurality of predicted classification labels and a plurality of confidence scores associated with the plurality of predicted classification labels;
determining each of the plurality of confidence scores are below a threshold confidence score;
and generating a prompt based on a subset of the plurality of predicted classification labels determined to have a highest probability based on the plurality of confidence scores.
However, in the same field of endeavor of object label classification Gentleman teaches processing the image input with an image classification model to generate a plurality of predicted classification labels and a plurality of confidence scores associated with the plurality of predicted classification labels(Gentleman, para 0086 discloses object/image label classification by a classification model “In some embodiments, the support vector machine learning model may intake additional tokenized information (e.g., from additional data objects, etc.) and vectorize the tokenized information to output predicted data classification labels”); and determining one or more similarity measures between the image input and at least a subset of the plurality of initial search results are below a similarity threshold(Gentleman, para 0306~0307 disclose candidate recognition by a threshold value of classification label scoring “if an accuracy score associated with a candidate, or predicted, data classification label is within a range of a accuracy score threshold value (e.g., +/−5% of the threshold value……..”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the feature of classifying image/object labels by probability scoring of Gentleman into generation additional prompt for search clarification of Bhangare and Badjatiya to produce an expected result of matching relevant contents from search. The modification would be obvious because one of ordinary skill in the art would be motivated to match images by a threshold classification score value of labels for accuracy(Gentleman, para 0307).
Regarding claim 8, Bhangare and Badjatiya teaches all the limitations of claim 1 and Bhangare further teaches wherein processing the image input to generate the prompt comprises(Bhangare, para 0058 discloses generation of subsequent prompt for validation of initial query for matching visual image when needed based on previous/initial findings (not enough information to decide the match or weak signal)“The method 200 can further include receiving, by the VLM, a subsequent prompt to validate the one or more attributes within the visual data. That is, the prompt can request a validation of details of the visual content or scenarios during testing………..the structure of the subsequent prompt can be based at least in part on the initial prompt and the outcomes of previous tests”):
But Bhangare and Badjatiya don’t explicitly teach processing the image input with an image classification model to generate a plurality of predicted classification labels and a plurality of confidence scores associated with the plurality of predicted classification labels;
determining a first score and a second score of the plurality of confidence scores are similar; obtaining first object details associated with a first object classification of the plurality of predicted classification labels, wherein the first object classification is associated with the first score;
obtaining second object details associated with a second object classification of the plurality of predicted classification labels, wherein the second object classification is associated with the second score;
determining a differentiating feature between the first object details and the second object details; and generating the prompt based on the differentiating feature.
However, in the same field of endeavor of object label classification Gentleman teaches processing the image input with an image classification model to generate a plurality of predicted classification labels and a plurality of confidence scores associated with the plurality of predicted classification labels (Gentleman, para 0086 discloses object/image label classification by a classification model “In some embodiments, the support vector machine learning model may intake additional tokenized information (e.g., from additional data objects, etc.) and vectorize the tokenized information to output predicted data classification labels”); determining a first score and a second score of the plurality of confidence scores are similar(, para 0115 discloses predicted classification values are getting compared for similarity validation “first data object generated by an application may be compared to training data (i.e., validation data) to determine an accuracy value for a predicted classification label for the first data object”);
obtaining first object details associated with a first object classification of the plurality of predicted classification labels, wherein the first object classification is associated with the first score(Gentleman, para 0295 discloses obtaining object details (first or second) based on classification label scoring “individual words may be assigned accuracy score values associated with particular data classification labels. For example, the data classification server 106 may associate the word “street” with the “PII/DirectRestricted” data classification label because there is a sufficiently high probability (e.g., greater than or equal to 90%) that a text message data object containing the word “street” will contain a physical address”);
obtaining second object details associated with a second object classification of the plurality of predicted classification labels, wherein the second object classification is associated with the second score(Gentleman, para 0295 discloses obtaining object details (first or second) based on classification label scoring “individual words may be assigned accuracy score values associated with particular data classification labels. For example, the data classification server 106 may associate the word “street” with the “PII/DirectRestricted” data classification label because there is a sufficiently high probability (e.g., greater than or equal to 90%) that a text message data object containing the word “street” will contain a physical address”);
determining a differentiating feature between the first object details and the second object details; and generating the prompt based on the differentiating feature(Gentleman, para 0307 discloses generation of prompt based on classification difference or not matching between object featues “a truth interface may be generated if an accuracy score associated with a candidate, or predicted, data classification label is within a range of a accuracy score threshold value (e.g., +/−5% of the threshold value. For example, a truth interface may be generated to prompt user review of a candidate, or predicted, data classification label because a threshold value”);
and determining one or more similarity measures between the image input and at least a subset of the plurality of initial search results are below a similarity threshold(Gentleman, para 0115 disclose matching data objects based classification labels “the application may further compare the generated first data object with a substantially similar validation data object and, thereby, associate the first data object with an accuracy value based on how closely the suggested first data object classification label matches the classification label associated with the validation data object……..” ; disclosed teachings which can similarly be applied to input query image and search result images taught by Badjativa).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the feature of classifying image/object labels by probability scoring of Gentleman into generation additional prompt for search clarification of Bhangare and Badjatiya to produce an expected result of matching relevant contents from search. The modification would be obvious because one of ordinary skill in the art would be motivated to match images by a threshold classification score value of labels for accuracy(Gentleman, para 0307).
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Bhangare, Shreyas et al(PGPUB Document No. 20260105573), hereafter referred as to “Bhangare”, in view of Badjatiya, Pinkesh et al (PGPUB Document No. 20250335775), hereafter, referred to as “Badjatiya”, in view of Guy, Ido (PGPUB Document No. 20230011114), hereafter, referred to as “Guy”, in further view of Stachowski, Carey (PGPUB Document No. 20150242875 ), hereafter, referred to as “Stachowski”.
Regarding claim 15, Bhangare, Badjatiya and Guy teach all the limitations of claim 13 and Bhangare further teaches wherein providing, by the computing system, the prompt for display with the plurality of initial search results in the search results interface comprises(Badjatiya, Fig. 8 discloses prompt (element 810) is being displayed along with initial search results):
But Bhangare, Badjatiya and Guy don’t explicitly teach providing the plurality of selectable images for display in a carousel interface.
However, in the field of endeavor of displaying contents Stachowski teaches providing the plurality of selectable images for display in a carousel interface (Stachowski, para 0105 disclose displaying selectable contents in a carousel manner “ The publishing interface enables the user to select many options regarding publishing such as choosing a stack, a category for an image carousel, add notes and tags, and select other settings”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the feature of displaying content in carousel manner of Stachowski into displaying search results of Bhangare, Badjatiya and Guy to produce an expected result of displaying search results. The modification would be obvious because one of ordinary skill in the art would be motivated to display matched images in a easily selectable manner to improve user experience(Stachowski, para 0105).
Claim 17-19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Guy, Ido (PGPUB Document No. 20230011114), hereafter referred as to “Guy”, in view of Barros, Brett et al (WIPO Publication No. WO2022081191), hereafter, referred to as “Barros”, in further view of Kozlov, Alexander (PGPUB Document No. 20260094442), hereafter, referred to as “Kozlov”.
Regarding claim 17, Guy teaches One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising(Guy, fig. 6 discloses a system with memory and storages):
obtaining a visual query, wherein the visual query comprises an image input; processing the visual query to determine a plurality of initial search results, wherein the plurality of initial search results are determined to be responsive to the visual query(Guy, Fig. 2A-B and para 0073 disclose receiving/obtaining an image search query and providing initial search results “ search image 206 corresponding to “women's heels” is received as a search query at a search engine. Stated differently from the perspective of a user, example search image 206 is provided as a search query at the search engine. In response to search engine 106 receiving the search query from the image R/T component 118, identified images 208 are provided for display on the GUI 200. In aspects, a plurality of images 208 are identified from an image corpus” );
determining a search intent of the visual query (Guy, para 0066 discloses determining search intent for visual query “user intent may be determined by identifying items in item listing images and comparing the items to a population of stored images or a sample of the stored images specific to a particular search engine, and identifying the most relevant item in the item listing images”)is ambiguous based on at least one of the image input and the plurality of initial search results(Guy, para 0071 discloses determining quality of search or search intent is being determined by query image quality and similarity of query image compared to a set of images or initial search results which is further compared to a threshold (here the examiner is interpreting quality below the threshold is poor or ambiguous quality) “the search query performance is determined based on a set of images from the image corpus having an image quality indication above a threshold, and then subsequently determining image similarity for the set of images”)in response to determining the search intent of the visual query is ambiguous, processing the image input with an image classification model to generate an image classification(Guy, para 0066 discloses identifying intent for the query image and then classifying the image “user intent may be determined by identifying items in item listing images and comparing the items to a population of stored images or a sample of the stored images specific to a particular search engine………………... image similarity determiner 124 will then classify the sunglasses as the most relevant item in the image”);
But Guy does not explicitly tach generating a plurality of prompts based on the image classification, wherein the plurality of prompts comprise a plurality of suggested data processing actions; providing the plurality of prompts for display with the plurality of initial search results in a search results interface; receiving a user input via the search results interface; and processing the multimodal query and the user input to determine a plurality of second search results.
However, in the same field of endeavor of image classification Barros teaches generating a plurality of prompts based on the image classification, wherein the plurality of prompts comprise a plurality of suggested data processing actions (Barros, para 0004 discloses generation of prompts based on image classification and suggesting users to select objects for performing actions “instructions can be configured to cause the computing system to perform image classification within a displayed image, while performing the image classification, determine that a first object of an object class is present within the displayed image, generate a prompt, receive a selection of the first object from a user, in response to receiving the selection of the first object, perform a function on the first object” ).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the feature of classifying images and subsequently generation of prompts of Barros into image quality determination of Guy to produce an expected result of performing action for users based on image classification. The modification would be obvious because one of ordinary skill in the art would be motivated to identify actionable objects within images based on classification for conveniently listing performable actions for users (Barros, Fig. 1B and para 0026).
But Guy and Barros don’t explicitly teach providing the plurality of prompts for display with the plurality of initial search results in a search results interface; receiving a user input via the search results interface; and processing the multimodal query and the user input to determine a plurality of second search results.
However, in the same field of endeavor of input prompt generation Kozlov teaches providing the plurality of prompts for display with the plurality of initial search results in a search results interface(Kozlov, Fig. 1 and para 0029 disclose displaying obtained initial results with prompts “The user can also have an option to refine the prompt if needed to obtain more accurate results, such as by changing the existing prompt or entering a supplemental prompt in the same field or a different field”);
receiving a user input via the search results interface(Kozlov, Fig. 1 and para 0029 disclose multimodal input for generation of search results “This example GUI 100 includes a prompt field 102 that allows a user to enter a prompt to attempt to identify relevant content. This might be a text-based prompt in at least one embodiment, but in other embodiments may also allow for inclusion or identification of at least one image, video, audio, speech, gesture, and/or other such input. A prompt can provide information for a topic of interest, and allows the user to be as specific as possible”); and processing the multimodal query and the user input to determine a plurality of second search results(Kozlov, Fig. 1 and para 0029 further disclose obtained initial results are being refined by additionally generated prompts “The user can also have an option to refine the prompt if needed to obtain more accurate results, such as by changing the existing prompt or entering a supplemental prompt in the same field or a different field”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the feature of generation of additional prompts via multimodal user input of Kozlov into image quality determination of Guy and Barros to produce an expected result of refining obtained results. The modification would be obvious because one of ordinary skill in the art would be motivated to improve the search result quality by further refining the initial results using additional input prompts (Kozlov, Fig. 1 and para 0029).
Regarding claim 18, Guy, Barrons and Kozlov teach all the limitations of claim 17 and Barrons further teaches wherein the image classification comprises a text-focused image classification (Barrons, para 0029 text focus classification of images “In some examples in which the object class of the first object 104 A was a barcode or QR code, the function can include decoding the barcode or QR code and the transformed object 110A can include a text description of the item identified by the barcode or QR code included in the first object 104 A and/or an image of the item identified by the barcode or QR code included in the first object 104 A”); wherein generating the plurality of prompts based on the image classification comprises(Barros, para 0004 discloses generation of prompts based on image classification and suggesting users to select objects for performing actions “instructions can be configured to cause the computing system to perform image classification within a displayed image, while performing the image classification, determine that a first object of an object class is present within the displayed image, generate a prompt, receive a selection of the first object from a user, in response to receiving the selection of the first object, perform a function on the first object” ):
processing the image input with an optical character recognition model to generate text data descriptive of text within the image input(Barrons, para 0026 discloses user OCR for text data recognition and generation “perform optical character recognition (OCR) on text included in the object 104A, scan a document included in the object 104A, identify a person and/or human included in the object 104A, identify a type and/or breed of animal included in the object 104A, and/or identify a monument or point of interest included in the object 104A”); determining a plurality of text data processing actions based on the image classification being descriptive of the image input being text-focused; and generating the plurality of prompts based on the plurality of text data processing actions(Barrons, para 0004 and 0026 discloses generation of prompts based on classification “instructions can be configured to cause the computing system to perform image classification within a displayed image, while performing the image classification, determine that a first object of an object class is present within the displayed image, generate a prompt, receive a selection of the first object from a user, in response to receiving the selection of the first object, perform a function on the first object”).
Regarding claim 19, Guy, Barrons and Kozlov teach all the limitations of claim 18 and Kozlov further teaches wherein the user input comprises a selection of a particular prompt associated with a particular text data processing action of the plurality of text data processing actions; and wherein processing the multimodal query and the user input to determine a plurality of second search results comprises (Kozlov, Fig. 1 and para 0029 disclose multimodal input for generation of search results and further selecting prompts for initial search result refinement “This example GUI 100 includes a prompt field 102 that allows a user to enter a prompt to attempt to identify relevant content. This might be a text-based prompt in at least one embodiment, but in other embodiments may also allow for inclusion or identification of at least one image, video, audio, speech, gesture, and/or other such input. A prompt can provide information for a topic of interest, and allows the user to be as specific as possible”):
processing the text data with a search engine to determine a plurality of web search results(Kozlov, Fig. 1 and para 0029 further disclose obtained initial results are being refined by additionally generated prompts “The user can also have an option to refine the prompt if needed to obtain more accurate results, such as by changing the existing prompt or entering a supplemental prompt in the same field or a different field”; para 002 further discloses retrieved contents can be from web “providing links to articles or webpages that are determined to be relevant to a submitted query”); and processing the particular prompt and the text data with a generative language model to generate a model-generated response(Kozlov, para 0167 discloses use of generative language model for content retrieval “In the example illustrated in FIG. 16A, the generative language model system 1600 includes a retrieval augmented generation (RAG) component 1692”), wherein the plurality of second search results comprises the plurality of web search results and the model-generated response(Kozlov, Fig. 1 and para 0029 further disclose obtained initial results are being refined by additionally generated prompts “The user can also have an option to refine the prompt if needed to obtain more accurate results, such as by changing the existing prompt or entering a supplemental prompt in the same field or a different field”; para 002 further discloses retrieved contents can be from web “providing links to articles or webpages that are determined to be relevant to a submitted query”).
Regarding claim 20, Guy, Barrons and Kozlov teach all the limitations of claim 17 and Kozlov further teaches wherein the image classification comprises a text-focused image classification; wherein generating the plurality of prompts based on the image classification comprises(Barros, para 0004 discloses generation of prompts based on image classification and suggesting users to select objects for performing actions “instructions can be configured to cause the computing system to perform image classification within a displayed image, while performing the image classification, determine that a first object of an object class is present within the displayed image, generate a prompt, receive a selection of the first object from a user, in response to receiving the selection of the first object, perform a function on the first object” ): wherein the image classification comprises a text-focused image classification; wherein generating the plurality of prompts based on the image classification comprises(Barrons, para 0029 text focus classification of images “In some examples in which the object class of the first object 104 A was a barcode or QR code, the function can include decoding the barcode or QR code and the transformed object 110A can include a text description of the item identified by the barcode or QR code included in the first object 104 A and/or an image of the item identified by the barcode or QR code included in the first object 104 A”): processing the image input with an optical character recognition model to generate text data descriptive of text within the image input(Barrons, para 0026 discloses user OCR for text data recognition and generation “perform optical character recognition (OCR) on text included in the object 104A, scan a document included in the object 104A, identify a person and/or human included in the object 104A, identify a type and/or breed of animal included in the object 104A, and/or identify a monument or point of interest included in the object 104A”); processing the text data and the plurality of initial search results with a generative language model to generate the plurality of prompts(Kozlov, Fig. 1 and para 0029 disclose displaying obtained initial results with prompts “The user can also have an option to refine the prompt if needed to obtain more accurate results, such as by changing the existing prompt or entering a supplemental prompt in the same field or a different field”).
Claim 21-23 are objected and would be allowed upon overcoming the abstract idea rejection to claim 21-23.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ABDULLAH A DAUD whose telephone number is (469)295-9283. The examiner can normally be reached M~F: 9:30 am~6:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amy Ng can be reached at 571-270-1698. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ABDULLAH A DAUD/Examiner, Art Unit 2164
/AMY NG/Supervisory Patent Examiner, Art Unit 2164
Prosecution Insights