DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Prior arts cited in this office action:
Rolland et al. (US 12393597 B1, hereinafter “Rolland”)
Doan et al. (US 20250111648 A1, hereinafter “Doan”)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Rolland et al. (US 12393597 B1, hereinafter “Rolland”) in view of Doan et al. (US 20250111648 A1, hereinafter “Doan”).
Regarding claims 1, 16 and 20:
Rolland teaches a computer-implemented method for determining image relevance of a digital image based on a category-specific embedding (Rolland page 1 line 30-page 2 line 2; col. 3 lines 15-26), where Rolland teaches A method and system for optimally weighting search results retrieved using more than one vector-based search operator. A dynamic weighting module includes two or more embedding generators that transform a natural language search query into corresponding query embeddings. Each embedding generator may correspond to a respective different embedding model (e.g., one embedding generator may correspond to a textual embedding model and a second embedding generator may correspond to an image embedding model)), comprising:
generating a text embedding based on a category label and user input, based on the user input (Rolland col, 17 line 25-col. 19 line 18, figs. 3A and 3B, where Rolland teaches in examples, one of the two or more embedding generators 320 may be a textual embedding generator 320a, for example, for enabling the comparison of a search query text with a database of text embeddings 250a (e.g., product text embeddings in an e-commerce platform, among other text embeddings). In examples, the textual embedding generator 320a may include the E5 text embedding model (e.g., EmbEddings from bidirectional Encoder rEpresentations) which is a natural language processing model for encoding the user input 310, or another textual embedding model may be used);
obtaining an image embedding for an image belonging to a set of images identified based
on the user input (Rolland col, 17 line 25-col. 19 line 18, figs. 3A and 3B, where Rolland teaches In some examples, an image embedding generator may be used for enabling the comparison of a search query text with a database of image embeddings 250b or the multimodal embedding generator 320b may include an image embedding model for enabling the comparison of a search query text with a database of image embeddings 250b, for example, by encoding text and images into a shared embedding space);
generating a similarity score by combining the text embedding and the image embedding input (Rolland col, 17 line 25-col. 19 line 18, figs. 3A and 3B, where Rolland teaches In some examples, a multimodal embedding generator 320b may include a multimodal embedding model that has been trained to encode an input having more than one modality (e.g., a combination of text, image, video etc.) into a shared embedding space.;
determining that the image is an outlier image for the set of images based on comparing
the similarity score to a category-specific relevance threshold, wherein the category-specific
relevance threshold is selected from a set of category-specific relevance thresholds associated with the set of category labels (Rolland col. 21 lines 41-col. 22 line 8, where Rolland teaches In some implementations, the combined search result set 380 may represent a pre-determined number of the highest ranked search results in the search result sets 335 (e.g., the 50 highest ranked candidate embeddings, or the 500 highest ranked candidate embeddings, etc.), or in other implementations, the combined search result set 380 may represent a subset of the ranked search results in the search result sets 335 having a magnitude of a weighted similarity score above a pre-determined threshold, among other possibilities, with effect that the combined search result set 380 represents a more relevant search result set);
removing the image from the set of images based on the image being an outlier image for
the set of images (Rolland col. 22 lines 28-65, where Rolland teaches A traditional filter of a result set may simply exclude results that do not match the filter, while the remaining search results are maintained in the same order); and
providing the set of images without the outlier image in response to the user input (Rolland col. 22 lines 28-65, where Rolland teaches A traditional filter of a result set may simply exclude results that do not match the filter, while the remaining search results are maintained in the same order).
Rolland fails to explicitly teach wherein the labels are a plurality of sets of labels.
However, Rolland teaches the parameters of the CNN may be learned through training, using data having ground truth labels specific to the desired task (e.g., class labels if the CNN is being trained for a classification task, pixel masks if the CNN is being trained for a segmentation task, text annotations if the CNN is being trained for a captioning task, etc.)
Doan further teaches The classification system 408 includes a classifier 410 (e.g., a classifier configured to classify input signals into one or more labels using a machine-learning (ML) algorithm, such as a neural network), a hyperbolic contrastive loss module 412 configured to learn a hierarchical structural representation for each class, and a SuperClass Regularizer module 414 configured to regularize classes such that classes belonging to a same category are closer to one another in an embedding space (e.g., a hyperbolic embedding space) while being further away from classes of different categories. An Adaptive Relabeling module 416 leverages the learned hierarchical structure to relabel proposed candidates as unknowns if they meet a specific criterion as described below in more detail (Doan [0004], [0006], [0064]).
Therefore, it would have been obvious to one of or ordinary skill in the art before the effective filing date of the application to be able to select a category label among a set of category labels based on user input since that would allow for moving each of the projected embeddings closer to embeddings in a same category of the plurality of categories and further away from embeddings in different categories of the plurality of categories that would allow for faster and/or better search results (Doan [0079]).
Regarding claim 2:
Rolland in view of Doan teaches wherein generating the text embedding includes:
identifying the category label based on the user input;
generating a combined text string based on the category label and the user input; and
generating the text embedding by providing the combined text string to a text encoder
neural network (Rolland col. 2 line 56-col. 3 line 15, fig. 1; Doan [0004], [0006], [0064], where the combination teaches Examples of the disclosed dynamic weighting module may improve hybrid search engine performance by incorporating two or more different vector-based search operators for encoding both textual and image information about objects (e.g., products, websites, items etc.). In other examples, the user input 310 may be an audio input, for example, received via a microphone of computing system 200, or the user input 310 may be received in another format, for example, as a touch input, or the user input 310 may be received as a selection of an item (e.g., a topic or category, or another object) on a webpage of an e-commerce platform, among other inputs).
Regarding claim 3:
Rolland in view of Doan teaches wherein generating the text embedding includes:
identifying a category label text embedding based on the user input;
generating a user input text embedding by providing the user input to a text encoder neural network; and
generating the text embedding by combining the category label text embedding and the
user input text embedding (Rolland col. 2 line 56-col. 3 line 15, fig. 1; Doan [0004], [0006], [0064where Rolland teaches Examples of the disclosed dynamic weighting module may improve hybrid search engine performance by incorporating two or more different vector-based search operators for encoding both textual and image information about objects (e.g., products, websites, items etc.). In other examples, the user input 310 may be an audio input, for example, received via a microphone of computing system 200, or the user input 310 may be received in another format, for example, as a touch input, or the user input 310 may be received as a selection of an item (e.g., a topic or category, or another object) on a webpage of an e-commerce platform, among other inputs).
Regarding claim 4:
Rolland in view of Doan teaches further comprising:
identifying a set of hierarchical category labels associated with the user input; and selecting the category label from the set of hierarchical category labels based on the category label having a most specific hierarchy among category labels within the set of hierarchical category labels (Rolland col. 2 line 56-col. 3 line 15, col. 10 lines 37-49, fig. 1; Doan [0004], [0006], [0064], [0112]-[0113]).
Regarding claim 5:
Rolland in view of Doan teaches wherein generating the similarity score includes determining a cosine similarity between the text embedding and the image embedding (Rolland col. 18 lines 36-44; Doan [0021], [0080], where Rolland teaches In examples, a nearest neighbor approach may be used to identify the set of candidate embeddings, where the similarity measure may be a distance measure, such as a Euclidean distance measured between the query embedding 325 and the candidate embeddings in any direction within the embedding space, or the similarity measure may be a cosine similarity (e.g., a cosine of the angle between the query embedding 325 and the candidate embeddings), among other possibilities).
Regarding claim 6:
Rolland in view of Doan teaches wherein determining that the image is an outlier image for the set of images includes determining that the similarity score does not meet the category-specific relevance threshold for the category label input (Rolland col, 17 line 25-col. 19 line 18, figs. 3A and 3B; Doan [0004], [0006], [0032], [0064]).
Regarding claim 7:
Rolland in view of Doan teaches wherein providing the set of images without the outlier image in response to the user input includes:
combining the set of images and a text response responding to a user query into a
multimodal response, the user input including the user query; and providing the multimodal response in response to the user query (Rolland col, 17 line 25-col. 19 line 18, col. 21 lines 41-col. 22 line 8, figs. 3A and 3B; Doan [0004]-[0006], [0064]).
Regarding claim 8:
Rolland in view of Doan teaches further comprising:
generating similarity scores between multiple image embeddings of multiple images in the set of images and the text embedding; and
ranking the multiple images based on corresponding similarity scores, wherein providing the set of images in response to the user input includes providing one or more of the multiple images in the set of images based on similarity score rankings (Rolland col. 6 lines 41-62; Doan [0004]- [0006], [0064]).
Regarding claim 9:
Rolland in view of Doan teaches further comprising:
obtaining an additional image embedding for an additional image belonging to the set of
images;
generating an additional similarity score by combining the text embedding and the
additional image embedding;
determining that the additional image is not an outlier image for the set of images based on the additional similarity score meeting the category-specific relevance threshold; and
providing the set of images with the additional image in response to the user input (Rolland col. 6 lines 41-62, col, 17 line 25-col. 19 line 18, col. 21 lines 41-col. 22 line 8, figs. 3A and 3B; Doan [0004]-[0006], [0064]).
Regarding claims 10 and 17:
Rolland in view of Doan teaches further comprising:
identifying a collection of candidate images associated with the category label;
providing the collection of candidate images to a generative artificial intelligence (AI)
model with instructions to determine relevance scores between each candidate image and the
category label;
generating a set of training images that classify the collection of candidate images into a
positive subset of candidate images having relevance scores that meet a relevance score threshold and a negative subset of candidate images having relevance scores that do not meet a relevance score threshold; and
determining the category-specific relevance threshold for the category label based on the
set of training images (Rolland col. 9-col. 10, col. 11 lines 22-37; Doan [0004], [0006], [0064]).
Regarding claim 11:
wherein the collection of candidate images associated with the category label is received from an image retrieval system (Rolland col. 9-col. 10).
Regarding claim 12:
Rolland in view of Doan teaches wherein the relevance scores for each candidate image include a binary relevance score indicating whether a candidate image is relevant to the category label (Rolland col. 9-col. 10, where each output uses for the ground truth can be positive or negative and the generative network would operate and update the weights accordingly).
Regarding claim 13:
Rolland in view of Doan teaches wherein determining the category- specific relevance threshold for the category label includes:
generating a set of image encodings for the set of training images using an image encoding neural network;
generating a set of similarity scores for the set of training images by combining the set of
image encodings with the text embedding of the category label;
mapping the set of similarity scores to a mapping space to generate a graphical plot curve;
and
determining the category-specific relevance threshold for the category label based on
applying a measurement to the graphical plot curve (Rolland col. 9-col. 10; Doan [0004], [0006], [0064])
Regarding claim 14:
Rolland in view of Doan teaches wherein:
the graphical plot curve is a receiver operating characteristic (ROC) curve; and
applying the measurement to the graphical plot curve includes determining the category-
specific relevance threshold for the category label based on an area under the ROC curve
measurement (Doan [0004], [0006], [0064], where a distribution curve can be generated to determine the where each measurement falls within the curve to determine whether they belong to a particular class or not. The curve can help determine where and how to set the threshold value )
Regarding claim 15:
Rolland in view of Doan teaches further comprising:
receiving a user query that includes the user input, wherein the user input indicates an
entity; determining an entity identifier for the entity based on the user input; determining the category label assigned to the entity identifier; and identifying the set of images based on the set of images being associated with the entity identifier (Rolland col, 17 line 25-col. 19 line 18, figs. 3A and 3B; Doan [0004], [0006], [0064]).
Regarding claim 18:
Rolland in view of Doan teaches wherein obtaining the text embedding includes:
generating the text embedding for the category label before receiving the user input;
storing the text embedding in a text embedding data store;
upon receiving the user input, determining that the user input is associated with the
category label; and
obtaining the text embedding for the category label from the text embedding data store (Rolland col, 17 line 25-col. 19 line 18, figs. 3A and 3B; Doan [0004], [0006], [0064]).
Regarding claim 19:
Rolland in view of Doan teaches wherein obtaining the image embedding includes generating the image embedding by providing the image to an image encoder neural network to generate the image embedding (Rolland col, 17 line 25-col. 19 line 18, figs. 3A and 3B; Doan [0004], [0006], [0064]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEDNEL CADEAU whose telephone number is (571)270-7843. The examiner can normally be reached Mon-Fri 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chieh Fan can be reached at 571-272-3042. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEDNEL CADEAU/Primary Examiner, Art Unit 2632 June 8, 2026
Prosecution Insights