DETAILED ACTION
This final action is in response to the amendment and remarks filed on 03/30/2026 for application 17/910,756.
Claims 1, 6, 8, and 14 have been amended.
Claims 1-10 and 14-19 remain pending in the application. Claims 1, 10, and 14 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-10 and 14-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis of the claims will follow the 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50 (“2019 PEG”).
Independent Claims (Claim 1, Claim 10, Claim 14):
Step 1: Claim 1 is drawn to a method, claim 10 is drawn to a method, and claim 14 is drawn to a system/apparatus. Therefore, each of these claims falls under one of the four categories of statutory subject matter (process/method, machine/apparatus, manufacture/product, or composition of matter).
Step 2A Prong 1: Claims 1, 10, and 14 each recite a judicially recognized exception of an abstract idea.
Claim 1 recites, inter alia:
identifying a batch of training data that includes a set of queries, a set of electronic resources, and ground truth pairings, wherein each of the ground truth pairings comprises one query of the set of queries paired to one electronic resource of the set of electronic resources that is responsive to the query, and wherein each of the electronic resources corresponds to only one of the ground truth pairings – This limitation amounts to a process of observing data, including data represented in a digital form (electronic resources), and identifying relationships (pairings) between the observed data through reasoning and logic. It therefore recites a process of evaluation that a human could reasonably perform in the mind or using pen and paper.
for each query in the set of queries, generating a relevance score based on (1) the corresponding query vector generated for the query and (2) the corresponding electronic resource vector generated for the electronic resource paired to the query; – This limitation amounts to a process of using mathematical methods of vector comparison (query vector and electronic resource vector) to determine a numerical value (relevance score), and therefore recites mathematical calculation.
and generating, for each electronic resource that is in addition to the electronic resource paired to the query, a corresponding negative relevance score based on (1) the corresponding query vector generated for the query and (2) the corresponding electronic resource vector generated for the electronic resource that is in addition to the corresponding electronic resource paired to the query; – This limitation likewise amounts to a process of using mathematical methods of vector comparison (query vector and electronic resource vector) to determine a numerical value (negative relevance score), and therefore further recites mathematical calculation.
generating a query loss based on (1) the relevance score generated for the query and (2) at least one corresponding negative relevance score generated for at least one additional query in the set of queries; generating a batch loss, for the batch of training data, based on the generated query losses; – This limitation amounts to a process of using mathematical methods to determine numerical loss values (query loss, batch loss) from numerical parameters (relevance score, negative relevance score), and therefore further recites mathematical calculation.
Claim 10 recites, inter alia:
determining a bounding box for the object captured in the image; – This limitation amounts to a process of observing an image, and drawing a rectangular frame or boundary (bounding box) surrounding an observed object or region within the image. It therefore recites a process of evaluation that a human could reasonably perform in the mind or using pen and paper.
wherein determining the bounding box for the object comprises: comparing the image vector to a plurality of pre-stored candidate bounding box vectors, selecting a pre-stored candidate bounding box vector based on the comparing; and determining the bounding box for the object based on the selected pre-stored candidate bounding box vector; – This limitation amounts to a process of using mathematical methods of vector comparison (image vector and bounding box vector) and mapping to determine coordinates of a rectangular boundary (determining bounding box based on the selected bounding box vector), and therefore recites mathematical calculation.
perform one or more actions based on the determined bounding box for the object – This limitation amounts to a process of performing generic actions as a response to conclusions drawn from the observed calculations, and therefore further recites a process of evaluation that a human could reasonably perform in the mind or using pen and paper.
Claim 14 recites substantially similar abstract idea limitations to those found in claim 1, and therefore recites the same judicial exception.
Step 2A Prong 2: The following additional elements recited in claims 1, 10, and 14 also do not integrate the recited judicial exceptions into a practical application.
Claim 1 additionally recites:
for each query in the set of queries, generating a corresponding query vector by processing the query; for each electronic resource in the set of electronic resources, generating a corresponding electronic resource vector by processing the electronic resource – These limitations do no more than recite pre-solution steps of gathering and pre-processing data for further analysis, and therefore recite insignificant extra-solution activity.
A method implemented by one or more processors; [processing the query] using an input model; [processing the electronic resource] using a resource model; updating one or more portions of the input model and/or the resource model based on the generated batch loss – These limitations do no more than invoke generic computing components or computational models as tools to perform existing processes.
Claim 10 additionally recites:
receiving an image capturing an object; generating an image vector by processing the image; wherein each pre-stored candidate bounding box vector, in the plurality of pre-stored candidate bounding box vectors, is previously generated by processing a corresponding candidate bounding box; – These limitations do no more than recite pre-solution steps of gathering and pre-processing data for further analysis, and therefore recite insignificant extra-solution activity.
[processing the image] using an input model; [processing a corresponding candidate bounding box] using a resource model; and causing a computing device to [perform one or more actions] – These limitations do no more than invoke generic computational models, computers, and/or computing components as tools to perform existing processes.
Claim 14 recites substantially similar additional elements to those found in claim 1, and further recites:
A system comprising: memory storing instructions; one or more processors operable to execute the instructions, stored in the memory, to: – These limitations amount to mere instructions to implement an abstract idea on a computer or computer components.
Step 2B: The additional elements recited in claims 1, 10, and 14, viewed individually or as an ordered combination, do not provide an inventive concept or otherwise amount to significantly more than the recited abstract ideas themselves.
Claim 1 additionally recites:
for each query in the set of queries, generating a corresponding query vector by processing the query; for each electronic resource in the set of electronic resources, generating a corresponding electronic resource vector by processing the electronic resource – Applying deep architectures to learning multi-modal (e.g., image and text) embeddings is well-understood, routine, and conventional activity (see Gu et al., “Multi-Modal and Multi-Domain Embedding Learning for Fashion Retrieval and Analysis” [page 1524 Introduction, pages 1525-1526 Related Work – Deep Learning Based Techniques]), and therefore does not provide an inventive concept or significantly more to the recited abstract idea.
A method implemented by one or more processors; [processing the query] using an input model; [processing the electronic resource] using a resource model; updating one or more portions of the input model and/or the resource model based on the generated batch loss – Invoking generic computing components and/or computational models as mere tools to perform existing processes (e.g., via general recitation of model updation) does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 10 additionally recites:
receiving an image capturing an object; generating an image vector by processing the image; wherein each pre-stored candidate bounding box vector, in the plurality of pre-stored candidate bounding box vectors, is previously generated by processing a corresponding candidate bounding box; – Applying deep architectures to learning multi-modal embeddings (see Gu et al., “Multi-Modal and Multi-Domain Embedding Learning for Fashion Retrieval and Analysis” [page 1524 Introduction, pages 1525-1526 Related Work – Deep Learning Based Techniques]) and applying deep architectures to vision analysis tasks (e.g., object recognition, visual tracking) (see Hu et al., “Deep Metric Learning for Visual Tracking”, [pages 2057-2058 Deep Learning]) are well-understood, routine, and conventional activities, and therefore do not provide an inventive concept or significantly more to the recited abstract idea.
[processing the image] using an input model; [processing a corresponding candidate bounding box] using a resource model; and causing a computing device to [perform one or more actions] – Invoking generic computational models, computers, and/or computing components as tools to perform existing processes does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 14 recites substantially similar additional elements to those found in claim 1, and further recites:
A system comprising: memory storing instructions; one or more processors operable to execute the instructions, stored in the memory, to: – Mere instructions to implement an abstract idea on a computer or computer components do not provide an inventive concept or significantly more to the recited abstract idea.
Even when considered as an ordered combination, the additional elements recited in the claims ultimately do no more than enable an abstract procedure of calculation and reasoning through steps of gathering and pre-processing input data and invoking generic underlying computational models as a means of execution. As such, claims 1, 10, and 14 are not patent eligible.
Dependent Claims (Claims 2-9, Claims 15-19):
Dependent claims 2-9 and 15-19 narrow the scope of independent claims 1 and 10, and thus merely narrow the recited judicial exceptions. With respect to the independent claims, the recited judicial exceptions are not meaningfully integrated into a practical application, and also do not amount to significantly more than the recited abstract ideas themselves. The dependent claims recite abstract idea limitations similar to those recited within the independent claims, as they also do not provide anything more than mathematical concepts or mental processes that are capable of being performed in the human mind and/or using pen and paper. The dependent claims also do not recite any further additional elements that successfully integrate the recited judicial exceptions into a practical application or amount to significantly more than the recited abstract ideas themselves. Consequently, claims 2-9 and 15-19 are also rejected under 35 U.S.C. 101.
Step 1: Claims 2-9 are drawn to a method, and claims 15-19 are drawn to a system/apparatus. Therefore, each of these claims falls under one of the four categories of statutory subject matter (process/method, machine/apparatus, manufacture/product, or composition of matter).
Step 2A Prong 1: Claims 2-9 and 15-19 each recite a judicially recognized exception of an abstract idea.
Claim 2 recites, inter alia:
generating the query loss based on (1) the relevance score generated for the query, (2) the corresponding negative relevance scores generated for the query, and (3) all of the corresponding negative relevance scores generated for each of the additional queries in the set of queries – Expanding upon the procedure recited in the parent claim, this limitation likewise amounts to a process of using mathematical methods to determine numerical loss values (query loss) from numerical parameters (relevance score, negative relevance score), and therefore further recites mathematical calculation.
Claim 3 recites, inter alia:
selecting a subset of the negative relevance scores, wherein the selected subset includes at least one negative relevance score generated for at least one additional query in the set of queries, and wherein selecting the subset is based on the corresponding negative relevance scores of the subset satisfying one or more conditions; and generating the query loss based on (1) the relevance score generated for the query and (2) the subset of the corresponding negative relevance scores – Expanding upon the procedure recited in the parent claim, this limitation likewise amounts to a process of using mathematical methods (e.g., selecting subsets of scores) to determine numerical loss values (query loss) from numerical parameters (relevance score, negative relevance score), and therefore further recites mathematical calculation.
Claim 4 recites the same judicial exception as claim 1.
Claim 5 recites the same judicial exception as claim 1.
Claim 6 recites, inter alia:
determining a user electronic resource responsive to the user query – This limitation amounts to a process of observing query data (e.g., in text or image form) and identifying, through logic and reasoning, an additional piece of data (e.g., text or image represented in a digital form) that is in some way related to content in the query data. It therefore recites a process of evaluation that a human could reasonably perform in the mind or using pen and paper.
wherein determining the user electronic resource responsive to the user query comprises: comparing the user query vector with a plurality of pre-stored candidate electronic resource vectors, selecting a pre-stored candidate electronic resource vector based on the comparing; and determining the user electronic resource based on the selected pre-stored candidate electronic resource vector – Expanding upon the procedure recited in the parent claim, this limitation amounts to a process of using mathematical methods of vector comparison (user query vector and electronic resource vector) and mapping to identify a resource (electronic resource) corresponding to a determined value (electronic resource vector), and therefore further recites mathematical calculation.
and perform[ing] one or more actions based on the determined user electronic resource – Expanding upon the procedure recited in the parent claim, this limitation amounts to a process of performing generic actions as a response to conclusions drawn from the observed calculations, and therefore further recites a process of evaluation that a human could reasonably perform in the mind or using pen and paper.
Claim 7 recites, inter alia:
determining a dot product between (1) the corresponding query vector generated for the query and (2) the corresponding electronic resource vector generated for the electronic resource with the ground truth pairing to the query; and generating the relevance score based on the determined dot product – Expanding upon the procedure recited in the parent claim, this limitation likewise amounts to a process of using mathematical methods (e.g., dot product) of vector comparison (query vector and electronic resource vector) to determine a numerical value (relevance score), and therefore further recites mathematical calculation.
Claim 8 recites, inter alia:
determining a dot product between (1) the corresponding query vector generated for the query and (2) the corresponding electronic resource vector generated for the electronic resource that is in addition to the corresponding electronic resource paired to the query; and generating the negative relevance score based on the determined dot product – Expanding upon the procedure recited in the parent claim, this limitation likewise amounts to a process of using mathematical methods (e.g., dot product) of vector comparison (query vector and electronic resource vector) to determine a numerical value (negative relevance score), and therefore further recites mathematical calculation.
Claim 9 recites the same judicial exception as claim 1.
Claims 15-19 recite substantially similar abstract idea limitations to those found in claims 2-5 and 9, and therefore recite the same judicial exceptions.
Step 2A Prong 2: Claims 2-3, 7-8, and 15-16 do not recite any further additional elements besides those recited in the independent claims, and the following additional elements recited in claims 4-6, 9, and 17-19 also do not integrate the recited judicial exceptions into a practical application.
Claim 4 additionally recites:
wherein each query, in the set of queries, is a natural language query, and wherein each electronic resource, in the set of electronic resources, is an image or a web page – This limitation amounts to specifying particular data sources and/or types of data to be manipulated, and therefore recites insignificant extra-solution activity.
Claim 5 additionally recites:
wherein each query, in the set of queries, is an image capturing an object, and wherein each electronic resource, in the set of electronic resources, represents one or more corresponding bounding boxes – This limitation amounts to specifying particular data sources and/or types of data to be manipulated, and therefore recites insignificant extra-solution activity.
Claim 6 additionally recites:
deploying the trained input model on a computing system; – Wherein the parent claim recites updation of a underlying input model as a means of merely invoking a generic computational model as tool to perform existing processes, this limitation likewise amounts to further instructions to generically apply the model on a computer to perform existing processes.
[receiving a user query] via one or more user interface input devices of the computing system; [processing the user query] using the trained input model; [processing a corresponding electronic resource] using the resource model; and causing the computing system to [perform one or more actions] – These limitations do no more than further invoke generic computational models, computers, and/or computing components as tools to perform existing processes.
receiving a user query; determining a user query vector by processing the user query; wherein each pre-stored candidate electronic resource vector, in the plurality of pre-stored candidate electronic resource vectors, is previously generated by processing a corresponding electronic resource – These limitations do no more than recite pre-solution steps of gathering and pre-processing data for further analysis, and therefore recite insignificant extra-solution activity.
Claim 9 additionally recites:
wherein the generated query vector, for each query in the set of queries, projects the query into a shared embedding space, and wherein the electronic resource vector, for each electronic resource in the set of electronic resources, projects the electronic resource into the shared embedding space – These limitations do no more than recite pre-solution steps of pre-processing data for further analysis, and therefore recite insignificant extra-solution activity.
Claims 17-19 recite substantially similar additional elements to those found in claims 4-5 and 9, and therefore also do not integrate the recited judicial exceptions into a practical application.
Step 2B: The additional elements recited in claims 4-6, 9, and 17-19, viewed individually or as an ordered combination, do not provide an inventive concept or otherwise amount to significantly more than the recited abstract ideas themselves.
Claim 4 additionally recites:
wherein each query, in the set of queries, is a natural language query, and wherein each electronic resource, in the set of electronic resources, is an image or a web page – Applying deep architectures to multi-modal retrieval tasks (e.g., text-based image retrieval) is well-understood, routine, and conventional activity (see Hua et al., “Deep Multi-Modal Metric Learning with Multi-Scale Correlation for Image-Text Retrieval”, [page 2 Introduction]) and therefore does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 5 additionally recites:
wherein each query, in the set of queries, is an image capturing an object, and wherein each electronic resource, in the set of electronic resources, represents one or more corresponding bounding boxes – Applying deep architectures to vision analysis tasks (e.g., object recognition, visual tracking) is well-understood, routine, and conventional activity (see Hu et al., “Deep Metric Learning for Visual Tracking”, [pages 2057-2058 Deep Learning]) and therefore does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 6 additionally recites:
deploying the trained input model on a computing system; – Further instructions to generically apply an underlying model on a computer to perform existing processes do not provide an inventive concept or significantly more to the recited abstract idea.
[receiving a user query] via one or more user interface input devices of the computing system; [processing the user query] using the trained input model; [processing a corresponding electronic resource] using the resource model; and causing the computing system to [perform one or more actions] – Generically invoking computational models, computers, and/or computing components as tools to perform existing processes does not provide an inventive concept or significantly more to the recited abstract idea.
receiving a user query; determining a user query vector by processing the user query; wherein each pre-stored candidate electronic resource vector, in the plurality of pre-stored candidate electronic resource vectors, is previously generated by processing a corresponding electronic resource – Applying deep architectures to learning multi-modal (e.g., image and text) embeddings is well-understood, routine, and conventional activity (see Gu et al., “Multi-Modal and Multi-Domain Embedding Learning for Fashion Retrieval and Analysis” [page 1524 Introduction, pages 1525-1526 Related Work – Deep Learning Based Techniques]), and therefore does not provide an inventive concept or significantly more to the recited abstract idea.
Claim 9 additionally recites:
wherein the generated query vector, for each query in the set of queries, projects the query into a shared embedding space, and wherein the electronic resource vector, for each electronic resource in the set of electronic resources, projects the electronic resource into the shared embedding space – Applying deep architectures to learning multi-modal (e.g., image and text) embeddings (i.e., vectors in a shared embedding space) is well-understood, routine, and conventional activity (see Gu et al., “Multi-Modal and Multi-Domain Embedding Learning for Fashion Retrieval and Analysis” [page 1524 Introduction, pages 1525-1526 Related Work – Deep Learning Based Techniques]), and therefore does not provide an inventive concept or significantly more to the recited abstract idea.
Claims 17-19 recite substantially similar additional elements to those found in claims 4-5 and 9, and therefore also do not provide an inventive concept or significantly more to the recited abstract ideas.
Even when considered as an ordered combination, the additional elements recited in the claims ultimately do no more than enable an abstract procedure of calculation and reasoning through steps of gathering and pre-processing input data and invoking generic underlying computational models, and/or generically tie the claimed procedure to known and conventional techniques in the art. As such, claims 2-9 and 15-19 also are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 6-9, 14-16, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Sohn (“Improved Deep Metric Learning with Multi-class N-pair Loss Objective”, published 2016, cited in IDS filed 09/09/2022) in view of Mishchuk et al. (“Working hard to know your neighbor’s margins”, available conference 2017), hereinafter Mishchuk.
Regarding claim 1, Sohn teaches A method implemented by one or more processors (“In this paper, we propose to address this problem with a new metric learning objective called multi-class N-pair loss. The proposed objective function firstly generalizes triplet loss by allowing joint comparison among more than one negative examples – more specifically, N-1 negative examples – and secondly reduces the computational burden of evaluating deep embedding vectors via an efficient batch construction strategy using only N pairs of examples, instead of (N+1)×N. We demonstrate the superiority of our proposed loss to the triplet loss as well as other competing loss functions for a variety of tasks on several visual recognition benchmark, including fine-grained object recognition and verification, image clustering and retrieval, and face verification and identification” [Sohn Abstract]), the method comprising:
identifying a batch of training data that includes a set of queries, a set of electronic resources, and ground truth pairings, wherein each of the ground truth pairings comprises one query of the set of queries paired to one electronic resource of the set of electronic resources that is responsive to the one query, and wherein each of the electronic resources corresponds to only one of the ground truth pairings (“Suppose we directly apply the (N+1)-tuplet loss to the deep metric learning framework. When the batch size of SGD is M, there are M×(N+1) examples to be passed through f at one update. Since the number of examples to evaluate for each batch grows in quadratic to M and N, it again becomes impractical to scale the training for a very deep convolutional networks. Now, we introduce an effective batch construction to avoid excessive computational burden. Let {(x1, x1+), · · · ,(xN , xN+)} be N pairs of examples from N different classes, i.e., yi =/ yj , ∀i =/ j. We build N tuplets, denoted as {Si} i=1..N, from the N pairs, where Si = {xi , x1+ , x2+ , · · · , xN+ }. Here, xi is the query for Si , xi+ is the positive example and xj+, j =/ i are the negative examples. Figure 2(c) illustrates this batch construction process” [Sohn page 3 N-pair loss for efficient deep metric learning]; see Figure 2(c) N-pair-mc loss [Sohn page 4]; The disclosed batch construction process (i.e., identifying a batch of training data) comprises construction of N tuplets, where each tuplet comprises a query xi, a single positive example (i.e., electronic resource) xi+ (i.e., {xi, xi+} being ground truth pairing), and additional negative examples (i.e., electronic resources) xj+, j =/ i)
for each query in the set of queries, generating a corresponding query vector by processing the query using an input model; (“Let x ∈ X be an input data and y ∈ {1, · · · , L} be its output label. We use x + and x − to denote positive and negative examples of x, meaning that x and x + are from the same class and x − is from different class to x. The kernel f(·; θ) : X → R K takes x and generates an embedding vector f(x). We often omit x from f(x) for simplicity, while f inherits all superscripts and subscripts.” [Sohn page 2 Preliminary: Distance Metric Learning]; see Figure 2(c) N-pair-mc loss including {fi} i=1...N – “…the N-pair batch construction in (c) leverages all 2 × N embedding vectors to build N distinct (N+1)-tuplets with {fi} N i=1 as their queries” [Sohn page 4]; Input data xi (i.e., queries), and their corresponding positive/negative examples (i.e., electronic resources), are passed through kernel f() (i.e., model) to generate corresponding vector embeddings, which are then further used in the batch construction procedure)
for each electronic resource in the set of electronic resources, generating a corresponding electronic resource vector by processing the electronic resource using a resource model; ([Sohn page 2 Preliminary: Distance Metric Learning] and Figure 2(c) [Sohn page 4] as detailed above)
for each query in the set of queries,
generating a relevance score based on (1) the corresponding query vector generated for the query and (2) the corresponding electronic resource vector generated for the electronic resource paired to the query; (“Consider an (N+1)-tuplet of training examples {x, x+, x1, · · · , xN−1}: x+ is a positive example to x and {xi} i=1.. N−1 are negative. The (N+1)-tuplet loss is defined as follows:
PNG
media_image1.png
100
682
media_image1.png
Greyscale
where f(·; θ) is an embedding kernel defined by deep neural network” [Sohn page 3 Learning to identify from multiple negative examples]; In the disclosed loss function L({x,x+,{xi}i=1..N-1}; f), fT is the transpose of f, the corresponding embedding vector for query x, and f+ is the corresponding embedding vector for x+, which is the positive example paired to the query – fTf+ (see red outline above) is thereby a calculated dot product (i.e., relevance score) of the two vectors) and
generating, for each electronic resource that is in addition to the electronic resource paired to the query, a corresponding negative relevance score based on (1) the corresponding query vector generated for the query and (2) the corresponding electronic resource vector generated for the electronic resource that is in addition to the corresponding electronic resource paired to the query (
PNG
media_image2.png
100
683
media_image2.png
Greyscale
[Sohn page 3 Learning to identify from multiple negative examples]; fTfi is the calculated dot product (i.e., negative relevance score) of vectors fT (i.e., query vector) and fi (i.e., vector corresponding to each negative example (i.e., electronic resource) {xi}i=1..N-1, i.e., vector for each example in addition to positive example x+))
generating a query loss based on (1) the relevance score generated for the query and (2) at least one corresponding negative relevance score; ([Sohn page 3 Learning to identify from multiple negative examples] as detailed above; (N+1)-tuplet loss
PNG
media_image3.png
36
257
media_image3.png
Greyscale
(i.e., query loss) is based on dot products (i.e., relevance scores)
PNG
media_image4.png
38
67
media_image4.png
Greyscale
(positive) and
PNG
media_image5.png
40
52
media_image5.png
Greyscale
(negative))
generating a batch loss, for the batch of training data, based on the generated query losses; (“Figure 2(c) illustrates this batch construction process. The corresponding (N+1)-tuplet loss, which we refer to as the multi-class N-pair loss (N-pair-mc), can be formulated as follows:2
PNG
media_image6.png
86
792
media_image6.png
Greyscale
” [Sohn page 4 N-pair loss for efficient deep metric learning]; N-pair-mc loss
PNG
media_image7.png
54
269
media_image7.png
Greyscale
is determined based on an averaging of tuplet losses
PNG
media_image8.png
66
344
media_image8.png
Greyscale
(i.e., query losses) determined for each tuplet in the batch of N tuplets) and
updating one or more portions of the input model and/or the resource model based on the generated batch loss (“We assess the impact of our proposed N-pair loss functions, such as multi-class N-pair loss (N-pairmc) or one-vs-one N-pair loss (N-pair-ovo), on several generic and fine-grained visual recognition and verification tasks…We use Adam [11] for mini-batch stochastic gradient descent with data augmentation” [Sohn page 5 Experimental Results]; “Besides, we use Adam for stochastic optimization and other hyperparameters such as learning rate are tuned accordingly via 5-fold cross-validation on the train set” [Sohn page 6 Distance metric learning for unseen object recognition]).
However, Sohn does not expressly teach generating a query loss based on (2) at least one corresponding negative relevance score generated for at least one additional query in the set of queries.
In the same field of endeavor, Mishchuk teaches a deep metric learning framework applied to visual recognition tasks (“We introduce a loss for metric learning, which is inspired by the Lowe’s matching criterion for SIFT. We show that the proposed loss, that maximizes the distance between the closest positive and closest negative example in the batch, is better than complex regularization methods; it works well for both shallow and deep convolution network architectures. Applying the novel loss to the L2Net CNN architecture results in a compact descriptor named HardNet. It has the same dimensionality as SIFT (128) and shows state-of-art performance in wide baseline stereo, patch verification and instance retrieval benchmarks” [Mishchuk Abstract]) that generat[es] a query loss based on (2) at least one corresponding negative relevance score generated for at least one additional query in the set of queries (“Our learning objective mimics SIFT matching criterion. The process is shown in Figure 1. First, a batch X = (Ai; Pi)i=1::n of matching local patches is generated, where A stands for the anchor and P for the positive… Next, for each matching pair ai and pi the closest non-matching descriptors i.e. the 2nd nearest neighbor, are found respectively:
PNG
media_image9.png
346
949
media_image9.png
Greyscale
” [Mishchuk pages 2-3 Sampling and loss]; For each pair (ai, pi), the proposed loss determines distance to a closest non-matching descriptor (i.e., negative relevance score), wherein a closest non-matching descriptor can be determined based on distance to the positive descriptor from a different anchor (i.e. query) in the set of anchors a1…an).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated generating a query loss based on (2) at least one corresponding negative relevance score generated for at least one additional query in the set of queries as taught by Mishchuk into Sohn because they are both directed towards deep metric learning frameworks applied to visual recognition tasks. Given that Sohn expressly discloses an objective of mining hardest negatives through comparison among several negative examples while simultaneously reducing computational load through efficient batch construction (“With success of deep learning [13, 20, 23, 5], deep metric learning has received a lot of attention…Although yielding promising progress, such frameworks often suffer from slow convergence and poor local optima, partially due to that the loss function employs only one negative example while not interacting with the other negative classes per each update. Hard negative data mining could alleviate the problem, but it is expensive to evaluate embedding vectors in deep learning framework during hard negative example search…To address this problem, we propose an (N+1)-tuplet loss that optimizes to identify a positive example from N-1 negative examples. Our proposed loss extends triplet loss by allowing joint comparison among more than one negative examples; when N=2, it is equivalent to triplet loss. One immediate concern with (N+1)-tuplet loss is that it quickly becomes intractable when scaling up since the number of examples to evaluate in each batch grows in quadratic to the number of tuplets and their length N. To overcome this, we propose an efficient batch construction method that only requires 2N examples instead of (N+1)N to build N tuplets of length N+1” [Sohn page 1 Introduction]), a person of ordinary skill in the art would recognize the value of incorporating the bidirectional negative sampling method of Mishchuk, which comprehensively compares negative examples, to mine the hardest possible negative in the batch while maintaining computational efficiency (see Figure 1 – “Figure 1: Proposed sampling procedure. First, patches are described by the current network, then a distance matrix is calculated. The closest non-matching descriptor– shown in red– is selected for each ai and pi patch from positive pair (green) respectively. Finally, among two negative candidates the hardest one is chosen. All operations are done in a single forward pass” [Mishchuk page 2]).
Regarding claim 2, the combination of Sohn and Mishchuk teaches the limitations of parent claim 1, and Sohn further teaches generating the query loss based on (1) the relevance score generated for the query, (2) the corresponding negative relevance scores generated for the query ([Sohn page 3 Learning to identify from multiple negative examples] as detailed in claim 1 above). Mishchuk further teaches generating the query loss based on (3) all of the corresponding negative relevance scores generated for each of the additional queries in the set of queries (see Figure 1 – determined Distance matrix calculates distances (i.e., negative relevance scores for each additional anchor (e.g., for (a1,p1), distances d(a2,p1), d(a3, p1), d(a4,p1) are calculated) to select hardest negative).
Regarding claim 3, the combination of Sohn and Mishchuk teaches the limitations of parent claim 1, and Sohn further teaches selecting a subset of the negative relevance scores, and wherein selecting the subset is based on the corresponding negative relevance scores of the subset satisfying one or more conditions; (“Since the N-pair-mc loss already considers comparison to N-1 negative examples in its training objectives, negative data mining won’t be necessary in learning from small or medium-scale datasets in terms of the number of output classes. For datasets with large number of output classes, we propose a hard negative “class” mining scheme which greedily adds examples to form a batch from a class that violates the constraint with the previously selected classes in the batch” [Sohn pages 1-2 Introduction]; “To overcome such difficulty, we propose negative “class” mining, as opposed to negative “instance” mining, which greedily selects negative classes in a relatively efficient manner. More specifically, the negative class mining for N-pair loss can be executed as follows: 1. Evaluate Embedding Vectors: choose randomly a large number of output classes C; for each class, randomly pass a few (one or two) examples to extract their embedding vectors. 2. Select Negative Classes: select one class randomly from C classes from step 1. Next, greedily add a new class that violates triplet constraint the most w.r.t. the selected classes till we reach N classes. When a tie appears, we randomly pick one of tied classes [28]. 3. Finalize N-pair: draw two examples from each selected class from step 2” [Sohn pages 4-5 Hard negative class mining]; For datasets with large numbers of output classes, Sohn discloses a negative mining scheme to select a subset of negative examples based on their respective classes). and
generating the query loss based on (1) the relevance score generated for the query and (2) the subset of the corresponding negative relevance scores ([Sohn page 3 Learning to identify from multiple negative examples] as detailed in claim 1 above).
Mishchuk further teaches wherein the selected subset includes at least one negative relevance score generated for at least one additional query in the set of queries ([Mishchuk pages 2-3 Sampling and loss] as detailed in claim 1 above).
Regarding claim 6, the combination of Sohn and Mishchuk teaches the limitations of parent claim 1, and Sohn further teaches subsequent to updating the one or more portions of the input model and/or the resource model based on the generated batch loss (“We assess the impact of our proposed N-pair loss functions, such as multi-class N-pair loss (N-pairmc) or one-vs-one N-pair loss (N-pair-ovo), on several generic and fine-grained visual recognition and verification tasks…We use Adam [11] for mini-batch stochastic gradient descent with data augmentation” [Sohn page 5 Experimental Results]):
deploying the trained input model on a computing system (“We train networks for 40k iterations with 144 examples per batch. This corresponds to 72 pairs per batch for N-pair losses. We perform 5-fold cross-validation on the training set and report the average performance on the test set” [Sohn page 5 Fine-grained visual object recognition and verification]);
receiving a user query via one or more user interface input devices of the computing system; (“We evaluate deep metric learning algorithms on fine-grained object recognition and verification tasks. Specifically, we consider car and flower recognition problems on the following database: Car-333 [29] dataset is composed of 164, 863 images of cars from 333 model categories collected from the internet. Following the experimental protocol [29], we split the dataset into 157,023 images for training and 7,840 for testing. • Flower-610 dataset contains 61,771 images of flowers from 610 different flower species and among all collected, 58,721 images are used for training and 3,050 for testing” [Sohn page 5 Fine-grained visual object recognition and verification]; Flower/car images are received as queries to trained model for object recognition/verification tasks)
determining a user query vector by processing the user query using the trained input model; (“We evaluate both recognition and verification accuracy. Specifically, we consider verification setting where there are different number of negative examples from different classes, and determine as success only when the positive example is closer to the query example than any other negative example. Since the recognition task is involved, we also evaluate the performance of deep networks trained with softmax loss. The summary results are given in Table 1” [Sohn page 5 Fine-grained visual object recognition and verification]; As explained above ([Sohn page 2 Preliminary: Distance Metric Learning]), input query x is passed through kernel f() to generate embedding vector f(x) for comparison to existing positive/negative examples) determining a user electronic resource responsive to the user query, wherein determining the user electronic resource responsive to the user query comprises:
comparing the user query vector with a plurality of pre-stored candidate electronic resource vectors, wherein each pre-stored candidate electronic resource vector, in the plurality of pre-stored candidate electronic resource vectors, is previously generated by processing a corresponding electronic resource using the resource model; ([Sohn page 5 Fine-grained visual object recognition and verification] as detailed above; The input query vector is compared to existing positive/negative examples (i.e., pre-stored electronic resource vectors))
selecting a pre-stored candidate electronic resource vector based on the comparing; ([Sohn page 5 Fine-grained visual object recognition and verification] as detailed above; The vector closest to the query vector is the “selected” one) and
determining the user electronic resource based on the selected pre-stored candidate electronic resource vector; ([Sohn page 5 Fine-grained visual object recognition and verification] as detailed above; As explained above ([Sohn page 2 Preliminary: Distance Metric Learning]), each vector embedding corresponds to the underlying example and its respective class) and
causing the computing system to perform one or more actions based on the determined user electronic resource ([Sohn page 5 Fine-grained visual object recognition and verification] as detailed above; In a verification setting, a success is determined when the positive example is found to be closest to the query)
Regarding claim 7, the combination of Sohn and Mishchuk teaches the limitations of parent claim 1, and Sohn further teaches determining a dot product between (1) the corresponding query vector generated for the query and (2) the corresponding electronic resource vector generated for the electronic resource with the ground truth pairing to the query; ([Sohn page 3 Learning to identify from multiple negative examples] as detailed above; (N+1)-tuplet loss
PNG
media_image3.png
36
257
media_image3.png
Greyscale
(i.e., query loss) is based on dot products (i.e., relevance scores)
PNG
media_image4.png
38
67
media_image4.png
Greyscale
(positive) and
PNG
media_image5.png
40
52
media_image5.png
Greyscale
(negative)) and
generating the relevance score based on the determined dot product ([Sohn page 3 Learning to identify from multiple negative examples] as detailed above).
Regarding claim 8, the combination of Sohn and Mishchuk teaches the limitations of parent claim 1, and Sohn further teaches determining a dot product between (1) the corresponding query vector generated for the query and (2) the corresponding electronic resource vector generated for the electronic resource that is in addition to the corresponding electronic resource paired to the query; ([Sohn page 3 Learning to identify from multiple negative examples] as detailed above; (N+1)-tuplet loss
PNG
media_image3.png
36
257
media_image3.png
Greyscale
(i.e., query loss) is based on dot products (i.e., relevance scores)
PNG
media_image4.png
38
67
media_image4.png
Greyscale
(positive) and
PNG
media_image5.png
40
52
media_image5.png
Greyscale
(negative)) and
generating the negative relevance score based on the determined dot product ([Sohn page 3 Learning to identify from multiple negative examples] as detailed above).
Regarding claim 9, the combination of Sohn and Mishchuk teaches the limitations of parent claim 1, and Sohn further teaches wherein the generated query vector, for each query in the set of queries, projects the query into a shared embedding space, and wherein the electronic resource vector, for each electronic resource in the set of electronic resources, projects the electronic resource into the shared embedding space (“Distance metric learning aims to learn an embedding representation of the data that preserves the distance between similar data points close and dissimilar data points far on the embedding space [15, 30]. With success of deep learning [13, 20, 23, 5], deep metric learning has received a lot of attention. Compared to standard distance metric learning, it learns a nonlinear embedding of the data using deep neural networks…We unify the (N+1)- tuplet loss with our proposed batch construction method to form a novel, scalable and effective deep metric learning objective, called multi-class N-pair loss (N-pair-mc loss)” [Sohn page 1 Introduction]).
Regarding claims 14-16 and 19, they are system/apparatus claims that largely corresponds to the method of claims 1-3 and 9, which are already taught by the combination of Sohn and Mishchuk as detailed above. Sohn further teaches A system comprising: memory storing instructions; one or more processors operable to execute the instructions, stored in the memory, to: perform the claimed functions (“We evaluate deep metric learning algorithms on fine-grained object recognition and verification tasks. Specifically, we consider car and flower recognition problems on the following database: Car-333 [29] dataset is composed of 164, 863 images of cars from 333 model categories collected from the internet…• Flower-610 dataset contains 61,771 images of flowers from 610 different flower species” [Sohn page 5 Fine-grained visual object recognition and verification]; Evaluating deep metric learning algorithms on image datasets implicitly requires a computer or computing components (comprising adequate processing and memory components) capable of performing the necessary functions). Consequently, claims 14-16 and 19 are rejected for the same reasons as claims 1-3 and 9.
Claims 4 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Sohn and Mishchuk, as applied to claims 1 and 14 above, further in view of Hua et al. (“Deep Multi-Modal Metric Learning with Multi-Scale Correlation for Image-Text Retrieval”, published 10 March 2020), hereinafter Hua.
Regarding claim 4, the combination of Sohn and Mishchuk teaches the limitations of parent claim 1.
However, the combination does not expressly teach wherein each query, in the set of queries, is a natural language query, and wherein each electronic resource, in the set of electronic resources, is an image or a web page.
In the same field of endeavor, Hua teaches a deep metric learning framework applied to visual recognition tasks (“In this paper, we propose a deep multi-modal metric learning method with multi-scale semantic correlation to deal with the retrieval tasks between image and text modalities. A deep model with two branches is designed to nonlinearly map raw heterogeneous data into comparable representations” [Hua Abstract]) wherein each query, in the set of queries, is a natural language query, and wherein each electronic resource, in the set of electronic resources, is an image or a web page (“At the top in Figure 1, taking a text as query to retrieve the multi-modal dataset, related images, or texts are returned as the results, which are called text-to-image and text-to-text tasks respectively” [Hua page 1 Introduction]; “Figure 3 illustrates the framework of our method. Experimental on three widely-used datasets are conducted and the results demonstrate the effectiveness of our MS-DMML method on four kinds of retrieval tasks, i.e., image-to-text, text-to-image, image-to-image, and text-to-text” [Hua page 3 Introduction]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein each query, in the set of queries, is a natural language query, and wherein each electronic resource, in the set of electronic resources, is an image or a web page as taught by Hua into the combination because both Sohn and Hua are directed towards deep metric learning frameworks applied to visual recognition tasks. Given that Sohn already recognizes applicability of the disclosed N-pair procedure to image retrieval (“In experiment, we demonstrate the superiority of our proposed N-pair-mc loss to the triplet loss as well as other competing metric learning objectives on visual recognition, verification, and retrieval tasks” [Sohn page 2 Introduction]), one of ordinary skill in the art would recognize the value of incorporating the teachings of Hua to thereby enable retrieval tasks in a multi-modal (e.g., image and text) setting (“We propose a deep multi-modal metric learning method for image–text retrieval, in which two network branches are simultaneously learned as metric functions to measure the image–text distances according to multi-modal semantic relationship. Compared to most existing linear projection methods [2,3] and multilayer perception based on hand-crafted features [18,20], the proposed method effectively learns the comparable representations for heterogeneous data in an end-to-end way” [Hua page 3 Introduction]).
Regarding claim 17, it is a system/apparatus claim that largely corresponds to the method of claim 4, which is already taught by the combination of Sohn, Mishchuk, and Hua as detailed above. Consequently, claim 17 is rejected for the same reasons.
Claims 5 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Sohn and Mishchuk, as applied to claims 1 and 14 above, further in view of Rodriguez-Serrano (“Data-Driven Detection of Prominent Objects”, available IEEE 12 Sep 2016).
Regarding claim 5, the combination of Sohn and Mishchuk teaches the limitations of parent claim 1.
However, the combination does not expressly teach wherein each query, in the set of queries, is an image capturing an object, and wherein each electronic resource, in the set of electronic resources, represents one or more corresponding bounding boxes.
In the same field of endeavor, Rodriguez-Serrano teaches a deep metric learning framework applied to visual recognition tasks (“This article deals with the detection of prominent objects in images… We refer to this approach as data-driven detection (DDD). Our key novelty is to design or learn image similarities that explicitly optimize some aspect of the transfer unlike previous work which uses generic representations and unsupervised similarities. In a first variant, we explicitly learn to transfer, by adapting a metric learning approach to work with image and bounding box pairs” [Rodriguez-Serrano Abstract]) wherein each query, in the set of queries, is an image capturing an object, and wherein each electronic resource, in the set of electronic resources, represents one or more corresponding bounding boxes (“Our goal is to infer the bounding box of the prominent object from the global feature vector of the image, using a training set with annotated bounding boxes…We denote by {xi, Ri} a training set of feature-annotation pairs, where xi ∈ RD is the feature vector of the ith image and Ri ∈ R4 the ground-truth rectangle indicating the extent of its object of interest. We encode Ri by the x, y coordinates of its top-left and bottom-right corners, relatively to the image size…We assume a similarity function between features k : RD X RD –> R. For a query feature q, kn = k(q,xn) denotes its similarity to the nth element of the training set…We seek a function that predicts the rectangle R, using the L top-ranked samples” [Rodriguez-Serrano page 1972 Data-driven Detection – Baseline])
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein each query, in the set of queries, is an image capturing an object, and wherein each electronic resource, in the set of electronic resources, represents one or more corresponding bounding boxes as taught by Rodriguez-Serrano into the combination because Sohn and Rodriguez-Serrano are both directed towards deep metric learning frameworks applied to visual recognition tasks. Incorporating the teachings of Rodriguez-Serrano would further expand the applicability of the disclosed N-pair loss procedure of Sohn to a metric learning environment wherein the label space (e.g., bounding boxes) is continuous and multi-dimensional, rather than categorical (“All these works use categorical labels at the sample level (a sample belongs to a discrete class) or at the level of pairs (a pair of samples is labeled as positive or negative). We are not aware of previous works applying metric learning on object location labels. Here the label space is continuous and multi-dimensional, not categorical” [Rodriguez-Serrano page 1971 Metric Learning]).
Regarding claim 18, it is a system/apparatus claim that largely corresponds to the method of claim 4, which is already taught by the combination of Sohn, Mishchuk, and Rodriguez-Serrano as detailed above. Consequently, claim 18 is rejected for the same reasons.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Sohn (“Improved Deep Metric Learning with Multi-class N-pair Loss Objective”, published 2016, cited in IDS filed 09/09/2022) in view of Rodriguez-Serrano et al. (“Data-Driven Detection of Prominent Objects”, available IEEE 12 Sep 2016), hereinafter Rodriguez-Serrano.
Regarding claim 10, Sohn teaches A method implemented by one or more processors, the method comprising:
receiving an image; (“We evaluate deep metric learning algorithms on fine-grained object recognition and verification tasks. Specifically, we consider car and flower recognition problems on the following database: Car-333 [29] dataset is composed of 164, 863 images of cars from 333 model categories collected from the internet. Following the experimental protocol [29], we split the dataset into 157,023 images for training and 7,840 for testing. • Flower-610 dataset contains 61,771 images of flowers from 610 different flower species and among all collected, 58,721 images are used for training and 3,050 for testing” [Sohn page 5 Fine-grained visual object recognition and verification]; Flower/car images are received as queries to trained model for object recognition/verification tasks)
generating an image vector by processing the image using an input model; (“We evaluate both recognition and verification accuracy. Specifically, we consider verification setting where there are different number of negative examples from different classes, and determine as success only when the positive example is closer to the query example than any other negative example. Since the recognition task is involved, we also evaluate the performance of deep networks trained with softmax loss. The summary results are given in Table 1” [Sohn page 5 Fine-grained visual object recognition and verification]; As explained above ([Sohn page 2 Preliminary: Distance Metric Learning]), input query x is passed through kernel f() to generate embedding vector f(x) for comparison to existing positive/negative examples)
comparing the image vector to a plurality of pre-stored candidate vectors, wherein each pre-stored candidate vector, in the plurality of pre- stored candidate vectors, is previously generated by processing a corresponding candidate using a resource model; ([Sohn page 5 Fine-grained visual object recognition and verification] as detailed above; The input query vector is compared to existing positive/negative examples (i.e., pre-stored electronic resource vectors))
selecting a pre-stored candidate vector based on the comparing; ([Sohn page 5 Fine-grained visual object recognition and verification] as detailed above; The vector closest to the query vector is the “selected” one) and
determining the candidate based on the selected pre-stored candidate vector; ([Sohn page 5 Fine-grained visual object recognition and verification] as detailed above; As explained above ([Sohn page 2 Preliminary: Distance Metric Learning]), each vector embedding corresponds to the underlying example and its respective class) and
causing a computing device to perform one or more actions based on the determined candidate for the object ([Sohn page 5 Fine-grained visual object recognition and verification] as detailed above; In a verification setting, a success determined when the positive example is found to be closest to the query).
However, Sohn does not expressly teach for an image capturing an object, determining a bounding box for the object captured in the image via comparing [an] image vector to a plurality of pre-stored candidate bounding box vectors, wherein each pre-stored candidate bounding box vector, in the plurality of pre-stored candidate bounding box vectors, is previously generated by processing a corresponding candidate box using a resource model and selecting a bounding box vector based on the comparing, and determining the bounding box for the object based on the selected pre-stored candidate bounding box vector.
In the same field of endeavor, Rodriguez-Serrano teaches a deep metric learning framework applied to visual recognition tasks (“This article deals with the detection of prominent objects in images… We refer to this approach as data-driven detection (DDD). Our key novelty is to design or learn image similarities that explicitly optimize some aspect of the transfer unlike previous work which uses generic representations and unsupervised similarities. In a first variant, we explicitly learn to transfer, by adapting a metric learning approach to work with image and bounding box pairs” [Rodriguez-Serrano Abstract]) that for an image capturing an object, determin[es] a bounding box for the object captured in the image via comparing [an] image vector to a plurality of pre-stored candidate bounding box vectors, (“Our goal is to infer the bounding box of the prominent object from the global feature vector of the image, using a training set with annotated bounding boxes…We denote by {xi, Ri} a training set of feature-annotation pairs, where xi ∈ RD is the feature vector of the ith image and Ri ∈ R4 the ground-truth rectangle indicating the extent of its object of interest. We encode Ri by the x, y coordinates of its top-left and bottom-right corners, relatively to the image size…We assume a similarity function between features k : RD X RD –> R. For a query feature q, kn = k(q,xn) denotes its similarity to the nth element of the training set…We seek a function that predicts the rectangle R, using the L top-ranked samples. A straightforward choice is a non-parametric regression:
PNG
media_image10.png
108
278
media_image10.png
Greyscale
which expresses the predicted rectangle as a weighted combination of the ground-truth rectangles of the L best ranked samples. We choose
PNG
media_image11.png
32
193
media_image11.png
Greyscale
” [Rodriguez-Serrano page 1972 Data-driven Detection – Baseline]; “We assume that a similarity function over annotations ψ : R4 X R4 –> R is defined…Our goal is to learn a similarity function k : RD X RD –> R which ranks images as similarly as possible to the ranking induced by ψ. Intuitively, this means that image pairs with similar annotations are forced to have similar representations according to the learnt metric” [Rodriguez-Serrano page 1972 Data-driven Detection – Definitions and Similarity Function]; The disclosed procedure determines, for an ith image capturing an object of interest, a predicted rectangle (i.e., bounding box) R(q) for the image query based on its associated feature vector xi (i.e., image vector) – the feature vector xi is compared, via similarity function k, to identify and select L similar image-annotation pairs (annotations being rectangle vectors Ri ∈ R4), and further compute R(q) based on rectangle vectors of the selected L samples. The similarity function k is expressly trained via loss function (see also
PNG
media_image12.png
93
542
media_image12.png
Greyscale
[Rodriguez-Serrano page 4 Loss Function]) to compare feature vector xi to other feature vectors as closely as possible to the theoretical similarity function ψ comparing over rectangle vectors, and thereby executes indirect comparison of feature vector xi to rectangle vectors Ri which will produce the best possible matches), wherein each pre-stored candidate bounding box vector, in the plurality of pre-stored candidate bounding box vectors, is previously generated by processing a corresponding candidate box using a resource model; (“Ri ∈ R4 the ground-truth rectangle indicating the extent of its object of interest. We encode Ri by the x, y coordinates of its top-left and bottom-right corners, relatively to the image size” [Rodriguez-Serrano page 1972 Data-driven Detection – Baseline]) and
select[s] a bounding box vector based on the comparing, and determin[es] the bounding box for the object based on the selected pre-stored candidate bounding box vector (We seek a function that predicts the rectangle R, using the L top-ranked samples. A straightforward choice is a non-parametric regression:
PNG
media_image10.png
108
278
media_image10.png
Greyscale
which expresses the predicted rectangle as a weighted combination of the ground-truth rectangles of the L best ranked samples. We choose
PNG
media_image11.png
32
193
media_image11.png
Greyscale
” [Rodriguez-Serrano page 1972 Data-driven Detection – Baseline]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated for an image capturing an object, determining a bounding box for the object captured in the image via comparing [an] image vector to a plurality of pre-stored candidate bounding box vectors, wherein each pre-stored candidate bounding box vector, in the plurality of pre-stored candidate bounding box vectors, is previously generated by processing a corresponding candidate box using a resource model and selecting a bounding box vector based on the comparing, and determining the bounding box for the object based on the selected pre-stored candidate bounding box vector as taught by Rodriguez-Serrano into Sohn because they are both directed towards deep metric learning framework applied to visual recognition tasks. Incorporating the teachings of Rodriguez-Serrano would further expand the applicability of the disclosed N-pair loss procedure of Sohn to a metric learning environment wherein the label space (e.g., bounding boxes) is continuous and multi-dimensional, rather than categorical (“All these works use categorical labels at the sample level (a sample belongs to a discrete class) or at the level of pairs (a pair of samples is labeled as positive or negative). We are not aware of previous works applying metric learning on object location labels. Here the label space is continuous and multi-dimensional, not categorical” [Rodriguez-Serrano page 1971 Metric Learning]).
Response to Amendment and Arguments
The amendment filed 03/30/2026 has been entered.
Applicant’s amendment to the claims with respect to resolving claim objections has been considered, and the objections are consequently withdrawn.
Applicant’s amendment to the claims with respect to resolving subject matter eligibility rejections under 35 U.S.C. 101 has been considered, but does not overcome the standing rejection. Applicant is directed towards the grounds of rejection under 35 U.S.C. 101 with respect to claims 1-10 and 14-19 set forth above.
The remarks filed 03/30/2026 have been fully considered.
Applicant’s remarks with respect to the specification objection have been considered, and the objection is consequently withdrawn.
Applicant's remarks traversing the obviousness rejections under 35 U.S.C. 103 set forth in the office action mailed 12/29/2025, in view of claims 1-10 and 14-19 as amended, have been considered.
The examiner has determined a response necessary for the portion of the remarks (Remarks page 10) discussing independent claim 10, wherein the references applied in the prior rejection of record are still being relied upon to teach or suggest the subject matter being challenged in applicant’s argument.
The remaining remarks (Remarks page 9), while having been considered, are moot because a new grounds of rejection has been set forth above to teach independent claims 1 and 14 as amended.
Applicant alleges that the prior office action mischaracterizes how Rodriguez-Serrano calculates its bounding boxes. Given that the cited portions of Rodriguez-Serrano describe comparing an image vector to other image vectors to find similar images, and then mathematically averaging the coordinates of the bounding boxes associated with those similar images, applicant alleges that the reference thereby does not teach or suggest processing a candidate bounding box through a model to create a "bounding box vector," nor do they teach or suggest "comparing the image vector to a plurality of pre-stored candidate bounding box vectors, wherein each pre-stored candidate bounding box vector ... is previously generated by processing a corresponding candidate bounding box using a resource model."
The examiner respectfully disagrees. While applicant’s characterization of Rodriguez-Serrano is acknowledged and not necessarily refuted, the examiner notes, as further elaborated upon in the rejection above (see section Claim Rejections - 35 USC § 103 [pages 38-40]), that the reference utilizes a similarity function to compare feature vectors, wherein the similarity function is expressly trained to compare feature vector xi to other feature vectors as closely as possible to the theoretical similarity function ψ comparing over rectangle vectors. Under a broadest reasonable interpretation, it thereby executes indirect comparison of feature vector xi to rectangle vectors Ri to determine which will produce the best possible matches.
It is further noted by the examiner that a narrower recitation of the claimed “comparing the image vector to a plurality of pre-stored candidate bounding box vectors”, wherein a direct mathematical comparison between a high-dimensional vector embedding capturing an entire image and bounding box vectors representing mere spatial coordinate boundaries (represented in R4, e.g., [x, y, w, h]) is required as the step to select and determine an appropriate bounding box, calls into question whether the step would be appropriately enabled by the disclosure. It would not be clear to one of ordinary skill in the art how high-dimensional raw image vector embeddings could be meaningfully compared to a set of bounding box coordinates in a manner that determines an appropriate bounding box for a particular object in the image, and the specification provides no guidance for how to achieve such a determination beyond restating claim language that merely describes the intended result rather than provides actual explanation.
Applicant has not presented further arguments with respect to the dependent claims. As such, amended claims 1-10 and 14-19 stand rejected under 35 U.S.C. 103.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Strope et al. (Pub. No. US 20180240013 A1, “Cooperatively Training and/or Using Separate Input and Subsequent Content Neural Networks for Information Retrieval”, published 08/23/2018) discloses a method of of training a relevance model that identifies a batch of training instances, wherein each of the training instances may be a positive training instance with a corresponding input text segment, a corresponding subsequent content text segment, and one or more corresponding subsequent content contexts, generates input vectors from the training instances, and determines relevance values based on dot products of the input vectors and subsequent content vectors. The system may use additional negative training instance relevance values based on dot products of input vectors and subsequent content vectors that are based on different training instances when calculating errors during training.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VIJAY M BALAKRISHNAN whose telephone number is (571) 272-0455. The examiner can normally be reached 10am-5pm EST Mon-Thurs.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JENNIFER WELCH can be reached on (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/V.M.B./
Examiner, Art Unit 2143
/JENNIFER N WELCH/Supervisory Patent Examiner, Art Unit 2143
Prosecution Insights