DETAILED ACTION
This office action is in response to amendments filed on 05/15/2026.
Claims 1, 8-9, and 14 have been amended. Claims 1-6, 8-19, and 21-22 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 05/15/2026 has been entered.
Response to Arguments
35 U.S.C. 112(b) rejection:
In light of applicant’s amendments to the claims (pg. 2-8), the rejection under 35 USC § 112(b) has been withdrawn.
Prior Art Rejections:
Applicant’s arguments regarding the prior art rejections (pg. 10-14) have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Applicant argues that the cited references fail to teach or suggest the amended independent claim feature which recites “…wherein the search query tower and the related content tower are trained, using contrastive loss based on pairwise co-click signals associated with a plurality of search queries, each pairwise co-click signal indicating a positive pair between an instance of content that is responsive to a respective search query of the plurality of search queries and an instance of related content with which a user has interacted…”
Applicant specifically argues that cited references Yu and Zamani each teach training using pairwise click signals between queries and items with which the user has interacted (i.e. query-item pairs), while the claim requires training using pairwise co-click signals between items responsive to the query and related items with which the user has interacted (i.e. item-item pairs). Examiner notes that the Zhang reference, which discloses co-click signals between items, has been brought in to teach this limitation.
The prior art rejections have been updated to include the amended limitations and to clarify the reasoning given for the limitations that were not amended.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6, 14-17, 19, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over
Yu et al. (hereinafter Yu), “A Dual Augmented Two-tower Model for Online Large-scale Recommendation” in view of
Yao et al. (hereinafter Yao), “Self-supervised Learning for Large-scale Item Recommendations”, and
Zhang et al. (hereinafter Zhang), “MIRA: Leveraging Multi-Intention Co-click Information in Web-scale Document Retrieval using Deep Neural Networks”.
Regarding Claim 1,
Yu teaches A system comprising: at least one processor; and memory storing instructions that, when executed by the at least one processor, causes the system to perform a set of operations, (Examiner notes that this limitation is interpreted as a general-purpose computing environment. Yu, pg. 3, section 3.2: “We implemented these models by distributed TensorFlow…” The use of TensorFlow necessitates implementation of the described system in a computing environment.)
Yu teaches the set of operations comprising:
obtaining a search query; (Pg. 2, section 2.1: “Our objective is to efficiently select possibly thousands of candidate items from the entire item corpus given a certain query.”)
based on the search query, obtaining one or more candidate content items; (Pg. 2, section 2.1: “We consider a recommendation system with a query set
{
u
i
}
i
=
1
N
and an item set
{
v
j
}
j
=
1
M
, where 𝑁 is the number of users and 𝑀 is the number of items.” Pg. 3, section 3.1: “All models were evaluated on two offline large-scale datasets: a large dataset sampled from the daily logs of online systems on Meituan and a dataset from Amazon [3].” Item set
{
v
j
}
j
=
1
M
comprises candidate content items. During evaluation, a search query
u
is associated with a dataset, which includes a set of items (see pg. 3, table 1) (i.e. the set of candidate items is based on the search query).)
generating a set of features for the search query thereby generating a set of search query features; (Pg. 2, section 2.1-2.2: “We consider a recommendation system with a query set
{
u
i
}
i
=
1
N
and an item set
{
v
j
}
j
=
1
M
…each feature f𝑖 ∈ R (e.g., an item ID) in u𝑖 and v𝑗 goes through an embedding layer…For a certain query and candidate item, we create two corresponding augmented vectors a𝑢 and a𝑣 by their IDs, and concatenate them with feature embedding vectors to obtain the augmented input vectors z𝑢 , z𝑣 of the two towers.” Augmented input vector z𝑢 is the generated set of features for a query
u
. Figure 1 (pg. 2) shows the architecture of the proposed model, where the left tower is associated with the query, and the ‘embedding’ represents the generated set of search query features.)
generating a set of features for each candidate content item thereby generating candidate content item features for each candidate content item; (See the cited portion of sections 2.1-2.2 above. Augmented input vector z𝑣 is the generated set of features for an item
v
(i.e. candidate content item). Figure 1 (pg. 2) shows the architecture of the proposed model, where the right tower is associated with the content item, and the ‘embedding’ represents the generated set of candidate content item features.)
generating, by inputting the search query features into a pre-trained search query tower, a first feature vector corresponding to the search query; (Pg. 2, section 2.2: “Next, we feed z𝑢 and z𝑣 into the two towers… to get augmented representations of query p𝑢 and item p𝑣… p𝑢 and p𝑣, the output vectors of the L2 normalization layer, represent the query embedding and item embedding, respectively.” The augmented input vector z𝑢 (i.e. the search query features) is fed into the query tower of the two-tower model to obtain output vector p𝑢 (i.e. a first feature vector corresponding to the search query).)
for each candidate content item, generating, by inputting the candidate content item features into a pre-trained related content tower, a respective second feature vector for each candidate content item; (See the cited portion of section 2.2 above. Each augmented input vector z𝑣 (i.e. the candidate content item features) is fed into the item tower of the two-tower model to obtain output vector p𝑣 (i.e. a respective second feature vector for each candidate content item).)
determining, for each candidate content item, an inner product between the first feature vector and the respective second feature vector for the candidate content item; and (Pg. 3, section 2.2: “Finally, the output of the model is the inner product of the query embedding and item embedding: 𝑠 (u, v) = ⟨p𝑢, p𝑣⟩ where 𝑠 (u, v) denotes the score provided by our retrieval model.”)
generating, based on the outputs of the trained towers and the determined inner products, (Pg. 2, section 2.1: “Our objective is to efficiently select possibly thousands of candidate items from the entire item corpus given a certain query.” Candidate items (i.e. a set of recommended content items) are selected for the search query based on the model output (i.e. based on the outputs of the trained towers and the determined inner products).)
wherein the search query tower and the related content tower are trained, [using contrastive loss] based on pairwise [co]-click signals associated with a plurality of search queries, […], such that the inner product indicates a likelihood that a particular candidate content item is relevant to the search query. (Pg. 1, section 1: “In real-world applications, the input of one tower has a positive interaction with the input of the other for each click.” Pg. 3, section 2.3: “Model Training… Specifically, for the query in each positive query-item pair (label = 1), we randomly sample 𝑆 items from the item corpus to create 𝑆 negative query-item pairs (label = 0) with this query, and add these 𝑆 + 1 pairs to the training dataset. The cross-entropy loss for these pairs is as follows…” Pg. 3, section 2.2: “Finally, the output of the model is the inner product of the query embedding and item embedding: 𝑠 (u, v) = ⟨p𝑢, p𝑣⟩ where 𝑠 (u, v) denotes the score provided by our retrieval model.” A positive query-item pair is a click signal, and these query-item pairs make up the training data used to train the two-tower model. The inner product between the query and item embeddings represents the retrieval model’s relevance score.)
Yu does not appear to explicitly disclose using contrastive loss.
However, Yao teaches using contrastive loss (Pg. 3, section 3.1: “Inspired by the SimCLR framework [5] for visual representation learning, we adopt similar contrastive learning algorithms for learning representations of categorical features…use contrastive loss function to encourage the representations learned for the same training example to be similar.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Yu and Yao. Yu teaches a two-tower model for query-based online content recommendation using cross-entropy loss. Yao teaches a two-tower model for content recommendation using contrastive learning. One of ordinary skill would have motivation to combine Yu and Yao in order to “leverage self-supervised learning based auxiliary tasks to improve item representations, especially with long-tail distributions and sparse data” (Yao, pg. 2, section 1). According to Yao, contrastive learning can be used for exactly the type of two-tower query-item recommender model presented by Yu: “Contrastive loss was also applied in training two-tower DNNs… to make positive item agree with its corresponding queries” (Yao, pg. 3, section 3.1).
Yu and Yao do not appear to explicitly disclose pairwise co-click signals.
However, Zhang teaches pairwise co-click signals associated with a plurality of search queries, each pairwise co-click signal indicating a positive pair between an instance of content that is responsive to a respective search query of the plurality of search queries and an instance of related content with which a user has interacted, (Pg. 1, Abstract: “[W]e aim to leverage the additional information for each document from its co-click neighbours to help document retrieval.” Pg. 2, section 1: “[I]t provides us the feasibility to get potential information for a specific document understanding from other similar documents through co-click relationship, e.g. two documents clicked by the same query in history indicates their similarity in some respects.” The query-based document retrieval (i.e. content recommendation) model is trained based on co-click relationships, which indicate two documents that are clicked together in response to a search query (i.e. a positive pair between an instance of content that is responsive to the query and an instance of related content with which a user has interacted).)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Yu, Yao, and Zhang. Yu teaches a two-tower model for query-based online content recommendation using cross-entropy loss. Yao teaches a two-tower model for content recommendation using contrastive learning. Zhang teaches a two-tower model for document retrieval which leverages co-click relationships between content items. One of ordinary skill would have motivation to combine Yu, Yao, and Zhang because “It has been proved that the click-through logs over query-document pairs in real search engine provide rich information for multiple tasks in information retrieval… co-click relations mainly help improve web search quality from two aspects: key concept enhancing and query term complementary” (Zhang, pg. 1, Abstract).
Regarding Claim 2, Yu, Yao, and Zhang teach The system of claim 1, as shown above.
Yu also teaches wherein training the search query tower and related content tower further comprises utilizing a cascaded multilayer perceptron model comprised of multiple layers. (Pg. 2, section 2.2: “…the two towers, which are composed of fully connected layers…” Figure 1 (pg. 2) shows the architecture of the proposed model, where each of the query tower and item tower include multiple cascaded layers labeled ‘Linear + ReLU’ representing fully connected neural network layers (i.e. multilayer perceptron layers).)
Regarding Claim 3, Yu, Yao, and Zhang teach The system of claim 2, as shown above.
Yu also teaches wherein the cascaded multilayer perceptron model layers comprises one or more of an expand layer or a bottleneck layer, and between the multilayer perceptron layers data scaling is performed. (Pg. 3, section 3.2: “the number of FC layers in each tower was fixed to 3, with dimensions 256, 128 and 32, respectively.” Reducing dimensions between fully connected layers (e.g. 256 [Wingdings font/0xE0] 128) equates to a bottleneck layer. Pg. 2, section 2.2: “the two towers, which are composed of fully connected layers with the ReLU activation function…” ReLU activation, which occurs between the layers of the model, is a form of data scaling.)
Regarding Claim 4, Yu, Yao, and Zhang teach The system of claim 3, as shown above.
Yu also teaches wherein data scaling comprises one or more of data standardization or data normalization including batch normalization, activation, and dropout. (Pg. 2, section 2.2: “the two towers, which are composed of fully connected layers with the ReLU activation function…” ReLU activation, which occurs between the layers of the model, is a form of data scaling.)
Regarding Claim 6, Yu, Yao, and Zhang teach The system of claim 1, as shown above.
Yu also teaches wherein generating the set of features further comprises pre-processing the search query to generate a set of low-dimensional features using at least one of feature scaling, centering, or dimensionality reduction. (Pg. 2, section 2.2.1: “each feature f𝑖 ∈ R (e.g., an item ID) in u𝑖 and v𝑗 goes through an embedding layer and is mapped to a low-dimensional dense vector e𝑖 ∈ R𝐾, where 𝐾 is the embedding dimension. Specifically, we define an embedding matrix E ∈ R𝐾×𝐷 where E is to be learned and 𝐷 is the number of unique features, and the embedding vector e𝑖 is the 𝑖th column of the embedding matrix E.” This is a form of dimensionality reduction.)
Claims 14-17 and 19 are method claims, containing substantially the same elements as system claims 1-4 and 6, respectively. Yu, Yao, and Zhang teach the elements of claims 1-4 and 6, as shown above.
Regarding Claim 22, Yu, Yao, and Zhang teach The method of claim 14, as shown above.
Yao also teaches wherein the search query tower and the related content tower are trained using supervised contrastive loss or self-supervised representation learning employing the contrastive loss. (Pg. 3, section 3.1: “Inspired by the SimCLR framework [5] for visual representation learning, we adopt similar contrastive learning algorithms for learning representations of categorical features…use contrastive loss function to encourage the representations learned for the same training example to be similar.”)
Claims 5 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Yu in view of Yao and Zhang, and further in view of
Xiao et al. (hereinafter Xiao), “UPRec: User-Aware Pre-training for Recommender Systems”.
Regarding Claim 5, Yu, Yao, and Zhang teach The system of claim 2, as shown above.
Yu, Yao, and Zhang do not appear to explicitly disclose wherein a skip connection is utilized between an initial input layer and another multilayer perceptron layer of the cascaded multilayer perceptron model.
However, Xiao teaches wherein a skip connection is utilized between an initial input layer and another multilayer perceptron layer of the cascaded multilayer perceptron model. (Pg. 4, section 3.2.3: “In addition to multi-head self-attention layer, each transformer layer also contains a fully connected feed-forward layer… Then a residual connection [48] and layer normalization operation [48] are employed…” Figure 2 (pg. 4) shows the architecture of the transformer layer of the model, where the arrow connecting the input sequence directly to the ‘Add & Layer Normalization’ block represents a residual connection (skip connection) between the input layer and another fully connected (MLP) layer.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Yu, Yao, Zhang, and Xiao. Yu teaches a two-tower model for query-based online content recommendation using cross-entropy loss. Yao teaches a two-tower model for content recommendation using contrastive learning. Zhang teaches a two-tower model for document retrieval which leverages co-click relationships between content items. Xiao teaches enhancements to content recommendation systems using user-aware pretraining. One of ordinary skill would have motivation to combine Yu, Yao, Zhang, and Xiao because residual connections “stabilize and accelerate the network training process” (Xiao, pg. 4, section 3.2.3).
Claim 18 is a method claim, containing substantially the same elements as system claim 5. Yu, Yao, Zhang, and Xiao teach the elements of claim 5, as shown above.
Claims 8-9, 12-13, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over
Zamani et al. (hereinafter Zamani), “Joint Modeling and Optimization of Search and Recommendation” in view of Yu, Yao, and Zhang.
Regarding Claim 8,
Zamani teaches A method comprising:
receiving a search request including a search query for content; (Pg. 2, section 2.2: “JSR [joint search and recommendation] is a general framework for jointly modeling search and recommendation and consists of two major components: a retrieval component and a recommendation component. The retrieval component computes the retrieval score for an item i given a query q and a query context cq.” Query q is a search query, and item i is content.)
generating, using a related content model including a two-tower cascaded multilayer perceptron model, a set of recommended content associated with both the search query and an instance of content that is responsive to the search query, (Pg. 3, section 2.3: “we simply use fully-connected feed-forward networks to implement the components of the JSR [joint search and recommendation] framework.” Fully-connected feed-forward networks are multilayer perceptron models. Figure 2 (pg. 3) shows the architecture of the framework, where these networks are arranged in a cascading structure, with a separate retrieval model and recommendation model which share a loss function (i.e. a two-tower model). Figure 1 (pg. 1) shows an overview of the system, which includes inputting a ‘search query’ and outputting both a ‘Recommendation List’ and a ‘Search Result List’.)
providing, in response to the search request, the generated set of recommended content in association with the instance of content that is responsive to the search query. (Figure 1 (pg. 1) shows an overview of the system, including inputting a ‘search query’ and outputting both a ‘Recommendation List’ and a ‘Search Result List’ which are then provided for model evaluation. Pg. 5, section 3.3: “To evaluate the retrieval model, we use mean average precision (MAP) of the top 100 retrieved items and normalized discounted cumulative gain (NDCG) of the top 10 retrieved items (NDCG@10). To evaluate the recommendation performance, we use NDCG, hit ratio (Hit), and recall. The cut-off for all recommendation metrics is 10. Hit ratio is defined as the ratio of users that are recommended at least one relevant item.” The returned item lists are compared to actual relevant items for the search query/user to which they are responsive.)
Zamani does not appear to explicitly disclose the related content model being trained using contrastive loss based on pairwise co-click signals associated with a plurality of search queries, each pairwise co-click signal indicating a positive pair between content responsive to a respective search query of the plurality of search queries and related content with which a user has interacted;
However, Yu teaches the related content model being trained [using contrastive loss] based on pairwise [co]-click signals associated with a plurality of search queries (Pg. 1, section 1: “In real-world applications, the input of one tower has a positive interaction with the input of the other for each click.” Pg. 3, section 2.3: “Model Training… Specifically, for the query in each positive query-item pair (label = 1), we randomly sample 𝑆 items from the item corpus to create 𝑆 negative query-item pairs (label = 0) with this query, and add these 𝑆 + 1 pairs to the training dataset. The cross-entropy loss for these pairs is as follows…” A positive query-item pair is a click signal, and these query-item pairs make up the training data used to train the two-tower recommender model.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zamani and Yu. Zamani teaches a two-tower joint framework for search and recommendation based on a search query and an item set. Yu teaches a two-tower model for query-based online content recommendation trained using co-click signals. One of ordinary skill would have motivation to combine Zamani and Yu in order to “provide[] deeper insights into the information interaction of two-tower models in the retrieval task” (Yu, pg. 1-2, section 1).
Zamani and Yu do not appear to explicitly disclose using contrastive loss.
However, Yao teaches using contrastive loss (Pg. 3, section 3.1: “Inspired by the SimCLR framework [5] for visual representation learning, we adopt similar contrastive learning algorithms for learning representations of categorical features…use contrastive loss function to encourage the representations learned for the same training example to be similar.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zamani, Yu, and Yao. Zamani teaches a two-tower joint framework for search and recommendation based on a search query and an item set. Yu teaches a two-tower model for query-based online content recommendation using co-click signals and cross-entropy loss. Yao teaches a two-tower model for content recommendation using contrastive learning. One of ordinary skill would have motivation to combine Zamani, Yu, and Yao in order to “leverage self-supervised learning based auxiliary tasks to improve item representations, especially with long-tail distributions and sparse data” (Yao, pg. 2, section 1). According to Yao, contrastive learning can be used for exactly the type of two-tower query-item recommender model presented by Zamani and Yu: “Contrastive loss was also applied in training two-tower DNNs… to make positive item agree with its corresponding queries” (Yao, pg. 3, section 3.1).
Zamani, Yu, and Yao do not appear to explicitly disclose pairwise co-click signals.
However, Zhang teaches pairwise co-click signals associated with a plurality of search queries, each pairwise co-click signal indicating a positive pair between content responsive to a respective search query of the plurality of search queries and related content with which a user has interacted; (Pg. 1, Abstract: “[W]e aim to leverage the additional information for each document from its co-click neighbours to help document retrieval.” Pg. 2, section 1: “[I]t provides us the feasibility to get potential information for a specific document understanding from other similar documents through co-click relationship, e.g. two documents clicked by the same query in history indicates their similarity in some respects.” Pg. 8, section 5.4: “In training process, the click query-document pairs are used as positive data with in-batch negative sampling.” The query-based document retrieval (i.e. content recommendation) model is trained based on co-click relationships, which indicate two documents that are clicked together in response to a search query (i.e. a positive pair between an instance of content that is responsive to the query and an instance of related content with which a user has interacted).)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zamani, Yu, Yao, and Zhang. Zamani teaches a two-tower joint framework for search and recommendation based on a search query and an item set. Yu teaches a two-tower model for query-based online content recommendation using co-click signals and cross-entropy loss. Yao teaches a two-tower model for content recommendation using contrastive learning. Zhang teaches a two-tower model for document retrieval which leverages co-click relationships between content items. One of ordinary skill would have motivation to combine Zamani, Yu, Yao, and Zhang because “It has been proved that the click-through logs over query-document pairs in real search engine provide rich information for multiple tasks in information retrieval… co-click relations mainly help improve web search quality from two aspects: key concept enhancing and query term complementary” (Zhang, pg. 1, Abstract).
Regarding Claim 9, Zamani, Yu, Yao, and Zhang teach The method of claim 8, as shown above.
Yu also teaches wherein: the instance of content is a first instance of content; and
the related content model is trained using the [co]-click signals associated with the plurality of search queries, (Pg. 1, section 1: “In real-world applications, the input of one tower has a positive interaction with the input of the other for each click.” Pg. 3, section 2.3: “Model Training… Specifically, for the query in each positive query-item pair (label = 1), we randomly sample 𝑆 items from the item corpus to create 𝑆 negative query-item pairs (label = 0) with this query, and add these 𝑆 + 1 pairs to the training dataset. The cross-entropy loss for these pairs is as follows…” A query-item pair is a click signal, and these query-item pairs make up the training data used to train the two-tower model.)
the [co]-click signals including:
the positive pair between [the content responsive to] the respective search query and the related content with which the user has interacted; and (See the portions of sections 1 and 2.3 cited above. The click signals include a positive query-item pair (i.e. a positive pair between the search query and the related content with which the used has interacted).)
a negative pair between [the content responsive to] the respective search query and an instance of related content with which the user has not interacted. (See the portions of sections 1 and 2.3 cited above. The click signals include a negative query-item pair (i.e. a negative pair between the search query and an instance of related content with which the user has not interacted).)
Zhang teaches co-click signals including pairs between the content responsive to the respective search query and the related content with which the user has or has not interacted. (Pg. 1, Abstract: “[W]e aim to leverage the additional information for each document from its co-click neighbours to help document retrieval.” Pg. 2, section 1: “[I]t provides us the feasibility to get potential information for a specific document understanding from other similar documents through co-click relationship, e.g. two documents clicked by the same query in history indicates their similarity in some respects.” The query-based document retrieval (i.e. content recommendation) model is trained based on co-click relationships, which indicate whether two documents are clicked together in response to a search query (i.e. a pair between an instance of content that is responsive to the query and an instance of related content with which a user has or has not interacted).)
Regarding Claim 12, Zamani, Yu, Yao, and Zhang teach The method of claim 8, as shown above.
Zamani also teaches wherein the search request further includes an indication of the instance of content that is responsive to the search query. (Pg. 4, section 3.1: “The Amazon product data does not contain search queries… Van Gysel et al. [23] proposed to automatically generate queries based on the product categories. To be exact, for each item in a category c, a query q is generated based on the terms in the category hierarchy of c. Then, all the items within that category are marked as relevant for the query q.” This creates labeled training data, where items in category c are indicated as being responsive content to query q.)
Regarding Claim 13, Zamani, Yu, Yao, and Zhang teach The method of claim 8, as shown above.
Zamani also teaches further comprising identifying the instance of content that is responsive to the search query. (Figure 1 (pg. 1) shows an overview of the proposed system, which includes outputting a ‘Search Result List’ (i.e. content responsive to the search query) which is distinct from the ‘Recommendation List’ (i.e. clearly identified).)
Regarding Claim 21, Zamani, Yu, Yao, and Zhang teach The method of claim 8, as shown above.
Yao also teaches wherein the related content model is trained using supervised contrastive loss or self-supervised representation learning employing the contrastive loss. (Pg. 3, section 3.1: “Inspired by the SimCLR framework [5] for visual representation learning, we adopt similar contrastive learning algorithms for learning representations of categorical features…use contrastive loss function to encourage the representations learned for the same training example to be similar.”)
Claims 10-11 are rejected under 35 U.S.C. 103 as being unpatentable over Zamani in view of Yu, Yao, and Zhang and further in view of
Wang et al. (hereinafter Wang), “Learning Two-Branch Neural Networks for Image-Text Matching Tasks”.
Regarding Claim 10, Zamani, Yu, Yao, and Zhang teach The method of claim 8, as shown above.
Zamani, Yu, Yao, and Zhang do not appear to explicitly disclose wherein a first tower of the two-tower cascaded multilayer perceptron model is associated with a first content type and a second tower of the two-tower cascaded multilayer perceptron model is associated with a second content type.
However, Wang teaches wherein a first tower of the two-tower cascaded multilayer perceptron model is associated with a first content type and a second tower of the two-tower cascaded multilayer perceptron model is associated with a second content type. (Pg. 1-2, section 1: “As suggested by the above discussion, the network architecture for these tasks should consist of two branches that take in image and text features respectively, pass them through one or more layers of transformations, fuse them, and eventually output a learned similarity score.” A branch is equivalent to a tower. As can be seen in figure 1 (pg. 2), the right branch takes input Y representing a text query (first content type), the left branch takes input X representing an image (second content type), and each branch consists of fully connected layers (i.e. MLPs).)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zamani, Yu, Yao, Zhang, and Wang. Zamani teaches a two-tower joint framework for search and recommendation based on a search query and an item set. Yu teaches a two-tower model for query-based online content recommendation using co-click signals and cross-entropy loss. Yao teaches a two-tower model for content recommendation using contrastive learning. Zhang teaches a two-tower model for document retrieval which leverages co-click relationships between content items. Wang teaches a two-branch multimodal retrieval model for sentence-to-image and image-to-sentence search. One of ordinary skill would have motivation to combine Zamani, Yu, Yao, Zhang, and Wang because, according to Zamani, “any service that provides both search and recommendation functionalities can benefit from such joint modeling and optimization. This includes…media sharing services, such as YouTube…” (Zamani, pg. 2, section 1). Wang’s model obtains “near state-of-the-art accuracies on bi-directional image-sentence retrieval on Flickr30K [14] and MSCOCO [17] datasets” (Wang, pg. 3, section 1), and thus the combination would allow the extension of the capabilities of Zamani’s search and recommendation framework into the realm of multimodal search on media sharing services such as Flickr.
Regarding Claim 11, Zamani, Yu, Yao, Zhang, and Wang teach The method of claim 10, as shown above.
Wang also teaches wherein the first content type is a text content type associated with the search query and the second content type is an image content type. (Pg. 4, section 3.1: “Our second task, bi-directional image-sentence retrieval, refers both to image-to-sentence and sentence-to-image search. The definitions of the two scenarios are straightforward: given an input image (resp. sentence), the goal is to find the best matching sentences (resp. images) from a database.” In image-to-sentence search, text content associated with the search query is used to retrieve image content. Again, figure 1 (pg. 2) shows that the right branch (tower) takes text input, and the left branch (tower) takes image input.)
Conclusion
Claims 1-6, 8-19, and 21-22 are rejected.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BENJAMIN M ROHD whose telephone number is (571)272-6445. The examiner can normally be reached Mon-Thurs 8:00-6:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.M.R./Examiner, Art Unit 2147 /ERIC NILSSON/Primary Examiner, Art Unit 2151
Prosecution Insights