DETAILED ACTION
Status of Claims
This action is in reply to the preliminary amendment filed on 07 August 2025.
Claim 1 is canceled.
Claims 2-21 are new and have been added.
Claims 2-21 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The Information Disclosure Statement filed on 22 June 2026, has been considered. An initialed copy of the Form 1449 is enclosed herewith.
Allowable Subject Matter
Claims 2-21 recite allowable subject matter for reasons given in the Office Action below. The claims would be allowable if re-written, amended, or a terminal disclaimer filed, to overcome the Double Patenting rejection in the Office Action below.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Examiner notes: It is noted that Claim 1 of Patent ‘153 and Claim 2 of the instant claims include features that are recited similarly but have the same meaning. For example, Claim 2 of the instant claims recites a first modality model comprising a first set of first-modality layers and a second modality model comprising a second set of second-modality layers. The first modality model and the second modality model in the instant claim, are the same as the recited the user model and the recipe model in Patent ‘153. This is consistent with what is described in paragraph [0006] of the Applicant’s specification, which describes the machine learning recommendation model that further includes “a set of layers comprising a user model and a different set of layers comprising a recipe model, so the machine learning recommendation model has a two-tower architecture”. Therefore the instant Claim 2 and Claim 1 of Patent ‘153 recites features that have the same meaning.
Claims 2-13 and 15-21 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent 12,354,153, hereinafter referred to as Patent ‘153. Claim 14 is rejected on the ground of nonstatutory double patenting as being unpatentable over claims of Patent ‘153, and further in view of Mor, et al. (PGP No. US 2021/0256367 A1).
Patent ‘153 Claim 1
Instant Claim 2
A method comprising:
A method comprising:
obtaining, at an online concierge system, a plurality of recipes, wherein each recipe includes items;
obtaining, at an online system, a plurality of data instances related to item catalogs comprising items offered by one or more physical locations;
obtaining, at an online concierge system, training data from prior interactions by users with recipes, the training data including a plurality of examples, each example comprising a user, a recipe, and a label indicating whether the user performed a specific interaction with the recipe;
obtaining, at the online system, training data associated with the items,
the
training data includes a plurality of data modalities that are used in the plurality of data instances;
initializing a plurality of layers of a cross-modal machine learning model, the cross-modal machine learning comprising a user model comprising a first set of user layers and a recipe model comprising a second set of recipe layers that are different from the first set of user layers;
initializing a plurality of layers of a cross-modal machine learning model, the cross- modal machine learning comprising a first modality model comprising a first set of first-modality layers and a second modality model comprising a second set of second-modality layers that are different from the first set of first-modality layers;
training the cross-modal machine learning model, wherein training of the cross-modal machine learning model comprises:
training the cross-modal machine learning model, wherein training of the cross-modal machine learning model comprises:
generating, based on characteristics of a user in an example of the training data, a user embedding using the user model comprising the first set of user layers;
generating, using the first set of first-modality layers corresponding to the first modality model and the training data, a first-modality embedding;
generating, based on attributes of a recipe in the example, a recipe embedding for the recipe using the recipe model comprising the second set of recipe layers;
generating, using the second set of second-modality layers corresponding to the second modality model and the training data, a second-modality embedding;
generating a measure of similarity between the user embedding for the user of the example and the recipe embedding for the recipe of the example, the similarity measured in a latent space including the user embedding and the recipe embedding;
generating a measure of similarity between the first-modality embedding and the second-modality embedding, the
similarity measured in a latent space including the first-modality embedding and the second-modality embedding;
generating a cross-modal error term based on a difference between the measure of similarity and the label indicating whether the user performed the specific interaction with the recipe;
generating a cross-modal error term based on a difference between the measure of similarity;
backpropagating the error term through the user model and through the recipe model to update a set of parameters of the cross-modal machine learning model; and
backpropagating the error term through the first set of first-modality layers and through the second set of second-modality layers to update a set of parameters of the cross-modal machine learning model; and
stopping the backpropagation after one or more criteria are satisfied; and
stopping the backpropagation after one or more criteria are satisfied; and
applying the cross-modal machine learning model to select a recipe to recommend to the user.
applying the cross-modal machine learning model to select an item to recommend to a user.
Patent ‘153 Claims 1 and 2
Instant Claim 3
(claim 2) generating, based on
characteristics of the user, a user embedding using the user model;
initializing a first modality model configured to receive characteristics of a first modality;
(claim 2) generating, based on attributes of the recipe, a recipe embedding for the recipe using the recipe model;
initializing a second modality model configured to receive attributes of a second modality; and
(claim 1) generating a measure of similarity between the user embedding for the user of the example and the recipe embedding for the recipe of the example, the similarity measured in a latent space including the user embedding and the recipe embedding;
initializing a latent space configured to embed representations from both the first modality and the second modality.
Patent ‘153 Claim 3
Instant Claim 4
generating, based on characteristics of the user, a user embedding using the user model; generating, based on attributes of the recipe, a recipe embedding for the recipe using the recipe model;
retrieving historical interaction data including labels identifying whether a specific interaction occurred between data instances of the first modality and the second modality;
generating a predicted user embedding from one or more cross-modal user layers configured to receive the recipe embedding for the recipe as input, wherein the predicted user embedding corresponds to a predicted user having at least the threshold probability of performing the specific interaction with the recipe;
processing the historical interaction data to extract examples pairing a first-modality instance with a second-modality instance; and
for each of examples of the training data having a label indicating that a user of the example performed a specific interaction with a recipe of the example:
assigning each example a label based on whether the specific interaction occurred.
Patent ‘153 Claims 3 and 4
Instant Claim 5
(claim 4) applying the cross-modal machine learning model to generate a user embedding for the user;
applying the first modality model to characteristics associated with the first modality;
identify a recipe embedding based on the user embedding; and
producing a vector representation of the first modality in a latent space; and
(claim 3) a label indicating that a user of the example performed a specific interaction with a recipe of the example:
encoding behavioral or descriptive information associated with the first modality into the vector representation.
Patent ‘153 Claim 4
Instant Claim 6
(claim 4) applying the cross-modal machine learning model to generate a user embedding for the user;
applying the second modality model to attributes associated with the second modality;
identifying a recipe embedding based on the user embedding; and
generating multiple embeddings from different data types associated with the second modality; and
wherein one or more attributes of a recipe of the plurality of recipes comprise a fixed-length combined item embedding representing a combination of items included in the recipe
aggregating the multiple embeddings into a unified embedding for the second modality.
Patent ‘153 Claim 1
Instant Claim 7
generating a measure of similarity between the user embedding for the user of the example and the recipe embedding for the recipe of the example, the similarity measured in a latent space including the user embedding and the recipe embedding;
applying a similarity function to the first-modality embedding and the second-modality embedding, the similarity function comprising at least one of dot product, cosine similarity, or triplet loss;
generating a measure of similarity
generating a similarity score; and
generating a cross-modal error term based on a difference between the measure of similarity and the label indicating whether the user performed the specific interaction with the recipe;
comparing the similarity score to a reference value to generate the cross-modal error term.
Patent ‘153 Claims 1, 2, and 3
Instant Claim 8
backpropagating the error term through the user model and through the recipe model to update a set of parameters of the cross-modal machine learning model
wherein backpropagating the error term through the first and second modality models comprises:
model to update a set of parameters of the cross-modal machine learning model
computing gradient updates for parameters in the first-modality layers;
backpropagating the cross-modal recipe error term through cross-modal machine learning model to update the set of parameters of the cross-modal machine learning model
computing gradient updates for parameters in the second-modality layers; and
update the set of parameters of the cross-modal machine learning model; and stopping the backpropagation of the cross-modal user error term after one or more criteria are satisfied
simultaneously updating parameters of both modality models using a shared optimization objective.
Patent ‘153 Claims 4 and 5
Instant Claim 9
generate a user embedding for the user;
generating a first-modality embedding from a new instance;
identifying a recipe embedding based on the user embedding; and identifying the recipe corresponding to the recipe embedding.
identifying a set of second-modality embeddings with similarity measures above a threshold with the first-modality embedding; and
determining a set of candidate recipes based on distances between the user embedding users and the candidate recipes
selecting one or more recipes of the set of candidate recipes.
selecting one or more items associated with the second-modality embeddings for recommendation.
Patent ‘153 Claim 1
Instant Claim 10
generating a measure of similarity between the user embedding for the user of the example and the recipe embedding for the recipe of the example, the similarity measured in a latent space including the user embedding and the recipe embedding; generating a cross-modal error term based on a difference between the measure of similarity and the label indicating whether the user performed the specific interaction with the recipe;
generating classification error terms by comparing predicted categories of the first- modality embedding and second-modality embedding with their corresponding classification labels;
backpropagating the error term through the first set of user layers corresponding to the user model and through the second set of recipe layers corresponding to the recipe model to update a set of parameters of the cross-modal machine learning model
backpropagating the classification error terms through the first and second modality models; and
to update a set of parameters of the cross-modal machine learning model
modifying parameters to align the embeddings to reflect category-level associations.
Patent ‘153 Claim 8
Instant Claim 11
the fixed-length combined item embedding generated
extracting an item embedding for each item included in the second-modality instance;
by application of a regularization model to a combination of the item embeddings corresponding to the items in the recipe.
applying a regularization model to produce a fixed-length embedding from the item embeddings; and
application of a regularization model to a combination of the item embeddings corresponding to the items in the recipe
using the fixed-length embedding as the second-modality embedding.
Patent ‘153 Claim 9
Instant Claim 12
one or more attributes of a recipe of the plurality of recipes comprise one or more of: a term embedding of a name of an item included in the recipe of the example,
applying a term embedding model to textual attributes of the second-modality instance;
a document embedding generated from instructions in the recipe of the example for combining items included in the recipe
applying a document embedding model to instructional or descriptive content of the second-modality instance; and
or an image embedding of an image included in the recipe.
applying an image embedding model to any images associated with the second-modality instance.
Patent ‘153 Claims 1 and 2
Instant Claim 13
generating a cross-modal error term based on a difference between the measure of similarity and the label indicating whether the user performed the specific interaction with the recipe
wherein the training data comprises classification examples associating the first modality and second modality with common categorical labels, and further comprising:
generating a cross-modal recipe error term based on one or more differences between the predicted recipe embedding and the recipe embedding for the recipe
using a shared classification head to predict a common label for the first-modality embedding and the second-modality embedding; and
backpropagating the cross-modal recipe error term through cross-modal machine learning model to update the set of parameters of the cross- modal machine learning model
(claim 1) the similarity measured in a latent space including the user embedding and the recipe embedding
backpropagating classification loss to regularize the latent space.
Claim 14 is on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent 12,354,153, hereinafter referred to as Patent ‘153, and further in view of Mor, et al. (PGP No. US 2021/0256367 A1).
Patent ‘153 Claim 1
Instant Claim 14
stopping the backpropagation of the cross-modal recipe error term after one or more criteria are satisfied.
wherein stopping the backpropagation after one or more criteria are satisfied comprises:
monitoring a convergence condition on a validation loss function;
detecting when the convergence condition is met; and
terminating parameter updates in response to detecting convergence.
Although ‘153 discloses the stopping the backpropagating after one or more criteria are satisfied, ‘153 does not describe a validation loss function for monitoring a condition. ‘153 does not disclose:
a convergence condition on a validation loss function;
detecting when the convergence condition is met; and
terminating parameter updates in response to detecting convergence.
Mor, however, does teach:
a convergence condition on a validation loss function (Mor, see: paragraph [0086] teaching “performance metrics are monitored to ensure that the performance of the “B” set is at least as good as that of the “A” set”; and see: paragraph [0088] teaching “particular, static rank scores 224 that are aligned with objectives (e.g., loss functions, labels in training data, scores outputted by dynamic ranking models, etc.) related to ranking candidates in search results 232 allow for reductions in the number of candidates to be scored or rescored by first-level dynamic ranking models 210 and/or second-level dynamic ranking models 212”);
detecting when the convergence condition is met (Mor, see: paragraph [0086] teaching “performance metrics are monitored to ensure that the performance of the “B” set is at least as good as that of the “A” set”); and
terminating parameter updates in response to detecting convergence (Mor, see: paragraph [0102] teaching “The process may be repeated until the older versions of the scores are “phased out” in processing searches of the candidates”).
This step of Mor is applicable to ‘153, as they both share characteristics and capabilities, namely, they are directed to candidate items. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify ‘153, to include the features of a convergence condition on a validation loss function, detecting when the convergence condition is met; and terminating parameter updates in response to detecting convergence, as taught by Mor. One of ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to modify ‘153 to improve providing candidate recommendations to a user (Mor, see: paragraph [0005]).
Patent ‘153 Claim 2
Instant Claim 15
generating a predicted recipe embedding from one or more cross-modal recipe layers configured to receive the user embedding for the user as input
inputting the first-modality embedding into cross-modal second-modality layers to generate a predicted second-modality embedding;
generating a cross-modal recipe error term based on one or more differences between the predicted recipe embedding and the recipe embedding for the recipe;
generating a cross-modal error term from a difference between the predicted second- modality embedding and the second-modality embedding; and
backpropagating the cross-modal recipe error term through cross-modal machine learning model to update the set of parameters of the cross- modal machine learning model
updating the second modality model based on the cross-modal error term.
Patent ‘153 Claim 3
Instant Claim 16
generating a predicted user embedding from one or more cross-modal user layers configured to receive the recipe embedding for the recipe as input
inputting the second-modality embedding into cross-modal first-modality layers to generate a predicted first-modality embedding;
generating a cross-modal user error term based on one or more differences between the predicted user embedding and the user embedding for the user
generating a cross-modal error term from a difference between the predicted first- modality embedding and the first-modality embedding; and
update the set of parameters of the cross- modal machine learning model; and stopping the backpropagation of the cross-modal user error term after one or more criteria are satisfied.
updating the first modality model based on the cross-modal error term.
Regarding claims 17-19 and 20-21, claims 17-18 are directed to a non-transitory computer-readable medium, and claims 20-21 are directed to a system. Claims 17-19 and 20-21 recite limitations that are parallel in nature to those addressed above for claims 2-16 which are directed towards a method. Claims 17-19 and 20-21 are therefore rejected for the same reasons as set forth above for claims 2-16.
Reasons for Allowable Subject Matter
Prior Art Considerations:
Upon review of the evidence at hand, it is concluded that the totality of evidence in combination, neither anticipates, reasonably teaches, nor renders obvious the below noted features of the Applicant’s invention.
Regarding the independent claims, the features are as follows:
backpropagating the error term through the first set of first-modality layers and through the second set of second-modality layers to update a set of parameters of the cross-modal machine learning model
The most apposite prior art of record includes Pinel, F., et al. (PGP No. US 2019/0243922 A1), in view of Narayan, V., et al. (PGP No. US 2021/0104322 A1), and Mor, A., et al. (PGP No. US 2021/0256367 A1), to teach a method of obtaining a plurality of recipes.
The reference of Pinel discloses a system that utilizes a plurality of user histories that include previous ratings for recipes, where the data is used to train machine learning models and uses both history data and recipe rating data for each user (Pinel, see: paragraphs [0058]-[0059]). The system of Pinel is described to have a storage repository that stores the recipes and input data for each user, as well as predictor models that use machine learning techniques, that utilize the features of each recipes, such as photographs, types of recipes and cuisines, and how to prepare the recipe, as well as the history of interactions for each recipe of every user (Pinel, paragraphs [0051], [0059]-[0060]). The machine learning models use the system’s purchase predictors and the data of each user to determine top recipes for each particular customer (Pinel, see: paragraphs [0063] and [0065]-[0066]). Although Pinel discloses the user data and the data regarding ratings of each user, where the data is used to train machine learning models, Pinel does not disclose backpropagating the error term through the first set of first-modality layers and through the second set of second-modality layers to update a set of parameters of the cross-modal machine learning model.
The reference of Narayan teaches that results that are determined by AI, can be labeled and can be then fed into a front-end framework (Narayan, paragraph [0031]). Narayan describes that recommendations of recipes can be made customized to a user’s preference by using filtering techniques that can compare different types of recipes and generating similarity scores, where similarities can be measured and mapped in a high dimensional space (Narayan, paragraphs [0074] and [0083]). Although Narayan describes these features, Narayan does not describe backpropagating an error term and does not teach the claimed features of backpropagating the error term through the first set of first-modality layers and through the second set of second-modality layers to update a set of parameters of the cross-modal machine learning model.
Next, the reference of Mor describes a system that can determine errors between likely outputs by using machine learning models, where the outcomes can be positive or negative regarding candidates that are backpropagated across a plurality of layers, and the distance between the embeddings are measured and can reflect those outcomes (Mor, see: paragraphs [0073] and [0079]). Mor further describes that optimization techniques can be used to update parameters of the machine learning model (paragraph [0079]), and iterations can be performed until a convergence has been reached (paragraph [0106]). Although Mor describes the steps of backpropagation and updating parameters, and stopping the backpropagation once a convergence is reached, the backpropagation does include the error term through the first layer corresponding to the user model and the second layer that corresponds to the recipe model to update the parameters of a cross-modal machine learning model. Mor does not teach the allowable features of backpropagating the error term through the first set of first-modality layers and through the second set of second-modality layers to update a set of parameters of the cross-modal machine learning model.
The Examiner further emphasizes the claims as a whole and hereby asserts that the totality of the evidence fails to set forth, either explicitly or implicitly, an appropriate rationale for further modification of the evidence at hand to arrive at the claimed invention. Moreover, the combination of features of independent claims, would not have been obvious to one of ordinary skill in the art because any combination of evidence at hand to reach the combination of features as claimed would require substantial reconstruction of Applicant’s claimed invention relying on improper hindsight bias and resulting in an inappropriate combination.
It is hereby asserted by the Examiner, that in light of the above and in further deliberation over all of the evidence at hand, that the claims are allowable, as the evidence at hand does not anticipate the claims and does not render obvious any further modification of the references to a person of ordinary skill in the art.
Examiner’s Comment
The Examiner notes that the non-patent literature (NPL) document, titled Make more money from every day delivery order, found on shipday.com (2021), documented on PTO-892 form as reference U, and hereinafter referred to as ‘Make more’, describes and renders a website for a user to conveniently automate order delivery and improve customer communications with the retailer. Although ‘Make more’ describes such features, the reference does not disclose or teach the allowable features that are stated above, and does not remedy the deficiencies of the noted prior art.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ASHLEY PRESTON whose telephone number is (571)272-4399. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kambiz Abdi can be reached at 571-272-6702. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ASHLEY D PRESTON/Primary Examiner, Art Unit 3688
Prosecution Insights