DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in reply to the communications filed on September 4, 2026. The Applicant’s Amendment and Request for Reconsideration has been received and entered.
Claims 1, 2, 4, 6-9, 11, 13-16, 18, and 20 are currently pending and have been examined. Claims 1, 7, 8, 14, 15, and 18 have been amended. Claims 3, 5, 10, 12, 17, and 19 have been cancelled.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on September 4, 2026 has been entered.
Response to Arguments
Applicant’s amendments necessitated the new grounds of rejection.
The previous rejection of claims 1, 2, 4, 6-9, 11, 13-16, 18, and 20 under 35 USC 112(b) has been withdrawn in view of Applicant’s amendments.
Regarding the rejection of claims 1, 2, 4, 6-9, 11, 13-16, 18, and 20 under 35 USC 101, Applicant’s arguments have been fully considered but they are not persuasive for the reasons set forth infra.
Additionally, the Examiner respectfully argues that the recited pipeline specifies computation tools (e.g. separate machine-learned replacement models invoked through an API, user interface (UI)) for identifying, evaluating, and presenting replacement products without establishing an improvement to those tools. Indeed, identifying the out-of-stock item and the candidates appearing in an image provided by a user, supplying additional candidates that are not present in the image, scoring merged candidates, and providing a message and a set of replacement item cards with links to the recommended replacement items merely refine the commercial replacement item identification process. Their combination does not establish a technological improvement simply because it involves multiple machine-learning stages.
The Examiner respectfully notes that in Example 37, the technical problem asserted was computer users having “a large number of icons on their display, making it difficult to find the icons most used. . . . For example, traditional software does not automatically organize icons so that the most used icons are located near the “start” or “home” icon, where they can be easily accessed.” The claim of Example 37 includes additional elements reciting a specific manner of automatically displaying icons to the user based on usage (i.e. “automatically moving the most used icons to a position on the GUI closest to the start icon of the computer system based on the determined amount of use”), resulting in a technical improvement to user interfaces for electronic devices. The Examiner respectfully argues, that conversely, the present claims do not recite a specific manner of arranging a user interface element, but merely recites arranging the user interface element to a “prominent position” without specifying any particular arrangement for the asserted navigational improvement nor specifying how such interface arrangement produces a navigational or GUI improvement. Rather, Applicant’s specification [0102] describes emphasizing a customer’s preferred transaction option (e.g. refund), which supports preference-based placement, but does not by itself establish that the broadly claimed “prominent position” reflects an improvement to a GUI.
Further, the Examiner respectfully argues that the actively recited training steps, including obtaining a training example, generating the recited loss, and updating the parameters to reduce the loss, as recited, train the multi-modal machine learning model for the replacement item identification task without establishing an improvement to the training technology itself. Similarly, the recitations regarding the tensor specifies input organization/architecture without identifying how that organization/architecture improves the machine learning model operation. Considered as a whole, these limitations do not integrate the exception into practical application.
Thus, the previous rejections of claims 1, 2, 4, 6-9, 11, 13-16, 18, and 20 under 35 USC 101 are maintained.
The previous rejection of claims 1, 2, 4, 6-9, 11, 13-16, 18, and 20 under 35 USC 103 has been withdrawn in view of Applicant’s amendments, the reasons set forth infra.
The Examiner invites Applicant’s Representative to contact Examiner for a telephonic interview to further prosecution of this application.
Claim Objections
Claims 1, 8, and 15 are objected to because of the following informalities:
Claims 1, 8, and 15 each recite the limitation “the machine learning model”. There is insufficient antecedent basis for this limitation in the claims. For examination purposes, Examiner has interpreted “the machine learning model” to mean “the multi-modal machine learning model”.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 2, 4, 6-9, 11, 13-16, 18, and 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Step 1. When considering subject matter eligibility under 35 U.S.C. 101, it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter.
Step 2A – Prong One. If the claims fall within one of the statutory categories, it must then be determined whether the claims recite an abstract idea, law of nature, or natural phenomenon.
Step 2A – Prong Two. If the claims recite an abstract idea, law of nature, or natural phenomenon, it must then be determined whether the claims recite additional elements that integrate the judicial exception into a practical application. If the claims do not recite additional elements that integrate the judicial exception into a practical application, then the claims are directed to a judicial exception.
Step 2B. If the claims are directed to a judicial exception, it must be evaluated whether the claims recite additional elements that amount to an inventive concept (i.e. “significantly more”) than the recited judicial exception.
In the instant case, claims 1, 2, 4, 6, and 7 are directed to a process; claims 8, 9, 11, 13, and 14 are directed to a manufacture; and claims 15, 16, 18 and 20 are directed to a machine.
A claim “recites” an abstract idea if there are identifiable limitations that fall within at least one of the groupings of abstract ideas enumerated in MPEP 2106. In the instant case, claims 1, 8, and 15 each recite the steps of:
receiving an image, from a user, indicating that a target item is not available, and the image comprising one or more potential replacements items for the target item; training parameters, the training comprising: obtaining at least a training example including a training image including a tag for an out-of-stock item and a set of other items, generating a loss indicating a difference between (i) an estimated response generated by applying to at least the training image, and (ii) description for a set of replacement items known to be replacements for the out-of-stock items identified in the training image, and updating the parameters to reduce the loss; providing the image and a prompt requesting identification of the target item and the one or more potential replacement items in the image, wherein the image is input as a tensor, receiving, as a response, identification of the target item, and the one or more potential replacement items identified in the image; generating a first list of potential replacements items based on the one or more potential replacement items identified in the image; generating a second list of potential replacement items from the target item by transmitting a request to a replacement recommendation application, wherein the request triggers execution of one or more replacement models on the identified target item, and receiving the second list of potential replacement items in a response from the replacement recommendation, wherein the second list of potential replacement items are not present in the image; merging the first list and the second list of potential replacement items; assigning replacement scores to each item in the merged list of potential replacement items to create a list of recommended replacement items; generating a message based on the image and the list of recommended replacement items; and providing the message and a set of user elements including a set of replacement item cards with links to the list of recommended replacement items, the providing comprising: determining that a user element is more frequently selected by a customer user compared to remaining user elements in the set, based on the determination, arranging the user element to a prominent position -- these claim limitations set forth certain methods of organizing human activity, particularly commercial interactions including advertising, marketing, and sales activities/behaviors.
The limitations of the claims are not indicative of integration into a practical application. Taking the independent claim elements separately, the additional elements of performing the steps via a multi-modal machine learning model configured as a transformer architecture, a tensor including a series of pixel values arranged along a first dimension and a second dimension, and RGB channels arranged along a third dimension; an application programming interface (API), user interface elements, URL links, and a user interface (UI) merely implements the abstract idea on a computer environment. Additionally, taking the dependent claim elements separately, the additional elements of performing the steps via a user interface, an interactable user interface element, a machine-learned language model, and automatically pushing also merely implement the abstract idea on a computer environment. Considered in combination, the steps of Applicant’s method add nothing that is not already present when the steps are considered separately.
Thus, claims 1, 2, 4, 6-9, 11, 13-16, 18, and 20 are directed to an abstract idea.
Regarding the independent claims, the technical elements of performing the steps via a multi-modal machine learning model configured as a transformer architecture, a tensor including a series of pixel values arranged along a first dimension and a second dimension, and RGB channels arranged along a third dimension; an application programming interface (API), user interface elements, URL links, and a user interface (UI) are recited at a high level of generality and thus does not amount to significantly more. Additionally, regarding the dependent claims, the technical elements of performing the steps via a user interface, an interactable user interface element, a machine-learned language model, and automatically pushing also merely implement the abstract idea on a computer environment.
When considering the elements and combinations of elements, the claim(s) as a whole, do not amount to significantly more than the abstract idea itself. This is because the claims do not amount to an improvement to another technology or technical field; the claims do not amount to an improvement to the functioning of a computer itself; the claims do not move beyond a general link of the use of an abstract idea to a particular technological environment; the claims merely amounts to the application or instructions to apply the abstract idea on a computer; or the claims amounts to nothing more than requiring a generic computer to perform generic computer functions that are well-understood, routine and conventional activities previously known to the industry.
The analysis above applies to all statutory categories of invention. Accordingly, claims 1, 2, 4, 6-9, 11, 13-16, 18, and 20 are rejected as ineligible for patenting under 35 USC 101 based upon the same rationale.
Subject Matter Free of Prior Art
Claims 1, 8, and 15 are determined to have overcome the prior art of rejection and are free of prior art, however the claims remain rejected under 35 USC 101, as set forth above. All dependent claims are also free of prior art by virtue of dependency, but remain rejected under 35 USC 101.
Taking amended claim 1 as a representative claim, the claims as amended are found to overcome the prior art rejection for the reasons set forth below.
Claim 1 now recites the additional claimed features of training parameters of a multi-modal machine learning model configured as a transformer architecture, the training comprising: obtaining at least a training example including a training image including a tag for an out-of-stock item and a set of other items, generating a loss indicating a difference between (i) an estimated response generated by applying the transformer architecture to at least the training image, and (ii) description for a set of replacement items known to be replacements for the out-of-stock items identified in the training image, and updating the parameters of the transformer architecture to reduce the loss; providing, to the machine learning model, the image and a prompt requesting identification of the target item and the one or more potential replacement items in the image, wherein the image is input to the machine learning model as a tensor, the tensor including a series of pixel values arranged along a first dimension and a second dimension, and RGB channels arranged along a third dimension; generating a second list of potential replacement items from the target item by transmitting a request to a replacement recommendation application programming interface (API), wherein the request triggers execution of one or more machine-learned replacement models on the identified target item, and receiving the second list of potential replacement items in a response from the replacement recommendation API, wherein the second list of potential replacement items are not present in the image; and providing the message and a set of user interface elements including a set of replacement item cards with URL links to the list of recommended replacement items on a user interface (UI), the providing comprising: determining that a user interface element is more frequently selected by a customer user compared to remaining user interface elements in the set, based on the determination, arranging the user interface element to a prominent position on the UI.
The closest prior art was found to be as follows:
Wang (US PGP 2021/0232773) recites [0026]-[0028] Pre-trained language models, such as BERT, typically employ a transformer encoder as the backbone with either a two-stream architecture that uses two transformers to encode text and image independently, or a single-stream architecture, where a single transformer is applied to both text and image. [0041] The structure 400 employs a unified transformer encoder 450 to encode both the image input 410 and the multi-turn dialogue history input 420. The structure 400 employs the input sequence module 440 to first encode the image input 410 (including caption) and the multi-turn dialogue history input 420 concurrently with the unified transformer encoder 450. In some embodiments, the structure 400 can append an answer option to the input (e.g., appended to the user question 430) and model their interaction in an early fusion manner. Next, the structure 400 can adopt visually-grounded MLM and NSP learning objectives to train the unified transformer encoder 450 for effective vision and dialogue fusion using two types of self-attention masks (e.g., 334).) [0042], [0064]-[0065] Training a plurality of model parameters of the model using determined loss.
Lauka (US Pat No 10/592,962) recites [Figs. 3A-3C; Col. 5, Ln. 45 through Col. 6, Ln. 40] send[ing] information to the management module 102 regarding items that are out of stock and possible substitutions. . . . the handheld electronic device 106 can capture one or more images 300 of the shelf 108 on which the item is normally stocked, but is currently out of stock. So, for example, if the customer requested yogurt 302 c, which is currently out of stock, the picker 104 can take an image of the shelf proximate the out of stock item 302 c with the imaging device 206 and send it to the management module 102 with the communications system 214. The image can include the other yogurts 302 a, 302 b, 302 d, 302 e, etc. that are available and thus, represent possible substitutions. . . . The management module 102 can then send a message to the customer stating, for example, “The peach yogurt you requested is out of stock. We can offer you the following substitutions:”
Temam (US PGP 2019/0205756) recites [0023] the three dimensional tensor may represent the features of an image being classified by a neural network. A first dimension (Z) may represent the width of the image, the second dimension (Y) may represent the height of the image, and the third dimension (X) may represent RGB values for pixels in the image.
Kruck (US PGP 2023/0111745) recites [0016] the BOPUS system 110 includes an application programming interface (API) and [0051] the BOPUS system 110 generates an in-app notification to display the item that is unavailable, one or more recommended replacement items, and a selectable element to approve or reject the recommendation.
Sidman (US PGP 2021/0216599) recites [0199] In another embodiment, an ad agency may use the tracking information to refine MultiLink menus so that more popular menu selections are more prominently displayed. For example, an ad agency might move a popular menu selection towards the top so it is more easily selected.
Sears ("Split menus: effectively using selection frequency to organize menus." ACM Transactions on Computer-Human Interaction (TOCHI) 1.1 (1994): 27-51.) discloses (Page 29) frequency-based orderings typically refer to placing the most frequently used item at the top of the menu, followed by the next most frequent item.
It was found that no references alone or in combination, neither anticipates, reasonable teaches, nor renders obvious the below noted features of Applicant’s invention. The features of claim 1 (and parallel claims 8 and 15) in combination that overcome the prior art are:
training parameters of a multi-modal machine learning model configured as a transformer architecture, the training comprising: obtaining at least a training example including a training image including a tag for an out-of-stock item and a set of other items, generating a loss indicating a difference between (i) an estimated response generated by applying the transformer architecture to at least the training image, and (ii) description for a set of replacement items known to be replacements for the out-of-stock items identified in the training image, and updating the parameters of the transformer architecture to reduce the loss;
providing, to the machine learning model, the image and a prompt requesting identification of the target item and the one or more potential replacement items in the image, wherein the image is input to the machine learning model as a tensor, the tensor including a series of pixel values arranged along a first dimension and a second dimension, and RGB channels arranged along a third dimension;
generating a second list of potential replacement items from the target item by transmitting a request to a replacement recommendation application programming interface (API), wherein the request triggers execution of one or more machine-learned replacement models on the identified target item, and receiving the second list of potential replacement items in a response from the replacement recommendation API, wherein the second list of potential replacement items are not present in the image; and
providing the message and a set of user interface elements including a set of replacement item cards with URL links to the list of recommended replacement items on a user interface (UI), the providing comprising: determining that a user interface element is more frequently selected by a customer user compared to remaining user interface elements in the set, based on the determination, arranging the user interface element to a prominent position on the UI.
Therefore, none of the cited references disclose or render obvious each and every feature of the claimed invention and the claimed invention is determined to be free of the prior art. Although individually the claimed features could be taught, any combination of references would teach the claimed limitations using a piecemeal analysis, since references would only be combined and deemed obvious based on knowledge gleaned from the applicant's disclosure. Such a reconstruction is improper (i.e., hindsight reasoning). See In re McLaughlin, 443 F.2d 1392, 170 USPQ 209 (CCPA 1971). The examiner emphasizes that it is the interrelationship of the limitations that renders these claims free of the prior art/additional art.
Thus, it is hereby asserted by the Examiner that, in light of the above, that the claims are free of prior art as the references do not anticipate the claims and do not render obvious any further modification of the references to a person of ordinary skill in art.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Patil (US PGP 2023/0274410) – identifying missing/out-of-stock items
Fang (US PGP 2019/0251446) – generating a loss function and backpropagating an MLM to update parameters of the MLM based on loss function
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JENNIFER V LEE whose telephone number is (571)272-4778. The examiner can normally be reached Monday - Friday 9AM - 5PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JEFFREY A. SMITH can be reached at (571)272-6763. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JENNIFER V LEE/Examiner, Art Unit 3688
/VICTORIA E. FRUNZI/Primary Examiner, Art Unit 3689
Prosecution Insights