DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in reply to the communications filed on January 26, 2026. The Applicant’s Amendment and Request for Reconsideration has been received and entered.
Claims 1, 2, 4-9, 11-16, and 18-20 are currently pending and have been examined. Claims 1, 6, 8, 13, 15, and 20 have been amended. Claims 3, 10, and 17 have been cancelled.
Response to Arguments
Applicant’s amendments necessitated the new grounds of rejection.
The previous rejection of claims 3, 10, and 17 under 35 USC 112(b) has been withdrawn in view of Applicant’s amendments.
Regarding the rejection of claims 1, 2, 4-9, 11-16, and 18-20 under 35 USC 101, Applicant’s arguments have been fully considered but they are not persuasive for the reasons set forth infra. The Examiner respectfully argues that unlike Example 39, the claims do recite a judicial exception as “identifying potential replacement items from an image when a target item is out of stock” are related to advertising, marketing, and sales activities/behaviors.
Applicant’s remaining arguments have been fully considered but they are not persuasive. Particularly, Applicant’s arguments are directed to the instantly amended claims, and are thus moot in view of the new grounds of rejection.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 2, 4-9, 11-16, and 18-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA , the applicant, regards as the invention.
Claim 1, and similarly claims 8 and 15, recite “generating a loss indicating a difference between an estimated response generated by applying the transformer architecture to at least the training image and description for a set of replacement items known to be replacements for the out-of-stock item identified in the training image, and updating the parameters of the transformer architecture to reduce the loss; [emphasis added].” As recited, it is unclear what are the two elements for which there a difference is indicated. The term “between” signals that there should be two elements, but as recited there only appears to be an estimated response; it is unclear what that the other element. Consequently, one of ordinary skill in the art cannot determine how to avoid infringement of these claims because the metes and bounds of these claims are unclear. For examination purposes, the Examiner has interpreted these limitations as merely a training image and description for a set of replacement items known to be replacements for the out-of-stock item identified in the training image.
Claims 2 and 4-7 depend from claim 1 and thus inherit the deficiencies of claim 1.
Claims 9 and 11-14 depend from claim 8 and thus inherit the deficiencies of claim 8.
Claims 16 and 18-20 depend from claim 15 and thus inherit the deficiencies of claim 15.
Claim 1, and similarly claims 8 and 15, recite “wherein the second list of potential replacement items are obtained by invoking a replacement recommendation application programming interface (API) exposing the one or more machine-learned replacement models [emphasis added].” As recited it is unclear whether it is the API being invoked that is exposed or the API that is exposing. Consequently, one of ordinary skill in the art cannot determine how to avoid infringement of these claims because the metes and bounds of these claims are unclear. For examination purposes, the Examiner has interpreted this limitation as merely wherein the second list of potential replacement items are obtained.
Claims 2 and 4-7 depend from claim 1 and thus inherit the deficiencies of claim 1.
Claims 9 and 11-14 depend from claim 8 and thus inherit the deficiencies of claim 8.
Claims 16 and 18-20 depend from claim 15 and thus inherit the deficiencies of claim 15.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 2, 4-9, 11-16, and 18-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Step 1. When considering subject matter eligibility under 35 U.S.C. 101, it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter.
Step 2A – Prong One. If the claims fall within one of the statutory categories, it must then be determined whether the claims recite an abstract idea, law of nature, or natural phenomenon.
Step 2A – Prong Two. If the claims recite an abstract idea, law of nature, or natural phenomenon, it must then be determined whether the claims recite additional elements that integrate the judicial exception into a practical application. If the claims do not recite additional elements that integrate the judicial exception into a practical application, then the claims are directed to a judicial exception.
Step 2B. If the claims are directed to a judicial exception, it must be evaluated whether the claims recite additional elements that amount to an inventive concept (i.e. “significantly more”) than the recited judicial exception.
In the instant case, claims 1, 2, and 4-7 are directed to a process; claims 8-9 and 11-14 are directed to a manufacture; and claims 15, 16, and 18-20 are directed to a machine.
A claim “recites” an abstract idea if there are identifiable limitations that fall within at least one of the groupings of abstract ideas enumerated in MPEP 2106. In the instant case, claims 1, 8, and 15 each recite the steps of:
receiving an image, from a user, indicating that a target item is not available, and the image comprising one or more potential replacements items for the target item; providing the image and a prompt requesting identification of the target item and the one or more potential replacement items in the image, and parameters are finetuned by: obtaining at least a example including a image including a tag for an out-of-stock item and a set of other items, generating a loss indicating a difference between an estimated response generated by applying to at least the image and for a set of replacement items known to be replacements for the out-of-stock item identified in the image, and updating the parameters to reduce the loss; receiving, as a response, identification of the target item, and the one or more potential replacement items identified in the image; generating a first list of potential replacements items based on the one or more potential replacement items identified in the image; generating a second list of potential replacement items from the target item by applying one or more replacement models to the identified target item, wherein the second list of potential replacement items are obtained by invoking a replacement recommendation; merging the first list and the second list of potential replacement items; assigning replacement scores to each item in the merged list of potential replacement items to create a list of recommended replacement items; generating a message based on the image and the list of recommended replacement items; and providing the message and a set of replacement item cards associated with the list of recommended replacement items to the user for review -- these claim limitations set forth certain methods of organizing human activity, particularly commercial interactions including advertising, marketing, and sales activities/behaviors.
The limitations of the claims are not indicative of integration into a practical application. Taking the independent claim elements separately, the additional elements of performing the steps via a machine learning model, wherein the machine learning model is configured as a transformer architecture, and parameters of the transformer architecture are finetuned, training data, and an application programming interface (API) exposing the one or more machine-learned replacement models merely implements the abstract idea on a computer environment. Additionally, taking the dependent claim elements separately, the additional elements of performing the steps via a user interface, an interactable user interface element, a machine-learned language model, and automatically pushing also merely implement the abstract idea on a computer environment. Considered in combination, the steps of Applicant’s method add nothing that is not already present when the steps are considered separately.
Thus, claims 1, 2, 4-9, 11-16, and 18-20 are directed to an abstract idea.
Regarding the independent claims, the technical elements of performing the steps via a machine learning model, wherein the machine learning model is configured as a transformer architecture, and parameters of the transformer architecture are finetuned, training data, and an application programming interface (API) exposing the one or more machine-learned replacement models are recited at a high level of generality and thus does not amount to significantly more. Additionally, regarding the dependent claims, the technical elements of performing the steps via a user interface, an interactable user interface element, a machine-learned language model, and automatically pushing also merely implement the abstract idea on a computer environment.
When considering the elements and combinations of elements, the claim(s) as a whole, do not amount to significantly more than the abstract idea itself. This is because the claims do not amount to an improvement to another technology or technical field; the claims do not amount to an improvement to the functioning of a computer itself; the claims do not move beyond a general link of the use of an abstract idea to a particular technological environment; the claims merely amounts to the application or instructions to apply the abstract idea on a computer; or the claims amounts to nothing more than requiring a generic computer to perform generic computer functions that are well-understood, routine and conventional activities previously known to the industry.
The analysis above applies to all statutory categories of invention. Accordingly, claims 1, 2, 4-9, 11-16, and 18-20 are rejected as ineligible for patenting under 35 USC 101 based upon the same rationale.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4-9, 11-16, 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Lauka (US Pat No 10/592,962) in view of Wang (US PGP 2021/0232773).
As per claim 1, Lauka teaches [a] method comprising:
receiving an image, from a user, indicating that a target item is not available, and the image comprising one or more potential replacements items for the target item; (Lauka: Figs. 3A-3C; Col. 5, Ln. 45 through Col. 6, Ln. 40 (As shown in FIG. 3A, in some examples, the handheld electronic device 106 can capture one or more images 300 of the shelf 108 on which the item is normally stocked, but is currently out of stock. So, for example, if the customer requested yogurt 302 c, which is currently out of stock, the picker 104 can take an image of the shelf proximate the out of stock item 302 c with the imaging device 206 and send it to the management module 102 with the communications system 214. The image can include the other yogurts 302 a, 302 b, 302 d, 302 e, etc. that are available and thus, represent possible substitutions. . . . In some examples, the management module 102 may compare the shelf bar codes 304 from the shelf 108 to the product bar codes 306 on the products to determine what products are out of stock. In other words, if the management module 102 detects seven shelf bar codes 304 on the shelf, but only six product bar codes 306 on the products, then the bar code that appears on the shelf 108, but not on the product 306 is the missing, or out of stock item.))
providing, to a machine learning model, the image and a prompt requesting identification of the target item and the one or more potential replacement items in the image, . . . (Lauka: Figs. 3A-3C; Col. 5, Ln. 45 through Col. 6, Ln. 40 (The communications system 214, coupled with the imaging device 206 and/or scanner 208 can send information to the management module 102 regarding items that are out of stock and possible substitutions. . . . As shown in FIG. 3A, in some examples, the handheld electronic device 106 can capture one or more images 300 of the shelf 108 on which the item is normally stocked, but is currently out of stock. So, for example, if the customer requested yogurt 302 c, which is currently out of stock, the picker 104 can take an image of the shelf proximate the out of stock item 302 c with the imaging device 206 and send it to the management module 102 with the communications system 214. The image can include the other yogurts 302 a, 302 b, 302 d, 302 e, etc. that are available and thus, represent possible substitutions. . . . In some examples, the management module 102 may compare the shelf bar codes 304 from the shelf 108 to the product bar codes 306 on the products to determine what products are out of stock. In other words, if the management module 102 detects seven shelf bar codes 304 on the shelf, but only six product bar codes 306 on the products, then the bar code that appears on the shelf 108, but not on the product 306 is the missing, or out of stock item. . . . Based on the shelf bar codes 304, the management module 102 can consult an inventory list, database, or other reference to determine what products are in stock. The management module 102 can then send a message to the customer stating, for example, “The peach yogurt you requested is out of stock. We can offer you the following substitutions:”); Col. 12, Lns. 22-34 (The management module 102 . . . machine learning process.); Figs. 5A-5B; Col. 8, Ln.61 through Col. 9, Ln. 32)
obtaining at least a . . . example including a . . . image including a tag for an out-of-stock item and a set of other items, (Lauka: Figs. 3A-3C; Col. 5, Ln. 45 through Col. 6, Ln. 40 (The communications system 214, coupled with the imaging device 206 and/or scanner 208 can send information to the management module 102 regarding items that are out of stock and possible substitutions. . . . As shown in FIG. 3A, in some examples, the handheld electronic device 106 can capture one or more images 300 of the shelf 108 on which the item is normally stocked, but is currently out of stock. So, for example, if the customer requested yogurt 302 c, which is currently out of stock, the picker 104 can take an image of the shelf proximate the out of stock item 302 c with the imaging device 206 and send it to the management module 102 with the communications system 214. The image can include the other yogurts 302 a, 302 b, 302 d, 302 e, etc. that are available and thus, represent possible substitutions. . . . In some examples, the management module 102 may compare the shelf bar codes 304 from the shelf 108 to the product bar codes 306 on the products to determine what products are out of stock. In other words, if the management module 102 detects seven shelf bar codes 304 on the shelf, but only six product bar codes 306 on the products, then the bar code that appears on the shelf 108, but not on the product 306 is the missing, or out of stock item. . . . Based on the shelf bar codes 304, the management module 102 can consult an inventory list, database, or other reference to determine what products are in stock. The management module 102 can then send a message to the customer stating, for example, “The peach yogurt you requested is out of stock. We can offer you the following substitutions:”); Col. 12, Lns. 22-34 (The management module 102 . . . machine learning process.); Figs. 5A-5B; Col. 8, Ln.61 through Col. 9, Ln. 32)
. . . image and description for a set of replacement items known to be replacements for the out-of-stock item identified in the . . . image, and (Lauka: Figs. 3A-3C; Col. 5, Ln. 45 through Col. 6, Ln. 40 (The communications system 214, coupled with the imaging device 206 and/or scanner 208 can send information to the management module 102 regarding items that are out of stock and possible substitutions. . . . As shown in FIG. 3A, in some examples, the handheld electronic device 106 can capture one or more images 300 of the shelf 108 on which the item is normally stocked, but is currently out of stock. So, for example, if the customer requested yogurt 302 c, which is currently out of stock, the picker 104 can take an image of the shelf proximate the out of stock item 302 c with the imaging device 206 and send it to the management module 102 with the communications system 214. The image can include the other yogurts 302 a, 302 b, 302 d, 302 e, etc. that are available and thus, represent possible substitutions. . . . In some examples, the management module 102 may compare the shelf bar codes 304 from the shelf 108 to the product bar codes 306 on the products to determine what products are out of stock. In other words, if the management module 102 detects seven shelf bar codes 304 on the shelf, but only six product bar codes 306 on the products, then the bar code that appears on the shelf 108, but not on the product 306 is the missing, or out of stock item. . . . Based on the shelf bar codes 304, the management module 102 can consult an inventory list, database, or other reference to determine what products are in stock. The management module 102 can then send a message to the customer stating, for example, “The peach yogurt you requested is out of stock. We can offer you the following substitutions:”); Col. 12, Lns. 22-34 (The management module 102 . . . machine learning process.); Figs. 5A-5B; Col. 8, Ln.61 through Col. 9, Ln. 32)
receiving, as a response, identification of the target item, and the one or more potential replacement items identified in the image; (Lauka: Figs. 3A-3C; Col. 5, Ln. 45 through Col. 6, Ln. 40 (The image can include the other yogurts 302 a, 302 b, 302 d, 302 e, etc. that are available and thus, represent possible substitutions. . . . At the management module 102, the picture can undergo a variety of processing. In some examples, the management module 102 can attempt to read one or more shelf bar codes 304 provided by the store on the shelf 108 to determine what products are still in stock. In other examples, the management module 102 can attempt to read one or more product bar codes 306 on the product packaging to determine what products are still in stock. In some examples, the management module 102 may compare the shelf bar codes 304 from the shelf 108 to the product bar codes 306 on the products to determine what products are out of stock. In other words, if the management module 102 detects seven shelf bar codes 304 on the shelf, but only six product bar codes 306 on the products, then the bar code that appears on the shelf 108, but not on the product 306 is the missing, or out of stock item. . . . Based on the shelf bar codes 304, the management module 102 can consult an inventory list, database, or other reference to determine what products are in stock. The management module 102 can then send a message to the customer stating, for example, “The peach yogurt you requested is out of stock. We can offer you the following substitutions: The management module 102 can then provide a numbered list, . . . ”); Figs. 5A-5B; Col. 8, Ln.61 through Col. 9, Ln. 32 (At 508, the management module 102 can perform any necessary post-processing on the image. The management module 102 may extract bar codes 304, 306 from the image, for example, to determine what substitutions are available. In some examples, based on this information, the management module 102 may create a list of possible substitutions.)
generating a first list of potential replacement items based on the one or more potential replacement items identified in the image; (Lauka: Figs. 3A-3C; Col. 5, Ln. 45 through Col. 6, Ln. 40 (The image can include the other yogurts 302 a, 302 b, 302 d, 302 e, etc. that are available and thus, represent possible substitutions. . . . At the management module 102, the picture can undergo a variety of processing. In some examples, the management module 102 can attempt to read one or more shelf bar codes 304 provided by the store on the shelf 108 to determine what products are still in stock. In other examples, the management module 102 can attempt to read one or more product bar codes 306 on the product packaging to determine what products are still in stock. . . . Based on the shelf bar codes 304, the management module 102 can consult an inventory list, database, or other reference to determine what products are in stock. The management module 102 can then send a message to the customer stating, for example, “The peach yogurt you requested is out of stock. We can offer you the following substitutions: The management module 102 can then provide a numbered list, . . . ”); Figs. 5A-5B; Col. 8, Ln.61 through Col. 9, Ln. 32 (At 508, the management module 102 can perform any necessary post-processing on the image. The management module 102 may extract bar codes 304, 306 from the image, for example, to determine what substitutions are available. In some examples, based on this information, the management module 102 may create a list of possible substitutions.))
generating a second list of potential replacement items from the target item by applying one or more machine-learned replacement models to the identified target item, wherein the second list of potential replacement items are obtained by invoking a replacement recommendation application programming interface (API) exposing the one or more machine-learned replacement models; (Lauka: Fig. 3A-3C; Col. 7, Ln. 30-42 (As discussed above, and shown in FIG. 3C, in some examples, the products 302 a-302 n may occupy multiple shelves. In this case, it may be desirable to send an image of multiple shelves 108, as shown, or to send multiple images. In some examples, there may be some substitute products (e.g., yogurts 302 a-302 n) on the same shelf 108, or in the same area, with the requested product (e.g., yogurt 302 c), but other possible substitutes in another area of the store (e.g., sandwich meat may be available on the refrigerated aisle and in the deli). In this case, the picker 104 can provide multiple images to cover multiple locations.); Figs. 5A-5B; Col. 8, Ln.61 through Col. 9, Ln. 32 (In some examples, the picker 104 can provide a list of possible substitutions that are in stock. Alternatively or additionally, at 506, in some examples, the picker 104 can also take an image of the shelf, bin, freezer, cooler, or other area(s) immediately surrounding the missing item to provide an overview of possible substitutions. In some examples, as discussed above, the picker 104 can take multiple images in multiple locations associated with possible substitute items (e.g., bread may be located on the bread aisle and in the bakery). At 508, the management module 102 can perform any necessary post-processing on the image. The management module 102 may extract bar codes 304, 306 from the image, for example, to determine what substitutions are available. In some examples, based on this information, the management module 102 may create a list of possible substitutions); Col. 5, Ln. 45 through Col. 6, Ln. 40)
merging the first list and the second list of potential replacement items; (Lauka: Fig. 3A-3C; Col. 7, Ln. 30-42 (As discussed above, and shown in FIG. 3C, in some examples, the products 302 a-302 n may occupy multiple shelves. In this case, it may be desirable to send an image of multiple shelves 108, as shown, or to send multiple images. In some examples, there may be some substitute products (e.g., yogurts 302 a-302 n) on the same shelf 108, or in the same area, with the requested product (e.g., yogurt 302 c), but other possible substitutes in another area of the store (e.g., sandwich meat may be available on the refrigerated aisle and in the deli). In this case, the picker 104 can provide multiple images to cover multiple locations.); Figs. 5A-5B; Col. 8, Ln.61 through Col. 9, Ln. 32 (In some examples, the picker 104 can provide a list of possible substitutions that are in stock. Alternatively or additionally, at 506, in some examples, the picker 104 can also take an image of the shelf, bin, freezer, cooler, or other area(s) immediately surrounding the missing item to provide an overview of possible substitutions. In some examples, as discussed above, the picker 104 can take multiple images in multiple locations associated with possible substitute items (e.g., bread may be located on the bread aisle and in the bakery). At 508, the management module 102 can perform any necessary post-processing on the image. The management module 102 may extract bar codes 304, 306 from the image, for example, to determine what substitutions are available. In some examples, based on this information, the management module 102 may create a list of possible substitutions); Col. 5, Ln. 45 through Col. 6, Ln. 40)
assigning replacement scores to each item in the merged list of potential replacement items to create a list of recommended replacement items; (Lauka: Figs. 5A-5B; Col. 8, Ln.61 through Col. 9, Ln. 32 (In some examples, the picker 104 can provide a list of possible substitutions that are in stock. Alternatively or additionally, at 506, in some examples, the picker 104 can also take an image of the shelf, bin, freezer, cooler, or other area(s) immediately surrounding the missing item to provide an overview of possible substitutions. In some examples, as discussed above, the picker 104 can take multiple images in multiple locations associated with possible substitute items (e.g., bread may be located on the bread aisle and in the bakery). At 508, the management module 102 can perform any necessary post-processing on the image. The management module 102 may extract bar codes 304, 306 from the image, for example, to determine what substitutions are available. In some examples, based on this information, the management module 102 may create a list of possible substitutions); Col. 10, Ln. 54 through Col. 11, Ln. 64 (The efficiency of the substitution process, on the other hand, can also be improved by tracking those substitutions that are most often accepted by customers. Thus, if a particular flavor of ice cream is rarely accepted as a substitute (e.g., less than 50% of the time), then the substitution queue 606 can simply remove that flavor as a choice. In other examples, if a particular kind of ground beef is almost always accepted as a substitution for another kind of ground beef (e.g., 95% of the time), then that substitution can be placed at the top of the list of substitutions in the substitution request.); Fig. 3A-3C; Col. 7, Ln. 30-42; Col. 5, Ln. 45 through Col. 6, Ln. 40)
generating a message based on the image and the list of recommended replacement items; and (Lauka: Figs. 5A-5B; Col. 9, Ln.14 through Col. 10, Ln. 32 (At 508, the management module 102 can perform any necessary post-processing on the image. . . . In some examples, based on this information, the management module 102 may create a list of possible substitutions. . . . At 510, the management module 102 can create and send a substitution message to the customer requesting a response. In some examples, the substitution message can include an explanation, such as “The item you requested is not available. Would you like to substitute any of the following items?” This can be followed by a numbered list of substitutions, for example, or the aforementioned image with or without selection boxes 308. . . At 512, the management module 102 can receive a response from the customer. This can include a number from the number list, for example, an activated selection box 308, or a circle, box, or X drawn on the image by the customer. . . . At 518, if the substitution was discovered early in the picking process and the customer was quick with their response, the original picker (Picker 1) may still be in the picking process. If so, the substitution can be appended to Picker 1's list to complete the order. As mentioned above, in some examples, the substitution can simply be added to the end of Picker 1's list. In other examples, the substitution can be intelligently added to Picker 1's list—e.g., it can be added to the list near other products in the same area. . . . Regardless of who receives the substitution (i.e., Picker 1 or Picker 2), the management module 102 can send the substitution response to the appropriate picker 104. In some examples, the substitution response can comprise the name or UPC, for example, of the actual product (e.g., if the management module 102 was able to extract the information from the customer's response).))
providing the message and a set of replacement item cards associated with the list of recommended replacement items to the user for review. (Lauka: Col. 4, Lns. 33-40 (handheld electronic device 106 to receive orders, substitutions, and other information from the management module 102.); Figs. 5A-5B; Col. 9, Ln.14 through Col. 10, Ln. 32 (At 512, the management module 102 can receive a response from the customer. This can include a number from the number list, for example, an activated selection box 308, or a circle, box, or X drawn on the image by the customer. . . . At 518, if the substitution was discovered early in the picking process and the customer was quick with their response, the original picker (Picker 1) may still be in the picking process. If so, the substitution can be appended to Picker 1's list to complete the order. As mentioned above, in some examples, the substitution can simply be added to the end of Picker 1's list. In other examples, the substitution can be intelligently added to Picker 1's list—e.g., it can be added to the list near other products in the same area. . . . Regardless of who receives the substitution (i.e., Picker 1 or Picker 2), the management module 102 can send the substitution response to the appropriate picker 104. In some examples, the substitution response can comprise the name or UPC, for example, of the actual product (e.g., if the management module 102 was able to extract the information from the customer's response). In other examples, the management module 102 can simply send the image, for example, to the picker 104 for the picker 104 to decipher. If the image contains a selection box 308 activated by the customer, for example, the picker 104 can easily determine which product the customer has selected. Similarly, if the customer has drawn a circle or box around the substitution (assuming it is sufficiently clear), the picker 104 can easily identify the substitution.))
Lauka does not explicitly disclose the following known techniques which are taught by Wang:
. . . wherein the machine learning model is configured as a transformer architecture, and parameters of the transformer architecture are finetuned by: (Wang: [0017]-]0018]; [0026]-[0028] (Pre-trained language models, such as BERT, typically employ a transformer encoder as the backbone with either a two-stream architecture that uses two transformers to encode text and image independently, or a single-stream architecture, where a single transformer is applied to both text and image. . . Specifically, the subject technology encodes an image input and a multi-turn dialogue history input concurrently using a unified transformer and directly fine tune the pre-trained BERT on visual dialogue data for effective fusion of vision and dialogue contents. The trained model of the subject technology can either rank answer candidates or generate answers directly.))
the machine learning model is configured as a transformer architecture, and parameters of the transformer architecture are finetuned by obtaining at least a training example including a training image . . . (Wang: [0041] (The structure 400 employs a unified transformer encoder 450 to encode both the image input 410 and the multi-turn dialogue history input 420. The structure 400 employs the input sequence module 440 to first encode the image input 410 (including caption) and the multi-turn dialogue history input 420 concurrently with the unified transformer encoder 450. In some embodiments, the structure 400 can append an answer option to the input (e.g., appended to the user question 430) and model their interaction in an early fusion manner. Next, the structure 400 can adopt visually-grounded MLM and NSP learning objectives to train the unified transformer encoder 450 for effective vision and dialogue fusion using two types of self-attention masks (e.g., 334).); [0026]-[0028])
generating a loss indicating a difference between an estimated response generated by applying the transformer architecture to at least the training image . . . training image . . . , and (Wang: [0042]-[0043]; [0056]; [0064]-[0067]; Examiner notes that this limitation was interpreted according to the 112(b) supra, but for compact prosecution, applied art accordingly.)
updating the parameters of the transformer architecture to reduce the loss; (Wang: [0042]-[0043]; [0056]; [0064]-[0067]; Examiner notes that this limitation was interpreted according to the 112(b) supra, but for compact prosecution, applied art accordingly.)
This known technique is applicable to the method of Lauka as they both share characteristics and capabilities, namely, they are directed to machine learning models.
One of ordinary skill in the art at the time of filing would have recognized that applying the known technique of Wang would have yielded predictable results and resulted in an improved method. It would have been recognized that applying the technique of Wang to the teachings of Lauka would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such transformer features into similar methods. Further, applying the machine learning model is configured as a transformer architecture, and parameters of the transformer architecture are finetuned by: obtaining at least a training example including a training image; generating a loss indicating a difference between an estimated response generated by applying the transformer architecture to at least the training image; updating the parameters of the transformer architecture to reduce the loss to the teachings of Lauka would have been recognized by those of ordinary skill in the art as resulting in an improved method that would achieve increased performance metrics (Wang: Para [0018]-[0019]).
As per claim 2, Lauka teaches further comprising:
updating availability information to indicate that identified target item is not available at a location and the list of potential replacement items is available at the location; and (Lauka: Figs. 3A-3C; Col. 5, Ln. 45 through Col. 6, Ln. 40 (At the management module 102, the picture can undergo a variety of processing. In some examples, the management module 102 can attempt to read one or more shelf bar codes 304 provided by the store on the shelf 108 to determine what products are still in stock. In other examples, the management module 102 can attempt to read one or more product bar codes 306 on the product packaging to determine what products are still in stock. In some examples, the management module 102 may compare the shelf bar codes 304 from the shelf 108 to the product bar codes 306 on the products to determine what products are out of stock. In other words, if the management mo]dule 102 detects seven shelf bar codes 304 on the shelf, but only six product bar codes 306 on the products, then the bar code that appears on the shelf 108, but not on the product 306 is the missing, or out of stock item.. . . . Based on the shelf bar codes 304, the management module 102 can consult an inventory list, database, or other reference to determine what products are in stock. The management module 102 can then send a message to the customer stating, for example, “The peach yogurt you requested is out of stock. We can offer you the following substitutions:”. The management module 102 can then provide a numbered list . . . ); Figs. 5A-5B; Col. 8, Ln.61 through Col. 9, Ln. 32 (At 508, the management module 102 can perform any necessary post-processing on the image. The management module 102 may extract bar codes 304, 306 from the image, for example, to determine what substitutions are available. In some examples, based on this information, the management module 102 may create a list of possible substitutions.))
providing, to a second user, an indication that the target item is not available at the location and a recommendation to obtain a potential replacement item in the list to the second user. (Lauka: Fig. 3B; Col. 5, Ln. 45 through Col. 6, Ln. 62 (The management module 102 can then send a message to the customer stating, for example, “The peach yogurt you requested is out of stock. We can offer you the following substitutions: The management module 102 can then provide a numbered list, . . .” As shown in FIG. 3B, in other examples, the management module 102 can use image processing software to apply additional processing to the picture from the handheld electronic device 106. The management module 102 can apply selection boxes 308, or other indicia, to the picture, for example, which can provide a box around the product, an approximate outline to the packaging for the product, or other indicia to enable the customer to graphically select a substitution. Each selection box 308 can then be associated with the product over which it appears. In this manner, the management module 102 can send the picture with designated areas for the customer to touch to make a choice for the substitution.); Figs. 5A-5B; Col. 8, Ln.61 through Col. 9, Ln. 32 (In some examples, based on this information, the management module 102 may create a list of possible substitutions. In some examples, the management module 102 may add selection boxes 308 to the image to enable the customer to select a substitution. At 510, the management module 102 can create and send a substitution message to the customer requesting a response. In some examples, the substitution message can include an explanation, such as “The item you requested is not available. Would you like to substitute any of the following items?” This can be followed by a numbered list of substitutions, for example, or the aforementioned image with or without selection boxes 308. In some examples, the message can also include a “No thank you” or “Refuse” button if the customer does not wish to make any substitution.); Fig. 7; Col. 13, Ln 37-64 (In some examples, as discussed above, the GUI 700 can include an image 704 of possible substitutions. . . . In some examples, for customers with touchscreens, for example, the image 704 can also contain the aforementioned selection boxes 308. The selection boxes 308 can correspond to various products that can be substituted for the missing item. For customers without touchscreens, the image 704 can also contain one or more identifiers 706 to enable the customer to respond textually to the request. The identifiers 706 can be numbers or letters, for example, to enable the customer to select the desired substitution. In some examples, the customer can simply text the appropriate number back to indicate their selection.))
As per claim 4, Lauka teaches wherein the message includes an explanation for each of the list of recommended replacement items. (Lauka: Fig. 3B; Col. 5, Ln. 45 through Col. 6, Ln. 62 (The management module 102 can then send a message to the customer stating, for example, “The peach yogurt you requested is out of stock. We can offer you the following substitutions:”. The management module 102 can then provide a numbered list, . . . In other examples, the management module 102 can use text recognition software to attempt to extract the brand, product type, flavor, size, and other information from the product packaging. This can be used instead of, or in concert with, the bar codes 304, 306 on the packaging and/or shelf 108. In some examples, this information can be extracted to provide the customer with a “natural English” list, listing the brand, product, flavor, size, and other information as it would generally appear on a shopping website, for example.); Figs. 5A-5B; Col. 8, Ln.61 through Col. 9, Ln. 32 (At 510, the management module 102 can create and send a substitution message to the customer requesting a response. In some examples, the substitution message can include an explanation, such as “The item you requested is not available. Would you like to substitute any of the following items?”); Fig. 7; Col. 13, Ln 37-64 (The introductory message 702 can include the name of the item that is not available and provide instruction to the customer about how to affect the desired substitution. In some examples, as discussed above, the GUI 700 can include an image 704 of possible substitutions. . . . For customers without touchscreens, the image 704 can also contain one or more identifiers 706 to enable the customer to respond textually to the request. The identifiers 706 can be numbers or letters, for example, to enable the customer to select the desired substitution. In some examples, the customer can simply text the appropriate number back to indicate their selection.))
As per claim 5, Lauka teaches further comprising:
pushing, via a user interface, the message to a customer user; and (Lauka: Fig. 3B; Col. 5, Ln. 45 through Col. 6, Ln. 62 (The management module 102 can then send a message to the customer stating, for example, “The peach yogurt you requested is out of stock. We can offer you the following substitutions: The management module 102 can then provide a numbered list, . . .”); Figs. 5A-5B; Col. 8, Ln.61 through Col. 9, Ln. 32 (In some examples, based on this information, the management module 102 may create a list of possible substitutions. In some examples, the management module 102 may add selection boxes 308 to the image to enable the customer to select a substitution. At 510, the management module 102 can create and send a substitution message to the customer requesting a response. In some examples, the substitution message can include an explanation, such as “The item you requested is not available. Would you like to substitute any of the following items?” This can be followed by a numbered list of substitutions, for example, or the aforementioned image with or without selection boxes 308. In some examples, the message can also include a “No thank you” or “Refuse” button if the customer does not wish to make any substitution.); Fig. 7; Col. 13, Ln 37-64 (In some examples, as discussed above, the GUI 700 can include an image 704 of possible substitutions.))
presenting each of the list of recommended replacement items in an interactable user interface element that allows the customer to select the respective recommended replacement item for replacing the target item. (Lauka: Fig. 3B; Col. 5, Ln. 45 through Col. 6, Ln. 62 (The management module 102 can then send a message to the customer stating, for example, “The peach yogurt you requested is out of stock. We can offer you the following substitutions: The management module 102 can then provide a numbered list, . . .” As shown in FIG. 3B, in other examples, the management module 102 can use image processing software to apply additional processing to the picture from the handheld electronic device 106. The management module 102 can apply selection boxes 308, or other indicia, to the picture, for example, which can provide a box around the product, an approximate outline to the packaging for the product, or other indicia to enable the customer to graphically select a substitution. Each selection box 308 can then be associated with the product over which it appears. In this manner, the management module 102 can send the picture with designated areas for the customer to touch to make a choice for the substitution.); Figs. 5A-5B; Col. 8, Ln.61 through Col. 9, Ln. 32 (In some examples, based on this information, the management module 102 may create a list of possible substitutions. In some examples, the management module 102 may add selection boxes 308 to the image to enable the customer to select a substitution. At 510, the management module 102 can create and send a substitution message to the customer requesting a response. In some examples, the substitution message can include an explanation, such as “The item you requested is not available. Would you like to substitute any of the following items?” This can be followed by a numbered list of substitutions, for example, or the aforementioned image with or without selection boxes 308. In some examples, the message can also include a “No thank you” or “Refuse” button if the customer does not wish to make any substitution.); Fig. 7; Col. 13, Ln 37-64 (In some examples, as discussed above, the GUI 700 can include an image 704 of possible substitutions. . . . In some examples, for customers with touchscreens, for example, the image 704 can also contain the aforementioned selection boxes 308. The selection boxes 308 can correspond to various products that can be substituted for the missing item. For customers without touchscreens, the image 704 can also contain one or more identifiers 706 to enable the customer to respond textually to the request. The identifiers 706 can be numbers or letters, for example, to enable the customer to select the desired substitution. In some examples, the customer can simply text the appropriate number back to indicate their selection.))
As per claim 6, Lauka teaches wherein generating the message comprises:
generating a prompt for input to a machine-learned language model, the prompt includes the list of recommended replacement items and an instruction to include the list of recommended replacement items in a written message based on a given message template; and receiving, from the machine-learned language model, a response comprising the message. (Lauka: Fig. 3B; Col. 5, Ln. 45 through Col. 6, Ln. 62 (The management module 102 can then send a message to the customer stating, for example, “The peach yogurt you requested is out of stock. We can offer you the following substitutions: The management module 102 can then provide a numbered list, . . .”); Figs. 5A-5B; Col. 8, Ln.61 through Col. 9, Ln. 32 (At 510, the management module 102 can create and send a substitution message to the customer requesting a response. In some examples, the substitution message can include an explanation, such as “The item you requested is not available. Would you like to substitute any of the following items?” This can be followed by a numbered list of substitutions, for example, or the aforementioned image with or without selection boxes 308.); Fig. 7; Col. 13, Ln 37-64 (The introductory message 702 can include the name of the item that is not available and provide instruction to the customer about how to affect the desired substitution. In some examples, as discussed above, the GUI 700 can include an image 704 of possible substitutions. . . . For customers without touchscreens, the image 704 can also contain one or more identifiers 706 to enable the customer to respond textually to the request. The identifiers 706 can be numbers or letters, for example, to enable the customer to select the desired substitution. In some examples, the customer can simply text the appropriate number back to indicate their selection.))
As per claim 7, Lauka teaches further comprising: automatically pushing, via a user interface, the message to a customer user. (Lauka: Fig. 3B; Col. 5, Ln. 45 through Col. 6, Ln. 62 (The management module 102 can then send a message to the customer stating, for example, “The peach yogurt you requested is out of stock. We can offer you the following substitutions: The management module 102 can then provide a numbered list, . . .”); Figs. 5A-5B; Col. 8, Ln.61 through Col. 9, Ln. 32 (In some examples, based on this information, the management module 102 may create a list of possible substitutions. In some examples, the management module 102 may add selection boxes 308 to the image to enable the customer to select a substitution. At 510, the management module 102 can create and send a substitution message to the customer requesting a response. In some examples, the substitution message can include an explanation, such as “The item you requested is not available. Would you like to substitute any of the following items?” This can be followed by a numbered list of substitutions, for example, or the aforementioned image with or without selection boxes 308. In some examples, the message can also include a “No thank you” or “Refuse” button if the customer does not wish to make any substitution.); Fig. 7; Col. 13, Ln 37-64 (In some examples, as discussed above, the GUI 700 can include an image 704 of possible substitutions.))
As per claims 8, 9, and 11-14, these claims are substantially similar to claims 1, 2, and 4-7, respectively, and are therefore rejected in the same manner as these claims, as set forth above.
As per claims 15, 16, and 18-20, these claims are substantially similar to claims 1, 2, 7, 5, and 6, respectively, and are therefore rejected in the same manner as these claims, as set forth above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Patil (US PGP 2023/0274410) – identifying missing/out-of-stock items
Fang (US PGP 2019/0251446) – generating a loss function and backpropagating an MLM to update parameters of the MLM based on loss function
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JENNIFER V LEE whose telephone number is (571)272-4778. The examiner can normally be reached Monday - Friday 9AM - 5PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JEFFREY A. SMITH can be reached at (571)272-6763. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
[/JENNIFER V LEE/Examiner, Art Unit 3688
/Jeffrey A. Smith/Supervisory Patent Examiner, Art Unit 3688
Prosecution Insights