DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Adato et al., U.S. 20120398199 A1 and further in view of Gadou et al., FR 3071341 A1.
1. A method comprising:
accessing an image of an inventory structure captured by a robotic system while navigating through a store, (see Adato, ¶ 147 “image data captured by robotic devices with cameras traversing in aisle”), the inventory structure defining a set of shelf spaces, (see Adato, ¶ 273 “a map showing the location of the products associated with the offer or provide directions to the user to travel to the location of the products associated with the offer (e.g., “proceed to end of aisle ten, then turn right,” etc.”);
detecting a first shelf space, in the set of shelf spaces, within the inventory structure in the image, (see Adato, ¶ 115 “the system may determine the existence of a plurality of distinct products displayed on a store shelf. By detecting the plurality of products, the system may acquire different details relative to the plurality of products (e.g., how many products on a store shelf are associated with a same product type”);
detecting a first set of objects occupying the first shelf space in the image, (see Adato, ¶ 115);
extracting a first set of features from a first region of the image representing the first set of objects, (see Adato, “the identification may be made based at least in part on visual characteristics of the product (e.g., size, shape, logo, text, color, etc.). The unique identifier may include any codes that may be used to search a catalog, such as a series of digits, letters, symbols, or any combinations of digits, letters, and symbols. Consistent with the present disclosure, the terms “determining a type of a product” and “determining a product type” may also be used interchangeably in this disclosure with reference to the term “identifying a product.””);
based on the first set of features, identifying a first set of product units of a first product type occupying the first shelf space, (see Adato, ¶ 115);
accessing a target product type assigned to the first shelf space by a graphical representation of the store, (see Adato, abstract “The processor accesses a planogram describing a desired placement of products and identifies, based on the planogram, a discrepancy between the determined placement of products”);
detecting a first object in a first lateral position in the image, the first lateral position below the first shelf space, (see Adato, ¶ 56 “FIG. 8A is a schematic illustration of an example configuration for detecting products and empty spaces on a store shelf, consistent with the present disclosure”);
extracting a second set of features from a second region of the image representing the first object, (see Adato, ¶ 116 “Embodiments of the present disclosure further include determining at least one characteristic of the product for determining the type of the product. As used herein, the term “characteristic of the product” refers to one or more visually discernable features attributed to the product”);
based on the second set of features, identifying a first electronic shelf label depicted in the second region of the image, (see Adato, ¶ 18 “a system may process images captured in a retail store and automatically determining relationships between products and shelf labels”);
extracting a first set of product data from the second region of the image, the first set of product data corresponding to the target product type and rendered within an electronic display of the first electronic shelf label, (see Adato, ¶ 178 “one processor contained in second housing 504A may use any suitable image analysis technique (for example, object recognition, object detection, image segmentation, feature extraction, optical character recognition (OCR), object-based image analysis, shape region techniques, edge detection techniques, pixel-based detection, artificial neural networks, convolutional neural networks, etc.) to identify objects in the plurality of images”); and
in response to the first product type differing from the target product type assigned to the first shelf space and advertised by the first electronic shelf label, (see Adato, ¶ 412-13 “an employee may receive an instruction to move a misplaced product”):
accessing a second set of product data from a product database defined for the store, the second set of product data corresponding to the first product type of the first set of product units occupying the first shelf space, (see Adato, ¶ 409 “as shown in FIG. 28A, an identified discrepancy may be a misplaced box of pasta 2804a on a shelf labelled “canned goods”);
generating a first notification comprising the second set of product data, (see Adato, ¶ 410 “a discrepancy may be based on a product's placement relative to a label, e.g., label 2806. For example, image analysis component 2702 may identify a pasta box 2810a on a bottom shelf of unit 2802 and may also identify a shelf 2812 labelled “pasta.” Based on the distance between the pasta box 2810a and the “pasta” shelf label 2806 being greater than an allowable threshold, discrepancy determination component 2704 may determine that the box 2810a is misplaced, e.g., is on a non-compliant shelf”); and
transmitting the first notification to the first electronic shelf label for rendering within the electronic display of the first electronic shelf label, (see Adato, ¶ 242 “modifying a label associated with the store shelf”).
Although Adato discloses shelf spaces, Adato fails to disclose slots. However, Gadou discloses slots in conjunction with electronic shelf labels, (see Gadou, “Then, in step 950, the central server 2 consults the article image database A. A previously recorded image that is associated in the article image database with the identifier predetermined article Pijk / is recovered. This image corresponds to a front view provided for the determined article identifier Pijk-Said image is generally the image of the front view which is used in a visual representation of the realogram database P, to represent the associations between product placement fields and article IDs. In step 960, a similarity ratio is calculated between the actual frontal view visible on the acquired image I and the predicted front view retrieved from the article image database A. This step 960 is intended to determine if the actual front view of the article presented in the product slot inside the shelves corresponds to the intended front view. In particular, it is possible to detect a situation in which the article presented in the product location corresponds to a reference other than that provided in the realogram.”)(emphasis added).
It would have been obvious to one of ordinary skill in the art at the time of the invention to combine the features of Adato and Gadou because of the following findings of fact:
F. Known Work in One Field of Endeavor May Prompt Variations of It for Use in Either the Same Field or a Different One Based on Design Incentives or Other Market Forces if the Variations Are Predictable to One of Ordinary Skill in the Art
(1) the scope and content of the prior art in the same field of endeavor as that of the applicant’s invention include a similar or analogous device (method, or product), i.e. electronic shelf labels;
(2) there were design incentives or market forces which would have prompted adaptation of the known device (method, or product), (see Adato, ¶ 30, 181 “The at least one processor may then use the product capacity for the portion of the shelving unit dedicated to the particular product type and the accessed data to generate at least one suggestion for improving store execution and provide the at least one suggestion for improving store execution”; “embodiments of the present disclosure may also be applied to end caps or other displays; bins, shelves, or other organizers associated with a refrigerator or freezer units; or any other display structure associated with a retail environment”);
(3) the differences between the claimed invention and the prior art were encompassed in known variations or in a principle known in the prior art;
(4) one of ordinary skill in the art, in view of the identified design incentives or other market forces, could have implemented the claimed variation of the prior art, and the claimed variation would have been predictable to one of ordinary skill in the art.
This motivation and rationale to combine these references is applied to all claims below herein by reference.
2. The method of Claim 1: further comprising, in response to the first product type differing from the target product type assigned to the first shelf space, accessing an inventory database defining stock conditions of a set of product types within the store; and wherein generating the first notification comprising the second set of product data comprises, in response to the inventory database defining an out-of-stock condition for the target product type, generating the first notification comprising the second set of product data and indicating the out-of-stock condition for the target product type, (see Adato, ¶ 33 “restocking event”).
3. The method of Claim 2, further comprising, in response to the inventory database defining the out-of-stock condition for the target product type: accessing a product-ordering platform employed by the store; within the product-ordering platform, identifying an order of product units of the target product type scheduled for delivery at a future time; generating a second notification comprising a prompt to re-stock the first shelf space with product units of the first product type for a time window expiring at the future time; and transmitting the second notification to a computing device accessed by a store associate affiliated with the store, (See Adato, ¶ 230 “For example, an online customer may order a favorite cookie brand without knowing that the cookie brand is out-of-stock. Consistent with some embodiments, system 100 may use image data acquired by capturing devices 125 to provide the online customer with a near real-time display of the retail store and a list of the actual products on the shelf based on near real-time data. In one embodiment, server 135 may select images without occlusions in the field of view (e.g., without other customers, carts, etc.) for the near real-time display. In one embodiment, server 135 may blur or erase depictions of customers and other people from the near real-time display. As used herein, the term “near real-time display,” in the context of this disclosure, refers to image data captured in retail store 105 that was obtained by system 100 within a predefined period of time (such as less than a second, less than a minute, less than about 30 minutes, less than an hour, less than 3 hours, or less than 12 hours) from the time the image data was captured”).
4. The method of Claim 1: further comprising, in response to the first product type differing from the target product type assigned to the first shelf space: accessing an inventory database defining stock conditions of a set of product types within the store; and in response to the inventory database defining an in-stock condition for the target product type, accessing a first location of product units of the target product type associated with the in-stock condition;" wherein generating the first notification comprising the second set of product data comprises, in response to the inventory database defining the in-stock condition for the target product type, generating the first notification comprising the second set of product data and indicating the first location associated with stock of the target product type; and " wherein transmitting the first notification to the first electronic shelf label for rendering within the electronic display of the first electronic shelf label comprises transmitting the first notification to the first electronic shelf label for rendering within the electronic display of the first electronic shelf label for a first time window, (see Adato, ¶ 230).
5. The method of Claim 4, further comprising, in response to the inventory database defining the in-stock condition for the target product type:" generating a second notification comprising a prompt to stock the first shelf space with product units of the target product type from the first location prior to expiration of the first time window; and " transmitting the second notification to a store associate affiliated with the store, 9se Adato, ¶ 228 “Typical in-store execution may involve dealing with ongoing service events, such as a cleaning event, a restocking event, a rearrangement event, and more. In some embodiments, system 100 may improve in-store execution by providing adequate visibility to ensure that the right products are located at preferred locations on the shelf. For example, using near real-time data (e.g., captured images of store shelves) server 135 may generate customized online reports. Store managers and regional managers, as well as other stakeholders, may access custom dashboards and online reports to see how in-store conditions (such as, planogram compliance, promotion compliance, price compliance, etc.) are affecting sales”).
6. The method of Claim 1, wherein transmitting the first notification to the first electronic shelf label for rendering within the electronic display of the first electronic shelf label comprises transmitting the first notification to the first electronic shelf label for rendering within the electronic display of the first electronic shelf label, the electronic display configured to, responsive to receipt of the first notification: render the first set of product data within a third region of the electronic shelf label, the first set of product data corresponding to the target product type assigned to the first shelf space by the graphical representation of the store; and render the second set of product data within a fourth region of the electronic shelf label, the second set of product data corresponding to the first product type of the first set of product units occupying the first shelf space, (see Adato, ¶ 229 “GUI 1140 may include a first display area 1142 for showing a display of a video stream captured by output device 145C (e.g., a real-time display or a near real-time display) with augmented markings indicting a status of planogram compliance for each product (e.g., correct place, misplaced, not in planogram, empty, and so forth). GUI 1140 may also include a second display area 1144 for showing a summary of the planogram compliance for all the products identified in the video stream captured by output device 145C. Consistent with the present disclosure, server 135 may generate within minutes actionable tasks to improve store execution. These tasks may help employees of retail store 105 to quickly address situations that can negatively impact revenue and customer experience in the retail store 105”).
7. The method of Claim 1, wherein transmitting the first notification to the first electronic shelf label for rendering within the electronic display of the first electronic shelf label comprises transmitting the first notification to the first electronic shelf label for rendering within the electronic display of the first electronic shelf label, the electronic display configured to, responsive to receipt of the first notification: render the first set of product data within the electronic shelf label during a first time window; and render the second set of product data within the electronic shelf label during a second time window offset the first time window, the second set of product data corresponding to the first product type of the first set of product units occupying the first shelf space, (see Adato, ¶ 229).
8. The method of Claim 1: wherein transmitting the first notification to the first electronic shelf label comprises transmitting the first notification to the first electronic shelf label at a first time; and further comprising: accessing a second image of the inventory structure captured by the robotic system at a second time succeeding the first time; detecting the first shelf space within the inventory structure in the second image; detecting a second object in a second lateral position in the second image, the second lateral position below the first shelf space; extracting a third set of features from a third region of the second image representing the second object; based on the third set of features, identifying the first electronic shelf label depicted in the third region of the second image; extracting a set of product data from the third region of the second image; and in response to the set of product data corresponding to the second set of product data corresponding to the first product type, verifying updating of the first electronic shelf label according to the first notification, (see Adato, ¶ 229).
9. The method of Claim 8, further comprising, in response to the set of product data differing from the second set of product data: " flagging the first electronic shelf label for further investigation; " interpreting an error event associated with the first electronic shelf label; " generating a prompt to investigate the first electronic shelf label; and " transmitting the prompt to a computing device accessed by a store associate affiliated with the store, (see Adato, ¶ 229 “server 135 may cause real-time automated alerts when products are out of shelf (or near out of shelf), when pricing is inaccurate, when intended promotions are absent, and/or when there are issues with planogram compliance”)..
10. The method of Claim 1: " wherein transmitting the first notification to the first electronic shelf label comprises transmitting the first notification to the first electronic shelf label at a first time; and " further comprising: accessing a second image of the inventory structure captured by the robotic system at a second time succeeding the first time; detecting the first shelf space within the inventory structure in the second image; detecting a second set of objects occupying the first shelf space in the second image; extracting a third set of features from a first region of the second image representing the second set of objects; based on the third set of features, identifying a second set of product units of a second product type occupying the first shelf space at the second time; detecting a second object in a second lateral position in the second image, the second lateral position below the first shelf space; extracting a fourth set of features from a second region of the second image representing the second object; based on the fourth set of features, identifying the first electronic shelf label depicted in the second region of the second image; and in response to the second product type corresponding to the target product type assigned to the first shelf space and in response to the second product type differing from the first product type advertised by the first electronic shelf label: generating a second notification comprising the first set of product data corresponding to the target product type; and transmitting the second notification to the first electronic shelf label for rendering within the electronic display of the first electronic shelf label, (see Adato, ¶ 229 “GUI 1120 may also include a second display area 1124 for showing a map of the specific retail store 105 with real-time indications of selected in-store execution events that require attention, and a third display area 1126 for showing a list of the selected in-store execution events that require attention. In another example, shown in FIG. 11D, GUI 1130 may include a first display area 1132 for showing a list of notifications or text messages indicating selected in-store execution events that require attention”).
11. The method of Claim 1, wherein generating the first notification comprising the second set of product data comprises generating the first notification comprising: " the second set of product data; " a first command to render the second set of product data within a first subregion of the electronic display of the first electronic shelf label; and " a second command to render a first visual indicator within a second subregion of the electronic display, the first visual indicator configured to indicate replacement of product units of the target product type with product units of the first product in the first shelf space, (see Adato, ¶ 228 “Typical in-store execution may involve dealing with ongoing service events, such as a cleaning event, a restocking event, a rearrangement event, and more. In some embodiments, system 100 may improve in-store execution by providing adequate visibility to ensure that the right products are located at preferred locations on the shelf. For example, using near real-time data (e.g., captured images of store shelves) server 135 may generate customized online reports. Store managers and regional managers, as well as other stakeholders, may access custom dashboards and online reports to see how in-store conditions (such as, planogram compliance, promotion compliance, price compliance, etc.) are affecting sales”).
12. The method of Claim 1, wherein accessing the target product type assigned to the first shelf space by the graphical representation of the store comprises accessing the target product type assigned to the first shelf space by the graphical representation of the store, the graphical representation comprising a planogram generated for the store, (see Adato, ¶ 228).
As per claims 13-20, these claims contain the same or similar features as claims 1-12 rejected above, and therefore the remaining claims are rejected on the same basis and rationale as those rejected above herein by reference.
Response to Arguments
Applicant's arguments filed 9/3/26 have been fully considered but they are not persuasive. Applicant's arguments regarding claims 13, 18, and all dependent claims, fails to comply with 37 CFR 1.111(b) because they amount to a general allegation that the claims define a patentable invention without specifically pointing out how the language of the claims patentably distinguishes them from the references.
Applicant's argues that “a method for detecting a misplaced product in a slot and transmitting updated product data to an electronic shelf” is fundamentally different from
“a method for: detecting that a first product type advertised by an electronic shelf label differs from the product type assigned to a slot” and/or “a method for: detecting absence of product units of an assigned product type in a slot… and updating the electronic shelf label” and therefore not taught by Adato et al., U.S. 20120398199 A1 in view of Gadou et al., FR 3071341 A1.
lf, a recitation of the intended use of the claimed invention must result in a structural difference between the claimed invention and the prior art in order to patentably distinguish the claimed invention from the prior art. If the prior art structure is capable of performing the intended use, then it meets the claim.
Applicant also argues that Adato does not teach slots. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). In this case Gadou teaches slots, (see Gadou, “This step 960 is intended to determine if the actual front view of the article presented in the product slot inside the shelves corresponds to the intended front view”).
In response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In this case, using “product capacity for the portion of the shelving unit dedicated to the particular product type and the accessed data to generate at least one suggestion for improving store execution and provide the at least generate at least one suggestion for improving store execution” is cited as providing, teaching, suggestion, or motivation to combine the references.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RUSSELL S GLASS whose telephone number is (571)272-7285. The examiner can normally be reached M-F, 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, FLORIAN ZEENDER can be reached at 571-272-6790. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RUSSELL S GLASS/Primary Examiner, Art Unit 3627
