DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 37 is objected to because of the following informalities: in line 1, the word “configure” should be “configured”. Appropriate correction is required.
Claim Rejections – 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 20, 24-28 and 31-36 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hay (WO 2016/004235 A2) in view of Zalewski et al (9,911,290 B1).
Regarding Claim 20, Hay teaches a system for conducting a purchase
transaction, as illustrated in figure 6, for example, comprising:
a first sensor, i.e., camera (1102) as mentioned at paragraph 122 and as illustrated in figure 11 and light strip (1200) with LED lights (1202, 1204) as mentioned at paragraph 131 and as illustrated in figure 12, and as mentioned at paragraph 194, first and second sentences, stating “[a]lternatively, a product image recognition software system (PIRS) may be used to detect the presence, placement or the removal of products on or off the trays” and “[s]uch PIRS will utilize a built-in camera, such as the camera 1910 illustrated in FIG. 19, which may be affixed to the bottom of each tray, the bottom of the AMP unit 1100, or in such other suitable place so as to be able to take images of the entirety of the items and/or compartments of a tray”; configured to sense removal and return of a first item from a first region, as mentioned at paragraphs 135 and 136;
a computer vision sensor, i.e., image recognition system incorporating cameras as mentioned at paragraph 14, or camera 1910 as illustrated in figure 19 and as mentioned at paragraph 194, and noting that Hay’s system is intended to repetitively sense and record placement or removal of any items from the trays as mentioned in paragraphs 194 and 195, for example; configured to sense removal or return of a second item from the first region, as mentioned at paragraphs 135, 136, for example, noting that the items;
a transaction detector, i.e., main control network facility/MCNF (36) as illustrated in figure 4, and as mentioned at paragraphs 92-94, for example, configured to determine an accuracy that the sensed removal or return of the first item by the first sensor (1102, 1200) and the sensed removal or return of the second item by the computer vision sensor (1910) correspond to a single event of removal or return of an item by a consumer, as mentioned at paragraphs 134, 135 and 136, for example, and to forward, to a machine learning tool, as taught by Zalewski, information associated with the sensed removal or return of the first item by the first sensor (1102, 1200) and information associated with the sensed removal or return of the second item by the computer vision sensor (1910) when the accuracy is less than an accuracy threshold, as mentioned at paragraphs 36, 44, 130 and Claims 30 and 34, for example;
the machine learning tool, as taught by Zalewski, configured to verify or correct the information associated with the sensed removal or return of the first item or the information associated with the sensed removal or return of the second item and to provide verified or corrected information to the transaction detector (36), as mentioned at paragraphs 36, 44, 130 and Claims 30 and 34, and as mentioned in paragraphs 25, 128, 148, 194 and 195, last two sentences stating “[t]he system will verify that the scanned item matches the product's assignment which is associated with the compartment that the item was returned to and issue the credit to the customer” and “[i]f the customer wishes to return a product or item after the transaction is closed, the return may require customer service assistance”, for example; and
an automated billing processor, i.e., local computing system (LCS) and main control board (90) as illustrated in figure 7, coupled to the transaction detector (36), configured to apply a purchase price of the item against an account of the consumer for the item based on the verified or corrected information, thereby completing the purchase transaction, as mentioned at paragraphs 128-130, 134-136, 162 and 164, for example. See paragraphs 10, 14, 22, 25, 36, 44, 89, 122, 127-131, 134-136, 148, 162, 164, 194 and 195 and Claims 30 and 34, for example, which state as follows.
[Para 1 0] The present invention is directed to an unattended, automated checkout store outlet and a method for granting access to and tracking purchases from the computer-controlled, automated store outlet. The computer-controlled, automated store outlet of the present invention generally comprises a plurality of bays for holding items for purchase. A normally locked door is associated with each bay, and provides access to items within the bay by customers that have been approved by a computer of the automated store outlet. A lock is associated with each bay door. The lock is normally locked when the bay door is closed, and selectively unlocked by a computer of the automated store outlet when a customer or technician is given approval to open the door and access the bay. An access unit, having multi-scanning functionality, is associated with each bay door which is configured to receive customer identification for determination of approval of access to the bay. At least one tray is disposed within the one or more bays, the tray having a plurality of compartments adapted to hold the items. A sensor system is configured to detect the removal of an item from the bay and to identify the removed item.
[Para 1 4] The sensor system may alternatively comprise a RFID (radio frequency identification device) tag attached to each item and a RFID reader or receiver configured to determine the location or proximity of the tag relative to the reader or receiver. Removal of items may also be detected by an image recognition system incorporating cameras which either detect the removal of an item or compare photographs of trays to detect removed items, such as by determining if the product is on or off the tray.
[Para 22] The automated store outlet may include a check-in station for customer identification, payment, and the like. The one or more bays of the automated store outlet are normally locked to prevent access to the items within the bay. Typically, items are arranged within compartments of each tray. A plurality of sensors are arranged to correspond with the items held by the trays. The tray and item placement in each bay may be configured from a remote location. Use of an array of fixed sensors may be used, or sensors may be used which are manually placed in relation to each tray compartment and /or item within the bay.
[Para 25] Customer payment information is also input into the check-in station of the automated store outlet, and the customer payment information is verified by the system. The customer payment information may comprise inputting information related to previously created customer automated store outlet dedicated account, credit or debit card account information, or
depositing cash into a cash receiving device of the automated store outlet. The customer may transfer funds to the customer's automated store dedicated account online or using the check-in station of the automated store outlet as well.
[Para 36] The accuracy of a transaction or identification or price of an item may be confirmed by scanning the machine-readable code on the removed item using a scanner of the automated store outlet, such as a scanner associated with the door access unit. If the customer returns the item into the bay, the item is removed from the electronic shopping cart of the purchaser. However, when the bay door is closed, it is automatically locked. The automated store outlet detects whether the doors of each bay is opened or closed at any given time, and tracks the opening of a door by a customer and assigns all items removed from that bay to that customer's shopping cart while the door is open.
[Para 44] The automated store outlet of the present invention may electronically detect the presence and identity of all items within a bay, and create an inventory report. The one or more bays of the automated store outlet may be restocked as needed. The automated store outlet and method of the present invention may create a fulfillment report and pick list and track restocking activity, including at least one of accuracy of restocking in comparison to an item pick list and time to start and complete restocking.
[Para 89] With reference now to FIG. 2, another ASO 1 0 is shown having a check-in station 1 2 , as described above, and a plurality of bays 22 within two modular units. In this case, however, it will be seen that the modular bays 22 are of a different configuration and size than that illustrated in FIG. 1 . Moreover, one of the bays 22 includes items related to a drink dispenser 34. The drink dispenser 34 typically comprises a coffeemaker, but can also provide hot water for making tea, cold water for making cold flavored beverages, etc. The coffee K-cups.sup.® , teabags, flavoring packets, cups, etc. would be accessed through the locked door 24 of the bay 22 , and after retrieving and purchasing these items, the customer could make the desired beverage using the beverage device 34. The beverage device 34 may or may not be behind a closed and locked door 24. Other type of hot or cold single cu p beverage dispensers 34 may be provided. Such dispensers are controlled by the ASO's computer system and dispense selected beverages only to qualified customers. In order to activate the beverage dispenser 34 the customer must have provided customer identification and payment information which was verified, such as by entering such information into an access unit 26 at the bay in which the beverage dispenser 34 is disposed, or into such an access unit which is electronically connected to the beverage dispensing device 34. At that time, the system of the ASO changes the machine mode from standby to active and the machine processes the customer's drink selections. The beverage dispenser device 34 reports the selection to the system that adds the sale to the customer's shopping cart. The beverage dispenser 34 may process one drink at a time, and then return to standby mode, until the next customer becomes verified and approved to use the dispenser 34. It is also contemplated by the present invention that the customer could be charged for a cup that is removed and /or a component which is required to make a beverage, such as a K-cup.sup.®, tea bag, flavoring packet, etc. Upon removal of one of these items from the bay tray 28, the beverage dispenser device 34 could be activated to make and/or dispense a beverage corresponding to the beverage cup, beverage component, or the like.
[Para 127] The door access unit 26 may also be able to scan identification codes of the item itself, such as UPC barcodes or the like. This can be used by the customer to verify the identity of a product and its price. This may also be used in order to return an item to its appropriate spot within the bay 22. Such scanning of items may also be used during inventory fulfillment. Scanning a product's UPC or other code may also display additional information about the product, such as ingredients or other characteristics.
[Para 1 28] In some cases, the system may require the customer who desires to return an item to a tray of a bay to scan the UPC or any other type of
identification code that is printed on the item, by using a nearby scanner, in order to facilitate the return and to remove that item from the customer's shopping cart. At that time, the customer will be required to open the door and scan the UPC or any other type of identification code that is printed on the item and then return the product or item to its proper location on the tray. The system will verify that the scanned item matches the product's assignment which is associated with the compartment that the item was returned to and issue the credit to the customer. If the customer wishes to return a product or item after the transaction is closed, the return may require customer service assistance.
[Para 1 29] The activity history of each sensor is saved on the local computer system (LCS) of the ASO. Therefore, at any time a specific sensor report can be generated showing all removals and/or replacements associated with such sensor as well as the information of the customer who removed the item or replaced the item and the date and time that this occurred.
[Para 1 30] If the system detects suspicious behavior, the system may use audio and /or visual messages requesting that the customer scan the one or more items that were removed from the trays by using the nearby scanner, in order to confirm the accuracy of the transaction. Such a request may be with a live customer service agent.
[Para 1 31 ] With reference now to FIG. 1 2 , light strips 1 200 may be disposed on a lower portion of the AMP in order to eliminate the items on the tray or within the bay below the AMP 1 1 00. Such light strip 1 200 may include various electronic components, including visible LED lights 1 202 , and non-visible LED lights 1 204, such as infrared LEDs. Such infrared LEDs 1 204 may be repeatedly pulse-illuminated in order to improve detection thereof by sensors. Such light strips 1 200 are also on the bottom of the trays.
[Para 1 34] With reference again to FIGS. 5 and 6, the customer that accesses and opens the door to that bay can remove any item from the bay's trays. The customer that opens the door is responsible for all items removed from that bay until the door is closed and locked. As items are removed from the bay, the product's name, description and price are added to the customer's virtual shopping cart 50, which may be displayed on the door access unit's display screen as well as on the AMP display screen. This is illustrated in steps 62 and 64 of FIG. 6. As the customer removes an item 70 from the tray 28, sensors detect the removal of the item 70 and the system is able to determine exactly the product name, description and price of that removed item 70 so as to add it to the virtual shopping cart 64. As described above, the virtual shopping cart may also be provided in real time on the user's mobile electronic device, such as a smart phone, display screen, as well as on a screen associated with the electronic unit scanner 26, the AMP 1 1 00 and/or the display screen 1 4 associated with the check-in station 1 2. This may be automatically displayed, or only displayed if requested by the customer.
[Para 1 35] Thus, as the customer removes an item 70 from the tray 28 of the bay 22 , the system detects, in real time, which product was removed and logs such product as a "pending sale item" in the virtual shopping cart of the identified customer that opened that door. The system detects the removal and identifies the products that get removed based on the preconfigured database information of each compartment of the tray. Detection of the removal of products can be accomplished by using any optical or electrical sensor system or any digital measurement system and /or by using any RFID or NFC identifying tag system as well as any other suitable sensor technology.
[Para 1 36] If the customer returns the item 70 to the tray 28, the system in real time removes that item from the customer's virtual shopping cart. In some cases, if the customer wishes to return the item to the tray before closing the door, the customer must return the item to the same location that the item was removed from. In some cases, if the customer does not return the item to the right location, credit may not be issued for such return and an alarm and /or voice announcement will request the customer to move the item to the correct location. The misplaced item will be logged and if not corrected by the customer, such compartment's product may be automatically updated by the software. If the software is unable to update the database, the software issues a notice to the maintenance crew indicating such misplaced product. In the use of some sensors, such as RFID tags or the like, the location of each product is less critical and such functions are modified accordingly based on the situation of each occurrence.
[Para 1 48] The check-in station 1 2 components of a display screen 1 4, camera 1 6, scanner 1 8 and/or card reader payment accepter 20 is shown communicating with the local computing system (LCS) 58. The LCS 58 communicates with the MCNF 36, NOC, customer service tech support unit 38, etc. However, as discussed above, the fundamental functions of the ASO may be performed locally at the ASO 1 0 to prevent the ASO from not being able to function and perform transactions in the event of network disruptions, which would otherwise occur if the system were based on a cloud or remote computing system. Moreover, in order to ensure the ability to service customers efficiently and to have the fastest function responses at all times and in order to avoid down time due to Internet interferences and outages, the ASO computing system's LCS is locally placed. Nevertheless, when the network is operating, the LCS is in constant communication with the MCNF, which is a cloud-based or remote computing system, to exchange data and to facilitate transactions, as well as to download or upload all settings, configurations, reports and /or other technical or administrational data. The LCS conducts multiple functionalities simultaneously in the operation of the ASO, which includes identification and qualification of customers, the operation of the locking mechanisms and detection of removed items, applying charges, surveillance video, as well as all other customer interactive media functions such as video and audio live conferencing. Therefore, the LCS 58 consists of at least one suitable high grade computing system to accommodate such functionalities locally, without delays, and the outages or interferences that may occur by using only a cloud or other remote computing systems.
[Para 1 62] Typically, LED lights are placed above each tray, such as attached to the bottom of the AMP 1 1 00 or the tray immediately above the tray in question. The LED lights can serve a dual purpose in illuminating the items within the tray below the LED lights as well as triggering the photosensors when the items are removed from the compartments of the tray. Removal of the item from the compartment of the tray exposes the photosensor, which sends a signal and is detected by the computer system, which has previously assigned an item to that compartment, and adds that item, and its cost, to the customer's virtual electronic shopping cart. In order to further prevent the possibility of reflected light or ambient light activating photosensors when the item has not been removed from a compartment, or has been placed or replaced into the compartment at a less than ideal position, LED light sources, such as the LED strip 1 200 illustrated in FIG. 1 2 may include both visible light LEDs 1 202 as well as non-visible light, such as typically infrared, lights 1 204. The photosensor would comprise an infrared photosensor or receiver and detect the infrared light emitted from the infrared LED 1 204 when an item is removed from a compartment of the tray. This may be done instead of, or in conjunction with, the polarized sheet of material above the photosensors. Most ambient light within enclosed lighted buildings is emitted in wave lengths ranging from 400NM to 700NM. Commonly used infrared LEDs emit light in wave lengths from 880NM to 1 040NM, which is typically out of the range of regular ambient light. By using the appropriate sensors that only detect infrared light, the detector will not give a false reading from ambient light but will be sensitive to infrared light from the infrared LED 1 204 shining from above the tray, and the sensor will achieve a proper detection.
[Para 1 64] However, infrared light from intense ambient sunlight can still pose a problem for item detection by washing out the infrared light emitted by the infrared LED. This problem can be mitigated by reading the analog level of the infrared light and adjusting the detection threshold based on the presence of infrared light from sunlight. A reference detector which is always exposed and not covered by any item monitors the level of ambient infrared light from sunlight and other sources, and the computer compensates by adjusting these thresholds for proper detection.
[Para 1 94] Alternatively, a product image recognition software system (PIRS) may be used to detect the presence, placement or the removal of products on or off the trays. Such PIRS will utilize a built-in camera, such as the camera 1 91 0 illustrated in FIG. 1 9, which may be affixed to the bottom of each tray, the bottom of the AMP unit 1 1 00, or in such other suitable place so as to be able to take images of the entirety of the items and/or compartments of a tray. In accordance with this PIRS system, the camera will produce photos of the tray below, showing the existing products on the tray. The PIRS will continuously analyze the photo images to determine the type and the quantity of the product items that exist on the tray. When changes occur, due to the removal or placement of product items, the system will report such changes to the LCS. If the information indicates that products have been removed, the LCS will add the products to the customer's virtual electronic shopping cart and apply charges. If the information indicates that the products have been added to the tray, the LCS will either credit the customer's shopping cart or update the inventory accordingly.
[Para 1 95] In addition to the PIRS requirements, the same camera, or a secondary camera, may be used to take photographic images and/or video in order to document the customer's hand movement while such customer removes or places items from or on the tray below. Such could be displayed on one of the electronic displays associated with the ASO, such as the AMP 1 1 00 associated with that bay. Alternatively, or additionally, such video or photographic images may be saved at least temporarily in order to be able to confirm, if necessary, that a customer removed or replaced an item from the tray if a billing dispute were to arise.
[Claim 30] The method of claim 1 , including the step of restocking one or more bays of the automated store outlet, and tracking restocking activity, including at least one of accuracy of restocking in comparison to an item pick list and time to complete restocking.
[C lai m 34] The method of claim 1 , including the step of confirming accuracy of a transaction by scanning a machine readable code on the removed item using a scanner of the automated store outlet.
Emphasis provided.
Regarding Claim 20, Hay further teaches at paragraph 19 (a camera may be associated with each bay for taking images of the customer while the customer is accessing each bay), paragraph 20 (a camera disposed above each tray of the bay may also be used as part of the sensor system wherein an image recognition system detects the presence or removal of an item from the tray by comparing photographic images), paragraph 45 (using cameras or other sensors), paragraph 61 (array of camera sensors) and paragraph 63 (multiple cameras used).
Hay further states at paragraph 22, at the last three sentences, that “[a] plurality of sensors are arranged to correspond with the items held by the trays”, that “[u]se of an array of fixed sensors may be used, or sensors may be used which are manually placed in relation to each tray compartment and/or item within the bay”.
Regarding Claim 20, Hay does not expressly teach wherein the first sensor and the computer vision sensor are different types of sensors.
Regarding Claim 20, Hay does not expressly teach, but Zalewski teaches wherein the first sensor and the computer vision sensor are different types of sensors, i.e., a first non-vision sensor, such as both a weight sensor and a computer vision sensor, i.e., image sensor/camera, for example, as illustrated at figure 51b and of which the weight sensor and computer vision sensor are understood to be different sensors, noting that col. 11, lines 19-43, and col. 127, lines 7-22 state as follows.
(91) The systems can further process historical interaction by the user, or historical interaction by other users, to determine possibilities that items will be purchased and/or taken from a shelf. As will be described below, a number of learning processes may be executed by local and/or cloud computing, in order to make assumptions and predictions. This information may be beneficial information to the shopper, e.g., by providing the user/shopper with real-time information regarding items. The information can be custom tailored for the specific user, e.g., by custom display screens placed proximate to items or goods, and in some embodiments, audio can be provided to the user, providing custom information regarding an item or good. In still other embodiments, the tracking of users and tracking of items can share sensors, e.g., sensor fusion, to make determinations that are validated as true. For example, multiple sensors can be used to determine if a user has actually picked up an item from shelf. Some sensors may be weight sensors, some may be image sensors, e.g., cameras, and some may be motion sensors. It should be understood that the sensors that can be fused can be many, and some embodiments may use weighting to determine in real-time which sensors are most important for a specific interaction and which sensors may provide the least reliable data. The weighting can be dynamic, as the way users interact with items on shelves may be different.
(671) For purposes of completeness, FIG. 51B illustrates various sensors. By way of example and without limitation to others, a camera 5116 is provided to capture image data and the camera(s) 5116 may be placed in various locations. Infrared sensors 5128 may also be integrated in and around the shelf in order to detect changes within the volume or area being tracked. Ultrasonic sensor 5120 may also be integrated in various locations in and around the shelf 5104. Motion sensors 5126 may also be placed at various locations, to detect motion of either the user's hand or hands interfacing with items or the items themselves moving on the shelf. Weight sensors 5112 can also be placed in various locations, so as to detect motion and movement of specific items on the shelf 5104, and differentiate between which items are being picked or interacted with when on the shelf or when placed back on the shelf.
Emphasis provided.
Regarding Claim 20, before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to have provided wherein the first sensor and the computer vision sensor are different types of sensors, as taught by Zalewski, in Hay’s method and system for conducting a purchase, for the purpose of obtaining more and varying types of data for more efficiently, accurately and independently determining the classification of transactions and events surrounding said transactions.
Regarding Claim 20, Hay does not expressly teach a first non-vision sensor and does not mention using an accuracy score.
Regarding Claim 20, Hay does not expressly teach, but Zalewski further teaches a first non-vision sensor , i.e., weight sensors (WS, 5112), Infrared sensors (IRS, 5128), Ultrasonic sensors (USS, 5120) and proximity sensors (5204, 5114), volume sensed/area sensing (5124) and motion sensors (5126), as illustrated in figure 51a and 51b, as mentioned at col. 11, lines 19-43 (includes sensor fusion), col. 12, lines 30-62, col. 15, lines 25-39 (weight sensors), col. 17, lines 51-61 (sensors can be either image or non-image type sensors), col. 26, lines 15-33 (density sensors), col. 124, lines 7-26, col. 125, line 59-col. 126, line 6, col. 126, lines 7-39 (weight sensors), col. 128, lines 29-49 (weight sensors) and col. 129, lines 29-40, for example, and mentions using an accuracy score, i.e., noting the mention of the use of confidence/accuracy scores at col. 18, lines 24-45 and col. 18, line 59-col. 19, line 5, and noting the mention of sensor fusion, at col. 11, lines 19-43, for example.
Regarding Claim 24, Hay does not expressly teach wherein the first sensor is a weight sensor.
Regarding Claim 24, Hay does not expressly teach, but Zalewski teaches wherein the first sensor is a weight sensor (WS, 5112), as mentioned above with respect to the rejection of Claim 20, above.
Regarding Claim 25, Hay teaches wherein the computer vision sensor, i.e., the image recognition system of Hay’s Claim 33, stating “wherein the removal of items is detected by an image recognition system”, Claim 74, stating “wherein the camera collects images of a customer's hand while removing items from the tray” Claim 75, stating wherein the sensor system comprises an image recognition system that detects the presence or removal of an item from the tray by comparing photographic images”, and the product image recognition system (PIRS) monitors and senses the presence, placement or removal of any item of the plurality of items on the tray, as mentioned at paragraphs 194 and 195, for example, and is further configured to identify the second item, as mentioned at paragraph 34, which states as follows.
[Para 34] An identification of the item removed from a tray or bay is displayed on the customer's shopping cart. This typically also includes the price for the item removed. The customer's shopping cart, including the identification of each removed item and its purchase price, may be displayed on a computer-controlled display unit associated with each bay of the automated store outlet. Upon accessing another bay, the customer's electronic shopping cart, including all items removed from all bays of the automated store outlet for that shopping event, is displayed on the display unit of that bay so as to travel with the customer from bay to bay of the automated store outlet to track all removed items for purchase from all bays of the automated store outlet until the customer's purchase transaction is closed. This enables the closing purchase transaction to be performed once as a single transaction for all items removed from all bays by the customer during a shopping event, which presents not only a convenience to the customer but also saves transaction fees.
Emphasis provided.
See also paragraphs 134-136, noting that paragraph 135 mentions “[t]he system detects the removal and identifies the products that get removed based on the preconfigured database information of each compartment of the tray”.
Emphasis provided.
Note also that paragraph 155 mentions as follows.
[Para 1 55] In one embodiment of the present invention, the ASO 1 0 includes a motion detector. The motion detector may be part of the identification scanner 26, or may be independent and placed at one or more locations throughout the ASO. The motion detector will detect motion of someone approaching or passing by the ASO. In this case, the detection of motion by the motion detector can be used to power on the lights of the ASO, such as the LED lights 88 within the various bays 22, turn on an initial prompt screen at the screen 1 4 of the check-in station 1 2, activate the advertising monitor 30, and the like. Moreover, the detection of motion by the motion detector can be used to activate the surveillance video cameras and systems, and actuate live remote viewing for security purposes or even customer service purposes.
[Para 1 93] The identity of the item 70 is determined. This may be due to its placement within a compartment on the tray 1 36, the fact that all items on a given tray 1 36 are the same, or the RFID tag 1 44 specifying the identity of the item 70, or by any other satisfactory means. In this manner, the identity of the item and its cost is provided to the customer in the virtual shopping cart 64.
Emphasis provided.
Regarding Claim 25, Zalewski also teaches wherein the computer vision sensor, i.e,. side cameras and overhead cameras in figure 1a, for example, is further configured to identify the second item, noting the system is designed to repeatedly track multiple items/products. See also col. 26, lines 40-55, for example.
Regarding Claim 26, Hay teaches wherein the computer vision sensor (1102, 1910) is further configured to identify the second item (70) from a finite list of possible items offered in the first region, i.e., interpreted as the shelf (28) as illustrated in figure 26, for example, as illustrated in figures 14-26 and noting the items are stored in a database, as mentioned at paragraph 135, for example. See also paragraphs 135, 136 and 150, noting the mention of identification of an item from the local database and preconfigured database information, for example
Regarding Claim 26, note also Zalewski’s weight sensors, proximity sensors, motion sensors, overhead and side cameras all detect the items on a particular shelf as a group, as to whether or not one or more are taken or replaced on the particular shelf, as illustrated in figure 1a, for example.
Regarding Claim 27, Hay teaches wherein the transaction detector, i.e., main control network facility/MCNF (36), as illustrated in figure 36 and as mentioned at paragraphs 212 and 213, for example, is configured to locally verify that the sensed removal or return of the first item and the sensed removal or return of the second item correspond to the single event, i.e., construed as a single transaction, as mentioned at paragraphs 9, 14, 41, 94, 128, 141, 142 and 148, for example.
Regarding Claim 27, Zalewski teaches wherein the transaction detector, i.e., Tracker (T) with sensors connected to the cloud, as illustrated in figure 61a, for example, is configured to locally verify that the sensed removal or return of the first item and the sensed removal or return of the second item correspond to the single event, i.e., as mentioned at col. 132, line 65-col. 134, line 14, for example, noting that a take and return of an item is considered to be a single event and is interpreted by classifier model (5875, 5900) and machine learning algorithm (5850, 5950) as illustrated in figures 58 and 59, for example.
Regarding Claim 28, Hays teaches wherein the transaction detector (36) is configured to remotely verify that the sensed removal or return of the first item and the sensed removal or return of the second item correspond to the single event, as mentioned at paragraph 39, which states that “[p]urchase data from the automated store outlet is also relayed to a central control center via a wired or wireless network” and at paragraph 92, last sentence, which states “[t]hus, a two-way communication network is established, using wired or wireless technologies, for the ASOs to relay information to the MCNF 36, NOC 40, customer service 38, or the inventory and sales and reporting system 42 , and receive communication back in the form of customer service, surveillance, remote operation, etc”. See also paragraphs 9, 14, 41, 128 and 148, for example.
Regarding Claim 28, Zalewski teaches wherein the transaction detector, i.e., Tracker (T) with sensors connected to the cloud, as illustrated in figure 61a, for example, is configured to remotely verify that the sensed removal or return of the first item and the sensed removal or return of the second item correspond to the single event, noting the system is entirely wireless based as a wireless coded communication (WCC) based system which is a type of internet of things (IOT) system, as mentioned at col. 6, lines 26-46, for example.
Regarding Claim 31, Hayes teaches wherein the machine learning tool is in the back office remote from the transaction tool, noting paragraph 22 mentions “tray and item placement in each bay may be configured from a remote location” in the second to last sentence. See also paragraphs 9, 14, 31, 41, 128 and 148, and noting the central control center or main control network facility (MCNF 36) mentioned in paragraph 93, for example. See also paragraph 148 mentioning a remote computing system and paragraph 207, second sentence, i.e., “[a]dditionally, such technique allows a local and remote reading of all trays in order to determine the existence of products and create a real time "on-the-shelf" inventory report”.
Regarding Claim 31, Hayes does not expressly teach wherein the machine learning tool is in the back office remote from the transaction tool.
Regarding Claim 31, Hayes does not expressly teach, but Zalewski teaches wherein the machine learning tool is in the back office, i.e., interpreted as the cloud, remote from the transaction tool (WCC 100), as mentioned at col. 69, line 61-col. 70, line 2 and col. 62, lines 30-46, for example.
Regarding Claim 32, Hays teaches wherein the automated billing processor is configured to delay applying the purchase price of the item against the account of the consumer until after remotely verifying that the sensed removal or return of the first item and the sensed removal or return of the second item correspond to the single event, as mentioned at paragraphs 27, 34 and 38, which state as follows.
[Para 27] After the customer has been identified, the customer's payment information verified, and the customer qualified, the customer is then allowed to shop at the automated store outlet. This requires unlocking a door to provide access to a bay. Typically, this requires that the customer provide identification information to an electronic unit associated with a lock at each bay door. This may be by means of an electronic unit which performs multiple scanning and reading and display functions simultaneously, including a motion detector, an electronic display screen, a keypad, two-way audio visual communication means, RFID reader, electronic code image reader, and a biometric scanner. The customer identification is verified before unlocking the door. Once the door is unlocked and opened, the system tracks the customer's activity at that bay.
[Para 34] An identification of the item removed from a tray or bay is displayed on the customer's shopping cart. This typically also includes the price for the item removed. The customer's shopping cart, including the identification of each removed item and its purchase price, may be displayed on a computer-controlled display unit associated with each bay of the automated store outlet. Upon accessing another bay, the customer's electronic shopping cart, including all items removed from all bays of the automated store outlet for that shopping event, is displayed on the display unit of that bay so as to travel with the customer from bay to bay of the automated store outlet to track all removed items for purchase from all bays of the automated store outlet until the customer's purchase transaction is closed. This enables the closing purchase transaction to be performed once as a single transaction for all items removed from all bays by the customer during a shopping event, which presents not only a convenience to the customer but also saves transaction fees.
[Para 38] After a customer has completed his or her shopping session, the purchase transaction is closed and the customer is charged for the one or more removed items. Closing the purchase transaction can comprise the customer closing the purchase transaction using the check-in station at the automated store outlet. The customer may request display of the removed items and their purchase prices by viewing the shopping cart at a computer-controlled display unit of the check-in station. Alternatively, the system will automatically close the transaction and charge the customer for the removed item after a predetermined period of customer inactivity at the automated store outlet.
Emphasis provided.
See also figure 10 and abstract and paragraphs 10, 81, 95, 96, 108-110, 127 and 146, for example. Note also that paragraphs 24 and 25 mention verification of identification and payment information before purchasing an item.
Regarding Claim 33, Hays teaches confirmation of a customer’s removal of items from the ASO, as mentioned at paragraphs 128, 129, 136, 143, 154, 155, 187, 194, 207 and 155, for example.
Regarding Claim 33, Hays does not expressly teach, but Zalewski teaches
wherein the machine learning tool is configured to correct either the sensed removal or return of the first item or the sensed removal or return of the second item in response to the verifying, as mentioned at col. 17, lines 26-40, col. 27, lines 36-45, col.. 134, lines 3-57.
Regarding Claim 34, Hays teaches wherein the automated billing processor, i.e., local computing system (LCS) and main control board (90) as illustrated in figure 7, is configured to adjust applying the purchase price of the item against the account of the consumer for purchase of the item in response to the verifying, as mentioned in paragraphs 25, 92, 128-130, 134-136, 146, 162 and 164, for example.
Regarding Claim 34, Zalewski teaches wherein the automated billing processor is configured to adjust applying the purchase price of the item against the account of the consumer for purchase of the item in response to the verifying, as mentioned in col. 27, lines 36-45, for example.
Regarding Claim 35, see the rejection of Claim 33, above, for example.
Regarding Claim 36, see Hays as mentioned at paragraphs 14, 36, 44,128, 130, 144, 148, 162 and 164, for example.
Regarding Claim 36, Hays does not expressly teach wherein the machine learning tool is configured to determine an accuracy of the verifying.
Regarding Claim 36, Hays does not expressly teach, but Zalewski teaches wherein the machine learning tool is configured to determine an accuracy of the verifying, as mentioned at col. 125, lines 48-58, col. 132, lines 29-39, col. 134, lines 3-14, for example.
Claim(s) 29, 30 and 37 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hay (WO 2016/004235 A2) in view of Zalewski et al (9,911,290 B1) and further in view of Ahmed et al (US 10,963,659 B1).
Regarding Claim(s) 29, 30 and 37, Hay teaches the system as described above.
Regarding Claim 29, Hays teaches wherein the transaction detector, i.e., construed as automated store outlet (ASO), as mentioned at paragraphs 79 or 80, or central control center or main control network facility (MCNF) (36), as mentioned at paragraphs 92 and 93, is configured to transmit, to a backend office, information associated with the sensed removal or return of the first item from the first region, and to transmit, to a backend office, information associated with the sensed removal or return of the second item from the first region, noting that all of the information processed at the (ASO) is transmitted to the (MCNF), which can be construed to be a backend office, and which “exchange(s) data and information with the (ASO), and that the (MCNF) is a comprises one or more servers in a cloud environment. Note that the term “backend” is merely construed as a backend node or remote centrally located node of a network system.
See also paragraphs 9, 14, 31, 41, 128 and 148, and noting the central control center or main control network facility (MCNF 36) mentioned in paragraph 93, for example. See also paragraph 148 mentioning a remote computing system and paragraph 207, second sentence, i.e., “[a]dditionally, such technique allows a local and remote reading of all trays in order to determine the existence of products and create a real time "on-the-shelf" inventory report”.
Regarding Claim 29, Hays does not expressly teach a backend office, construed as a backed device/server for the facility.
Regarding Claim 29, Hays does not expressly teach, but Ahmed teaches a backend office, construed as a backed device/server for the facility, as mentioned at col. 6, line 35-col. 7, line 20, col. 9, lines 4-30, col. 1, line 41-col. 12, line 45 and col. 19, lines 32-46, all of which state the following.
(23) After the user has moved throughout the materials handling facility and selected the items they desire to purchase or otherwise take from the facility, the user may end the shopping session in various ways. For instance, the user may return a tote to a tote corral or other storage location, provide input to the tote indicating an end of the shopping session (e.g., utterance, utilize a user interface element on a touch display, etc.), or simply remove bags or other item carriers from the tote and leave the facility. After the user has ended their shopping session, the list of item identifiers in the virtual item listing may be uploaded to one or more remote servers that manage user accounts for users of the facility. The servers may charge the appropriate user account for the listing of the items in the virtual item listing that the user took from the facility.
(24) Although some of the techniques described below are performed locally on the tote, in other examples, some or all of the techniques may be performed by one or more backend devices or servers associated with the facility. For instance, the sensor data and/or image data may be collected at the tote and sent over network(s) to backend devices at the facility, or server devices located remote from the facility, to be processed remotely. However, in some instances it may be advantageous for at least some of the processing to be performed on the tote to reduce latency in identifying items placed in the tote. For instance, it may be advantageous to have low latency when requesting that a user provide feedback to help identify an item recently placed in the tote, rather than the user continuing their shopping session and being asked later about an item. Further, while various techniques described below are with reference to purchasing items in a retail facility, the techniques are generally applicable to any materials handling facility in which a user may place items in a tote. For example, although the techniques described herein are primarily with reference to identifying items placed in a tote by a user for the purpose of identifying a user account to charge for items selected from a materials handling facility, the techniques are equally applicable to any industry in which user recognition may be helpful. For instance, the idem-identifying tote may be implemented for distribution centers where employees collect items from various locations to be shipped through the mail system.
(25) In some instances, the techniques may be performed with traditional totes, carts, or baskets that do not include any sensors or electronics at all. For instance, the facilities themselves may include sensors such as weight sensors, cameras, etc., that send sensor data and image data to backend devices to perform the techniques described herein. The backend devices may analyze the sensor data and/or image data using the cascading algorithms as described herein, and maintain a virtual item listing for the users in the facility.
(35) In some instances, the facility 102 may include various sensors configured to interact with the tote 104 and/or generate sensor data in addition to, or as an alternative to, the sensor data generated by the tote 104. For instance, the facility 102 may include one or more access points 114 configured to emit signals (e.g. WiFi, Bluetooth Beacons, etc.) that are detected by the tote 104 for use in determining the location of the tote 104 in the facility 102. Further, the facility 102 may include one or more camera(s) 116 configured to generate image data that depicts the facility 102, such as the events 110 occurring in the facility, items 106 involved in the events, and/or users 108 that selected the items 106. Further, the facility 102 may include one or more weight sensors 118 disposed in the inventory locations 112 to detect generate sensor data indicative of the removal of, or return of, items 106 from the inventory locations 112. The facility 102 may include one or more backend devices or servers, and/or may transmit the data to the servers 120 at a remote location, for analysis using the techniques described herein. That is, the techniques described herein may be performed entirely by the tote 104 using sensor data and image data generated by sensors of the tote, entirely by backend devices/servers of the facility 102 (or remote servers) using sensor data and image data generated by the tote, entirely by backend devices/servers of the facility 102 (or remote servers) using sensor data and image data generated by sensors in the facility, and/or any combination thereof.
(45) Once the user 108 has finished their shopping session, the user 108 may end the shopping session in various ways. For instance, the user 108 may return the tote 104 to the tote corral or other storage location, provide input to the tote 104 indicating an end of the shopping session (e.g., utterance, utilize a user interface element on a touch display, etc.), or simply remove item bags or other item carriers from the tote 104 and leave the facility 102. After the user 108 has ended their shopping session, the virtual listing 144 of item IDs 136 in the virtual item listing may be uploaded to one or more remote servers 120, over one or more networks 122, that manage user accounts for users 108 of the facility 102. The server(s) 120 may charge the appropriate user account for the items 106 in the virtual listing 144 that the user 108 took from the facility 102. For instance, the server(s) 120 may be configured to determine or generate information indicative of a cost of the items 106 picked by the user 108, including the appropriate tax. Additionally, the server(s) 120 may store payment information (e.g., credit card information, bank account information, etc.) for each user account. In this way, when the user 108 finished their shopping session and the tote 104 sends the listing 144 of item IDs 136 over the network(s) 122 to the server(s) 120, the server(s) 120 may be configured to determine a cost or price for all of the listed item IDs 136, and charge the user 108 via their payment information for the items 106 selected during their shopping session. In this way, the user 108 need not go through steps of a traditional check-out experience (e.g., waiting in line for a cashier, scanning items with the cashier, paying for items at the cashier, etc.).
(46) The network(s) 122 may include private networks such as an institutional or personal intranet, public networks such as the Internet, or a combination thereof. The network(s) 122 may utilize wired technologies (e.g., wires, fiber optic cable, and so forth), wireless technologies (e.g., radio frequency, infrared, acoustic, optical, and so forth), or other connection technologies. The network(s) 122 is representative of any type of communication network, including one or more of data networks or voice networks. The network(s) 122 may be implemented using wired infrastructure (e.g., copper cable, fiber optic cable, and so forth), a wireless infrastructure (e.g., cellular, microwave, satellite, etc.), or other connection technologies.
(47) The tote 104 may include communication interface(s) such as devices configured to couple to personal area networks (PANs), wired and wireless local area networks (LANs), wired and wireless wide area networks (WANs), and so forth. For example, the communication interfaces may include devices compatible with Ethernet, Wi-Fi™, and so forth. In some examples, the communication interface(s) may encode the data prior to sending over the network(s) 122 according to the type of protocol or standard being used. As noted above, in some examples, the servers 120 may perform some or all of the operations described below as being performed by the tote 104. While the servers 120 are illustrated as being in a location outside of the facility 102, in other implementations, at least a portion of the servers 122 may be located at the facility 102.
(48) FIG. 2 is a block diagram 200 of an example materials handling facility 102 that includes item-identifying totes 104, facility and tote sensors, and an inventory-management system 124 configured to generate output regarding events occurring in the tote 104 using sensor data. However, the following description is merely one illustrative example of an industry and environment in which the techniques described herein may be utilized. It is to be further appreciated that the inventory-management system may reside within the facility 102, remote from the facility 102, or a combination thereof. For example, in some instances components described with reference to the inventory-management system may reside on computing devices at the facility, at the tote 104, and at the servers 120.
(86) The systems 400 may also include one or more communication interfaces 230. The communication interfaces 230 are configured to provide communications between devices, such as the tote 104, the server(s) 120, sensors, interface devices, routers, and so forth. The communication interfaces 230 may include devices configured to couple to personal area networks (PANs), wired and wireless local area networks (LANs), wired and wireless wide area networks (WANs), and so forth. For example, the communication interfaces 230 may include devices compatible with Ethernet, Wi-Fi™, Bluetooth®, and so forth. The systems 400 may also include one or more busses or other internal communications hardware or software that allow for the transfer of data between the various modules and components.
Emphasis provided.
Regarding Claim 29, before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to have provided a backend office, as taught by Ahmed, in Hay’s method and system for conducting a purchase, for the purpose of properly balancing the distribution of tasks thoughout the system between local elements and a central backend server.
Regarding Claim 30, Hays teaches wherein the transaction detector, i.e., MCNF (36), is configured to transmit a video of the removal or return of the second item from the first region, as mentioned at paragraphs 94, 154 and 195, for example. See also paragraphs 9, 14, 31, 41, 128 and 148, and noting the central control center or main control network facility (MCNF 36) mentioned in paragraph 93, for example. See also paragraph 148 mentioning a remote computing system and paragraph 207, second sentence, i.e., “[a]dditionally, such technique allows a local and remote reading of all trays in order to determine the existence of products and create a real time "on-the-shelf" inventory report”.
Regarding Claim 30, Zalewski teaches wherein the transaction detector, i.e., wireless coded communication device (WCC) (100) and the “cloud compute” as illustrated in figure 1b, for example, is configured to transmit a video of the removal or return of the second item from the first region, as mentioned at col. 72, lines 31-44, col. 75, lines 51-61, col. 80, lines 9-14, for example and noting the motion tracking of users by side cameras as well as the overhead cameras that view removal and replacement of items on store shelves, as illustrated in figure 1a, for example.
Regarding Claim 37, Hayes does not expressly teach wherein the machine learning tool is configure(d) to remotely verify that the sensed removal or return of the first item and the sensed removal or return of the second item correspond to the single event when the accuracy of the verifying is below an accuracy threshold.
Regarding Claim 37, Hayes does not expressly teach, but Zalewski teaches wherein the machine learning tool is configure(d) to remotely verify that the sensed removal or return of the first item and the sensed removal or return of the second item correspond to the single event when the accuracy of the verifying is below an accuracy threshold, i.e., noting the mention of confidence scores in col. 18, lines 24-32, for example.
Regarding Claim 37, Hayes does not expressly teach, but Ahmed teaches wherein the machine learning tool is configure(d) to remotely verify that the sensed removal or return of the first item and the sensed removal or return of the second item correspond to the single event when the accuracy of the verifying is below an accuracy threshold, i.e., noting the mention of confidence scores in col. 11, lines 20-29, col. 14, line 25-col. 15, line 45, col. 18, lines 40-58, col. 22, line 59-col. 23, line 36, col. 26, line 52-col. 27, line , for example.
Regarding Claim 37, before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to have provided a wherein the machine learning tool is configure(d) to remotely verify that the sensed removal or return of the first item and the sensed removal or return of the second item correspond to the single event when the accuracy of the verifying is below an accuracy threshold, as taught by Ahmed, in Hay’s method and system for the purpose of increasing the accuracy of sensed data using confidence level/scoring to more align the data with reality of the physical world.
Response to Arguments
Applicant’s arguments with respect to Claim(s) 20 and 24-37 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant is encouraged to contact the Examiner should there be any questions about this rejection or in an endeavor to explore potential amendments or potential allowable subject matter.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Shmueli ‘360 is cited as teaching a backend system of a retail store (200) as mentioned at paragraphs 76, 119, 149, and teaches use of confidence levels at paragraph 154 and checks if the items list is accurate in real time, as mentioned in paragraph 225, for example.
Kumar ‘391 is cited as teaching determining the accuracy of data when items are removed/returned as mentioned at paragraphs 17, 36, 37, 51, 53, 59, 63, 64, 67-71, 75 and 78, for example.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEFFREY ALAN SHAPIRO whose telephone number is (571)272-6943. The examiner can normally be reached Monday-Friday generally between 8:30AM and 6:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anita Y Coupe can be reached on 571-270-3614. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JEFFREY A SHAPIRO/Primary Examiner, Art Unit 3619
September 17, 2026