DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Whitelaw et al (US 2025/0046161 A1) in view of Carranza et al (US 2019/0043207 A1), further in view of Landers, Jr., et al (US 2020/0402130 A1), further in view of Zheng et al (US 2021/0182922 A1) and further in view of Jiang et al (US 10,438,277 B1).
Regarding Claim 1, Whitelaw teaches a point-of-sale system comprising:
a first checkout station, i.e., checkout terminal (100), as illustrated at figures 1 and 8, at a first location, noting the mention of “a central server that exchanges data with a plurality of checkout terminals” as mentioned at paragraph 69, last sentence;
a plurality of first edge cameras, i.e., cameras (1110, 1111, 1120, 1121, 1130, 1131, 1140, 1141, 1250, 1251, 1260, 1261, 1270, 1280, 1290, 1210), as mentioned at paragraphs 96 and 97 and as illustrated in figures 11 and 12, associated with the first checkout station (100), each of the plurality of first edge cameras (1110, 1111, 1120, 1121, 1130, 1131, 1140, 1141, 1250, 1251, 1260, 1261, 1270, 1280, 1290, 1210) having a first primary viewing area, i.e., fields of view (1111, 1121, 1131, 1141, 1251, 1261), within the first checkout station (100), and a first peripheral viewing area outside the first checkout station;
a second checkout station (100), at the first location, as mentioned at paragraph 69, last sentence, which states “[i]n some embodiments, the steps of FIG. 8 may be at least partially executed by a central server that exchanges data with a plurality of checkout terminals”;
a plurality of second edge cameras (1110, 1111, 1120, 1121, 1130, 1131, 1140, 1141, 1250, 1251, 1260, 1261, 1270, 1280, 1290, 1210) associated with the second checkout station (100), as mentioned at paragraphs 96 and 97, each of the plurality of second edge cameras (1110, 1111, 1120, 1121, 1130, 1131, 1140, 1141, 1250, 1251, 1260, 1261, 1270, 1280, 1290, 1210) having a second primary viewing area, i.e., fields of view (1111, 1121, 1131, 1141, 1251, 1261), within the second checkout station (100) and a second peripheral viewing area outside the second checkout station,
the second peripheral viewing area of at least one of the second edge cameras being within the first checkout station;
a vision mesh network having a plurality of nodes in communication with each other, at least one of the plurality of first edge cameras and at least one of the plurality of second edge cameras being nodes within the plurality of nodes on the vision mesh network and in communication with each other; and
one of the plurality of first edge cameras receiving and processing information about the first checkout station from the at least one second camera.
Regarding Claim 1, Whitelaw does not expressly teach
a first peripheral viewing area outside the first checkout station;
a second peripheral viewing area outside the second checkout station,
the second peripheral viewing area of at least one of the second edge cameras being within the first checkout station;
a vision mesh network having a plurality of nodes in communication with each other, at least one of the plurality of first edge cameras and at least one of the plurality of second edge cameras being nodes within the plurality of nodes on the vision mesh network and in communication with each other; and
wherein one of the plurality of first edge cameras receives, from the at least one of the plurality of second edge cameras having a second peripheral viewing area within the first checkout station, image information corresponding to a target area within the first checkout station, combines the received image information with image information generated by the one of the plurality of first edge cameras, the received image information representing an alternate view of the target area within the first checkout station, and processes the combined image information to determine an event occurring at the first checkout station.
Regarding Claim 1, Whitelaw does not expressly teach, but Landers ‘130 teaches
a first peripheral viewing area, as seen in figure 2, showing several areas throughout the store (200), peripheral/outside the first checkout station (205), as illustrated in figure 2 and as mentioned at paragraphs 21-23 and 25-26, for example;
a second peripheral viewing area, as seen in figure 2, showing several areas throughout the store (200), peripheral/outside the second checkout station (205), as illustrated in figure 2 and as mentioned at paragraphs 21-23 and 25-26, for example,
the second peripheral viewing area of at least one of the second edge cameras, i.e., any of cameras (202, 425), being within the first checkout station (120, 205, 300, 400). See also paragraphs 28, 30, 31, 37 and 40, along with paragraphs 21-23 and 25-26 for example, which states as follows.
[0021] FIG. 2 illustrates a portion of an exemplary store 200 depicting shelves, POS terminals and an exit to the store, according to aspects of the present disclosure. Store 200 includes shelving units 203 with shelves 210 and items 215 that are available for selection, purchase, etc. Multiple shelving units 203 may be arranged in the store 200 to form aisles through which customers may navigate.
[0022] The store 200 includes a plurality of sensor modules 202 disposed in the ceiling 201. A POS system of the store 200 may use information gathered by sensor modules in determining items being purchased by a customer. For example, a POS system may receive imagery of a customer placing a box of corn flakes in the customer's basket and store a record that the customer picked up the box of corn flakes for use (e.g., as a reference) when the customer is checking out. Each sensor module 202 may include one or more types of sensors, such as visual sensors (e.g., cameras), audio sensors (e.g., microphones), and motion sensors. Sensor modules 202 may also include actuating devices for orienting the sensors. Sensor modules or individual sensors may generally be disposed at any suitable location within the store 200. Some non-limiting examples of alternative locations include below, within, or above the floor 230, within other structural components of the store 200 such as a shelving unit 203 or walls. In some embodiments, sensors may be disposed on, within, or near product display areas such as shelving unit 203. The sensors may also be oriented toward an expected location of a customer interaction with items, to provide data about the interaction, such as determining the customer's actions.
[0023] Store 200 also includes a number of POS terminals (e.g., kiosks) 205. Each POS terminal 205 may include computing devices or portions of computing systems, and may include various I/O devices, such as visual displays, audio speakers, cameras, microphones, key pads, and touchscreens for interacting with the customer. According to aspects of the disclosure, a POS terminal 205 may identify items a customer is purchasing, for example, by determining the items from images of the items.
[0025] In some embodiments, the shelving unit 203 may include attached and/or embedded visual sensors or other sensor devices or I/O devices. The sensors or devices may communicate with networked computing devices within the store 200. A POS system may use information gathered by sensors on a shelving unit to determine items being purchased by a customer. For example, the front portions 220 of shelves 210 may include video sensors oriented outward from the shelving unit 203 to capture customer interactions with items 215 on the shelving unit 203, and the data from the video sensors may be provided to a POS system for use in determining items in the customer's basket when the customer is checking out.
[0026] A POS system of the store 200 may utilize sensor modules 202 to build a transaction for customer 240. The POS system may recognize various items 215 picked up and placed in a bag or basket by the customer 240. The POS system may also recognize the customer 240, for example, by recognizing the customer's face or a mobile computing device 245 carried by the customer 240. The POS may associate each item 215 picked up by the customer 240 with the customer 240 to build a transaction for the customer 240.
[0027] FIG. 3 illustrates an exemplary POS terminal 300, according to one embodiment of the present disclosure. POS terminal 300 is generally similar in structure and function to POS terminal 205. POS terminal 300 includes a base portion 312, one or more vertical portions 311, 313, a support member 314 for supporting a shopping basket 343, and a credit card reader 322. POS terminal 300 includes a camera 320 oriented for identifying store items in a shopping basket 343. POS terminal 300 may include a touchscreen or display 318 and camera 317 that are generally oriented toward customers using the POS terminal, e.g., customer 350. Vertical portion 311 may also include a plurality of indicator lights 315.
[0028] In some embodiments of the present disclosure, the camera 320 may be oriented such that it can view both items in the shopping basket 343 and a customer 350 using the POS terminal 300. The camera 320 may be oriented to view items in the shopping basket 343 and a customer using the POS terminal 300 by placing the camera 320 high on the POS terminal 300, using a motor to move the camera 320 to change the viewpoint of the camera 320, or supplying the camera 320 with a wide-angle lens.
[0029] The support member 314 may have markings indicating where the basket 343 should be positioned during operation of the POS terminal 300. Similarly, the display 318 may present messages to assist a customer in positioning the basket 343. Indicator lights 315 may also be used to indicate proper or improper basket positioning on support member 314. The support member 314 may also include a scale for determining the weight of the basket 343. The weight of the basket may be used in identifying items within the basket 343.
[0030] According to aspects of the present disclosure, a POS system may simultaneously identify items 362, 364 in the basket 343 and the customer 350. The POS system may identify the items 362, 364 based on one or more images of the items captured by the cameras 317, 320. The POS system may identify the items based on barcodes, quick response (QR) codes, reflected light, colors, sizes, weight ranges, packaging dimensions, packaging shapes, and graphical design of the items. For example, a POS system may capture an image of basket 343 using camera 320. The POS system may determine that item 364 is a bag of chips based on a bar-code in the image, and item 362 is an apple, based on its color in the image and its weight, which is determined by the scale in support member 314.
[0031] The POS system may identify the customer 350 based on an image of the customer captured by cameras 317 and/or 320 (e.g., by use of facial recognition software), based on a mobile computing device carried by the customer (e.g., based on an app running on the customer's smartphone), based on the customer's voice (e.g., using voice recognition software), and/or based on a gesture made by the customer (e.g., captured using a touch-capable implementation of the display 318 at the POS terminal 300). For example, a POS system may receive an image of customer 350 from camera 317 on POS terminal 300, and the POS system may use facial recognition software with the image to determine that customer 350 is Susan Jones.
[0037] FIG. 4 illustrates an exemplary checkout area 400, according to one embodiment of the present disclosure. Checkout area 400 may be associated with or be a part of store 100 or store 200. Checkout area 400 includes two exemplary checkout lanes 405a and 405b, but other numbers of checkout lanes are included in the scope of the disclosure.
[0038] Each checkout lane may include a plurality of dividers 410L, 410R that bound each checkout lane. While the checkout lanes are shown with dividers, the dividers are optional, and checkout lanes may be bounded by markings on the floor or other means. As shown, the dividers 410L, 410R are attached to framing in the ceiling, but alternative embodiments may have one or more dividers attached to the floor or free-standing.
[0039] One or more of the dividers 410L, 410R may include input/output devices for customer interaction, such as a display 415. Other input/output devices such as audio speakers, a touchscreen, a keypad, a microphone, etc. may also be included.
[0040] The dividers may include cameras 420L, 420R for capturing images of items 430 included in shopping cart 440. The cameras 420L, 420R may be oriented toward an expected position of the shopping cart 440, such as relative to a segment of lane lines 412. The images may be analyzed based on properties of the items 430, as well as labeling such as barcodes 435. A separate camera 425 may be included in a checkout lane 405 for capturing additional images of the items 430 and/or images of the customer 401. A POS system may analyze the images of the items 430 to determine the items being purchased by the customer 401. For example, a POS system may determine the items being purchased by scanning one or more images and reading a bar-code on an item, reading a quick reference (QR) code from an item, reading a label from an item, and/or looking up the color, size, or shape of the item in a database of store items. A POS system may also combine the previously mentioned techniques and use incomplete information from a technique, either alone or in combination with another technique. For example, a POS system may read a partial bar-code and determine a color of an item from one or more images, then determine the item by looking up the partial bar-code in a database and determining a group of items matching the partial bar-code, and then using the determined color to select one item from the group. In a second example, a POS system may read a partial bar-code from an item, look up the partial bar-code in a store inventory database, and determine that only item in the inventory database matches the partial bar-code. A POS system may also use information provided from other types of sensors to determine items being purchased. For example, a POS system may include radio-frequency identification (RFID) scanners and determine items being purchased by scanning RFID chips included in the items.
Emphasis provided.
Regarding Claim 1, before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to have provided a first peripheral viewing area outside the first checkout station and a second peripheral viewing area, outside the second checkout station, the second peripheral viewing area of at least one of the second edge cameras being within the first checkout station;
as taught by Landers ’130, in Whitelaw’s checkout terminal system for the purpose of increasing security by adding various cameras in peripheral areas of the store where the point of sale checkout system.
Regarding Claim 1, Whitelaw does not expressly teach, but Carranza teaches
a vision mesh network (856, 920), as illustrated in figures 8 and 9, and as mentioned at paragraphs 91, 95, 102 for example, having a plurality of nodes, i.e., smart cameras (220a, 220b, 220c) and surveillance orchestration device (210), as illustrated in figure 2 and as mentioned in paragraphs 49 and 50, cameras (C1, C2, C3, C4), as mentioned at paragraphs 58 and 59, internet-of-things (IOT) devices (804), as illustrated in figure 8 and as mentioned at paragraphs 90-93, 96 and 97, in communication with each other, i.e., via mesh transceiver (1162), noting that the smart cameras (220a, 220b, 220c) are considered IoT processing devices (1150) as illustrated in figure 11, for each device (1150), at least one of the plurality of first edge cameras (220a, 220b, 220c) and at least one of the plurality of second edge cameras (220a, 220b, 220c) being nodes, noting the transceiver (1162) in each camera device, within the plurality of nodes on the vision mesh network (856), as mentioned in first sentence of paragraphs 91, 95 and 99, as well as at illustrated in figures 8 and 9, and in communication with each other, as mentioned at paragraph 90, second sentence, i.e., “a number of IoT devices 804 may communicate with a gateway 854, and with each other through the gateway 854” and paragraph 102, first sentence, i.e., “[c]ommunications from any IoT device 902 may be passed along a convenient path (e.g., a most convenient path) between any of the IoT devices 902 to reach the gateways 904”; and
one of the plurality of first edge cameras (220a-220c, 902, 1150) receiving and processing information, i.e., via processor (1152) as illustrated in figure 11, about the first checkout station, as taught by Whitelaw, from the at least one second camera (220a-220c, 902, 1150).
Regarding Claim 1, before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to have provided a vision mesh network having a plurality of nodes in communication with each other for each device, at least one of the plurality of first edge cameras and at least one of the plurality of second edge cameras being nodes, and
one of the plurality of first edge cameras receiving, about the first checkout station, from the at least one second camera,
as taught by Carranza, in Whitelaw’s checkout terminal system for the purpose of increasing security by adding various cameras in peripheral areas of the store where the point of sale checkout system, which are connected with each other and pass video information between them for processing at particular cameras.
Regarding Claim 1, Whitelaw does not expressly teach, but Zheng teaches
wherein one of the plurality of first edge cameras, i.e., sensors in the form of cameras (211-213, 221, 231, 241, 430, 440), as mentioned at paragraphs 41, 49 and 68 and as illustrated in figures 2, 3a, 3b and 4, , noting that paragraph 38 mentions in the last sentence that “a computing device equipped with sensors and sufficient computing resources may be both a sensor device, and an edge tier node or a root tier node as it is capable of performing various data processing task”, and noting the mention of sensor fusion of data from various cameras and sensors as mentioned in paragraph 54,
receives, from the at least one of the plurality of second edge cameras (211-213, 221, 231, 241, 430, 440) having a second peripheral viewing area within the first checkout station, i.e., noting the mention of cameras in the checkout area in paragraph 49, second sentence, i.e., “[o]ne or more sensors (e.g., single camera (SC), multi-camera (MC), cameras at the checkout area of the retail store) may monitor people, such as customers and employees, as they move around and through the store”, image information corresponding to a target area within the first checkout station, combines the received image information with image information generated by the one of the plurality of first edge cameras (211-213, 221, 231, 241, 430, 440), the received image information representing an alternate view of the target area within the first checkout station, and processes the combined image information to determine an event occurring at the first checkout station, as mentioned at paragraphs 41, 48, 49, 68. 73, 80, 88, 99 and 107.
[0041] In some embodiments, the sensors (e.g., the one or more sensors 211, 212, 213, 221, 231, and/or 241) placed in the automated-checkout store 100 may comprise one or more image sensors (e.g., RGB cameras, IR cameras, depth cameras), one or more weight sensors, one or more force sensors, one or more pressure sensors, one or more vibration sensors, one or more proximity sensors, one or more resistance-based film sensors, one or more capacitive sensors, other suitable sensors, or any combination thereof. The sensors may be used to collect signals associated with one or more product items and one or more persons. In some embodiments, the sensors may be powered through one or more network cables using power over ethernet (“POE”). The sensors may also be powered using one or more other suitable methods or devices.
[0048] FIG. 3A illustrates an example system for tracking persons and interactions between persons and product items in a store. In some embodiments, a computer system in an automated-checkout store may have the structure illustrated by FIG. 3A and may comprise one or more of the components or layers shown in FIG. 3A. In some embodiments, the computer system 300 may comprise a plurality of computers that can be classified into tiers (e.g., groups), and the computers of each tier may be used to implement the functionalities of one or more of the layers shown in FIG. 3A. In some embodiments, the system may be configured to receive many types of input from many different sensing sources. These inputs may include image and/or video signals from an array of cameras mounted in various parts of an automated-checkout store such as the ceiling, cashier level, shelf level, signals from touch capacitive sensors on shelves, signals from weight sensors on shelfs, signals from force sensors on shelfs, and/or vibration sensors on shelfs. The system may use these signals to identify a customer and monitor the product items that the customer takes out of the store, and thus automatically checkout the product items. The system may also use these signals to calculate an inventory level of each of a plurality of retail products within the store.
[0049] In some embodiments, the system may comprise a sensing layer 310 that include a plurality of sensors that provide the system with different types of data. One or more sensors (e.g., single camera (SC), multi-camera (MC), cameras at the checkout area of the retail store) may monitor people, such as customers and employees, as they move around and through the store. One or more sensors (e.g., capacitive sensors, force sensors, vibration sensors, weight sensors, location aware pressure sensors) may track objects such as retail products on a plurality of shelves. The sensing layer 310 may be implemented, for example, as the one or more sensors 211, 212, 213, 221, 231, and/or 241. The sensors may be controlled by the computer system. The computer system may deactivate one or more of the sensors due to inactivity to save energy. The sensors may be switched back to an active state based on, for example, signals collected by a neighboring sensor.
[0054] In some embodiments, the automated-checkout store may be associated with a computer system for processing the data collected by various sensors. The computer system may comprise one or more suitable electronic devices. In some embodiments, the computer system may use sensor fusion to aggregate data received from multiple sensors. In doing so, the computer system may combine sensor data or data derived from disparate sources using error-resistance algorithms such that the resulting information has less uncertainty than would be possible when these sources were used individually. Using sensor fusion, the computer system may derive additional data than that sensed by each sensor (e.g., calculation of depth information by combining two-dimensional images from two cameras at slightly different viewpoints). In some embodiments, the computer system may perform direct fusion, which may comprise the fusion of sensor data from a set of heterogeneous or homogeneous sensors, soft sensors, and history values of sensor data. The computer system may also perform indirect fusion that uses information sources like a priori knowledge about the environment and human input. The computer system may use one or more methods or algorithms such as the central limit theorem, Kalman filter, Bayesian networks, Dempster-Shafer theory, or convolutional neural network. In some embodiments, sensor fusion for an automated-checkout store may be performed at a centralized computer system. Alternatively or additionally, sensor fusion may be performed by one or more localized computer sub-systems, whose outputs may later be aggregated. In some embodiments, using sensor fusion, the computer system may calculate a distribution for a particular value to be determined, including an average, a range, and one or more deviations. The computer system may also assign trustworthiness values for different factors contributing and weigh the factors based on the trustworthiness values.
[0068] FIG. 4 illustrates an example sensor system 400 in an automated-checkout store. Illustrated by FIG. 4 may be an inner space of an automated-checkout store. FIG. 4 may include an entrance, exit, or fixture 410, one or more shelves 420, one or more tracking cameras 430, and one or more depth cameras 440. In some embodiments, no depth cameras 440 may be necessary for or used in the system. As an example, the one or more tracking cameras 430 may comprise RGB cameras or IR cameras. The one or more tracking cameras 430 and/or the one or more depth cameras 440 may be implemented as part of, controlled, or coordinated by a computer system, which may be implemented, for example, as system 300, as shown and described in FIGS. 3A-3B. The sensor system 400 may comprise a hierarchical, modularized sensor system in which the one or more tracking cameras 430 and/or the one or more depth cameras 440 may be divided into groups. Each of the groups may comprise one or more image sensors configured to capture and/or monitor a section of the automated-checkout store. Data collected by each group may be separately processed and analyzed by a computer sub-system comprising one or more processors (e.g., CPUs, GPUs). Then, the results of various computer sub-systems may be combined and derive results covering an entire space. Such a modularized design saves processing power using separate and parallel processing of data from the image sensors. With such a modularized sensor system, a complexity of a tracking a certain area may grow linearly with the size of the area, instead of squared or higher growth. Image sensors arranged in such a modularized manner may provide high scalability for large shopping spaces. The computer system for such a sensor system 400 may use a unique identifier for each person in the area monitored by the sensor system 400 across different groups of sensors. The computer system may determine locations of persons using a machine-learning model based on data collected by the image sensors 430 and 440. The sensor system 400 may determine and/or output the locations where the persons are present, for example, as bounding boxes.
[0070] At the entrance, exit, or fixture 410 of the store, the computer system may identify an account associated with a person who enters the store. In some embodiments, the computer system may determine that a person enters the store based on signals from one or more of the sensors such as, for example, a tracking camera 430 pointing at the door of the store or a weight sensor below the floor near the door. In some embodiments, the computer system may identify an account associated with the person based on information provided by the person. For example, the person may be required to swipe a credit card or provide login information before being granted access to the store. The computer system may identify the account associated with the person based on information associated with the credit card or the login. In some embodiments, the computer system may determine an account based on a determined identity of the user. The computer system may determine the identity of the person based on data received from one or more of the sensors, such as tracking cameras 430. The received data may be fed into a machine-learning model, such as a deep learning network, a feature matching model, and/or a combination of one or more neural networks, for determining an identity of users. The model may comprise features corresponding to characteristics such as facial, body, or hair features, clothing of the person, measurements of the person such as height or weight, a distinctive gait or walking style of the person, and/or other suitable features specified by human or generated by machine-learning algorithms, or any combination thereof. The machine-learning model may have been trained by real-world data or data generated by simulations. For example, the training data may be generated by modifying real-world images by changing a background or environment in which a person is located to be a plurality of options. The training data may be tested and verified. The collected data may be compared with data stored by or otherwise accessible to the computer system that correlate accounts or identities of persons with characteristics of their appearance. In this manner, the account associated with the person may be identified without input from the person.
[0073] In some embodiments, the computer system may be configured to utilize machine-learning models to keep track of inventory of product items and people interacting with the product items. The computer system may receive data from a plurality of sensors and apply the data to machine-learning models. The different sources may include digital images or videos from an array of cameras (e.g., tracking cameras 430) mounted in various parts of a retail store, such as the ceiling, cashier level, shelf level cameras (e.g., tracking cameras 430), signals from weight sensors, force sensors and vibration sensors on shelves 420. The different types of data may be used to train the machine-learning model, and allow the machine-learning model to learn and recognize actions performed by customers in the retail environment and label these actions for future recognition. In addition to currently-received sensor data, the computer system may also consider previously-received data in determining the interaction. In particular, the computer system may calculate a conditional probability of a particular activity of a person based on prior detected activities of the person.
[0075] In some embodiments, a plurality of image sensors may be used to predict the 3D location of body joints. Red-green-blue (RGB) cameras 430, as well as infrared (IR) image sensors (e.g., cameras 430) may be used to capture two-dimensional (2D) images. The 2D images may be analyzed to obtain 3D locations. Alternatively or additionally, one or more depth cameras 440 may be used to determine the depth of a body joint into an area containing a product item. The one or more depth cameras 440 may be used jointly with the other cameras (e.g., tracking cameras 430) to estimate the 3D locations of a person's body joints. The 3D hand location of the consumer may be estimated using digital images or video.
[0080] In some embodiments, the computer system may determine, based on data received from one or more of the sensors, a movement path within the automated-checkout store associated with the person. In some embodiments, the computer system may identify one or more features (e.g., appearance features of face, body, or cloths, visual patterns of walking, vibration frequency signature when walking) associated with the person and use the features to track the movement of the person. In some embodiments, the space within the automated-checkout store may be divided into a plurality of regions, each monitored with a set of image sensors (e.g., four cameras such as tracking cameras 430 at the corners of a rectangular region) and/or one or more other sensors. The person's movement path in each region may be separately tracked by the sensors corresponding to the region. The tracking results for different regions may then be integrated by concatenating overlapping areas and optimizing from the perspective of the entire store perspective. Based on the data collected by the sensors, the computer system may construct a model describing the 3D movement of the person. In some embodiments, one or more piezoelectric sensors or piezo film sensors may be used in conjunction with image sensors to determine a person's movement path within the automated-checkout store. The piezo film sensors may be embedded in the floor of the automated-checkout store (e.g., in one or more floor tiles). The piezo film sensors may generate electrical signals in response to steps of a person. The computer system may determine a trajectory of the person based on signals from a plurality of piezo film sensors located in a plurality of locations within the automated-checkout store.
[0088] FIG. 5A illustrates an example system for tracking objects in an automated-checkout store. The system may include a plurality of sensors 500 that collect data associated with persons shopping in the store, and a server 510 that receives, processes and analyzes the collected data. The sensing layer 310 and real time layer 320 illustrated in FIG. 3A may be implemented using one or more components of the system illustrated in FIG. 5A. Specifically, the sensing layer 310 may be implemented using the sensors 500, and the real time layer 320 may be implemented using the server 510. The sensors 500 may include image sensors (e.g., RGB cameras, IR cameras, depth cameras, video cameras), weight sensors, pressure sensors, etc. The data collection by the sensors 500 may be continuous, event-triggered, periodical, or in another suitable manner. The data collected by the sensors 500 may include information associated with a person in the store, such as appearance, location, hand position, hand motion, etc., as well as information associated with product items that are in the proximity of the person's hand, such as shape, color, weight, bar code, etc. In some embodiments, the information associated with the product items may be collected only when a triggering event is detected. For example, the information may be collected when a weight change is detected on the shelf or a container holding the product items (the weight change may indicate that an item is taken), when some image sensors detect that the person is in proximity to a shelf or one or more product items, or when some image sensors detect the person's hand motion such as reaching, grabbing, placing, holding, etc.
[0099] FIG. 6A illustrates an example method for tracking objects in an automated-checkout store. The method may start from step 610, where a plurality of sensors 500 may send sensor data to a first computing device 520. The plurality of sensors 500 may include one or more image sensors (e.g., RGB cameras, IR cameras, depth cameras, video cameras). The first computing device 520 may be equipped with computing and storage resources to execute operations such object identification to identify objects, and feature extraction from the identified objects. In some embodiments, the first computing device 520 may comprise a computer, a smart device, a sensor, another device with computing resource (such as CPU, GPU, or ASIC, etc.) and storage capacity (such as volatile memory, non-volatile memory), or any combination thereof. In some embodiments, the first computing device 520 may optionally have persistent storage capability.
[0107] In some embodiments, the second computing device 510 may send control commands to the first computing device 520 at step 684. The first computing device 520 may subsequently forward the commands to the sensors 500 at step 686. In some embodiments, the second computing device 510 may directly send control commands to the sensors 500 at step 688. In some embodiments, the commands may include reboot, zoom in, zoom out, adjust angle, power on, power off, update firmware, and update stored data. As an example, if a new version of firmware for the sensors 500 is released offering new security features and better performance, the second computing device 510 may download the firmware and broadcast firmware upgrade commands along with the firmware binary to the sensors 500 being upgraded. As another example, if one sensor is not monitoring the designed area (the designed area may be marked by patterns on the floor, shelves, walls, or ceiling), a control command may be delivered to the sensor to adjust the angle accordingly. In some embodiments, the first computing device 520 may use commands to control the associated sensors 500 at step 690. For example, if the first computing device 520 stops receiving sensor data or heartbeats from one sensor, it may send probing signals followed by reboot command to that sensor. In some embodiments, a command may be sent to one specific sensor (e.g., a point-to-point or unicast message to reboot a camera), a group of sensors (e.g. a unicast or multicast message to adjust angle), or all sensors (e.g., a broadcasting message to update firmware).
Emphasis provided.
Note that paragraph 99 mentions that various computing devices (520), which may be a smart device or sensor or another device with a computing resource and storage/memory capacity, which can be construed to mean that some cameras with computing resource and memory can communicate image data or commands between other computing devices/cameras as illustrated in figure 5b, for example.
Regarding Claim 1, before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to have provided wherein one of the plurality of first edge cameras receives, from the at least one of the plurality of second edge cameras having a second peripheral viewing area within the first checkout station, image information corresponding to a target area within the first checkout station, combines the received image information with image information generated by the one of the plurality of first edge cameras, the received image information representing an alternate view of the target area within the first checkout station, and processes the combined image information to determine an event occurring at the first checkout station,
as taught by Zheng, in Whitelaw’s checkout terminal system for the purpose of increasing security by adding various cameras in peripheral areas of the store where the point of sale checkout systems are located, which are connected with each other and pass video information between them for processing at particular cameras, so as to track items as they are processed at adjacent checkout counters.
Regarding Claim 1, Whitelaw does not expressly teach overlapping fields of view to track items/articles/products.
Regarding Claim 1, Whitelaw does not expressly teach, but Jiang et al (US 10,438,277 B1) overlapping fields of view to track items/articles/products, as illustrated in figure 5 and as mentioned at col. 9, lines 20-45, col. 10, lines 23-col. 11, line 7, col. 12, lines 45-51, col. 14, lines 15-27, col. 18, lines 1-42, col. 19, line 57-col. 20, line 10, col. 20, line 27-35 and col. 30, lines 50-65, which states as follows.
(58) In some implementations, the size of the cluster is designed to be as wide and long as possible to reduce the frequency with which users cross cluster boundaries. Likewise, as illustrated in FIG. 5, the field of view of cameras of adjacent clusters may overlap, thereby causing the clusters to overlap. For example, the cameras may be positioned so that the field of view of adjacent cameras intersect and begin to overlap approximately six feet above the surface of the materials handling facility. Overlapping the field of view of cameras in adjacent clusters results in user patterns representative of users being detected in multiple clusters as the user moves from one cluster to another. Each cluster processing system will identify the user pattern and send the corresponding location information and session identifier for the cluster to the cluster aggregation system. The cluster aggregation system will determine that the location information for the two clusters overlap and determine that the two user patterns correspond to a single user and normalize the information to a single location and single user identifier.
(59) The overlap of clusters increases the ability for the system to monitor a user as they move throughout the materials handling facility and transition between clusters. The cluster aggregation system consolidates this information so that other systems (e.g., the inventory management system) receive a single representation of the user's location within the materials handling facility.
(63) The height of the cameras from the surface, the distance between camera placement and/or direction of the cameras 508 within the cluster 560 may vary depending on the layout of the materials handling facility, the lighting conditions in the cluster, the volume of users expected to pass through a portion of the cluster, the activities and/or volume of activities expected to occur at different locations within the cluster, etc. For example, cameras may typically be mounted every three to four feet in one direction and every four to five feet in another direction along the grid 502 so that the field of view of each camera overlaps, as illustrated in FIG. 5.
(64) In some implementations, the height of the cameras from the surface and the distance between cameras may be set so that their fields of view intersect and begin to overlap approximately seven feet above the surface of the materials handling facility. Positioning the cameras so that the fields of view overlap at approximately seven feet will result in the majority of users being within a field of view of a camera at all times. If the field of view of the cameras did not overlap until they were approximately three feet above the surface, as a user moves between the fields of view, the portion of the user that is taller than approximately three feet would exit one field of view and not enter the next field of view until the user has moved into that range of the camera. As such, a portion of the user is not detectable as they transition between fields of view. While this example describes overlapping camera fields of view at approximately seven feet above the surface of the materials handling facility, in other implementations, the cameras may be positioned so that the fields of view begin to overlap at different heights (e.g., six feet, eight feet).
(65) In some areas of the cluster, such as cluster area 506, cameras 508 may be positioned closer together and/or closer to the surface area, thereby reducing their field of view, increasing the amount of field of view overlap, and/or increasing the amount of coverage for the area. Increasing camera density may be desirable in areas where there is a high volume of activity (e.g., item picks, item places, user dwell time), high traffic areas, high value items, poor lighting conditions, etc. By increasing the amount of coverage, the image data increases, thereby increasing the likelihood that an activity or action will be properly determined.
(66) In some implementations, one or more markers 510 may be positioned throughout the cluster and used to aid in alignment of the cameras 508. The markers 510 may be placed at any location within the cluster. For example, if the markers are placed where there is an overlap in the field of view of two or more cameras, the cameras may be aligned with respect to one another, thereby identifying the pixel overlap between the cameras and aligning the pixels of the cameras. The markers may be any identifiable indicator and may be temporary or permanent.
73) As discussed above, the fields of view of cameras within a cluster may overlap with other cameras of the cluster. Image data may be sent for each field of view and processed by the cluster processing system 602, as discussed further below. Likewise, the fields of view on the perimeter of each cluster may overlap with the fields of view of cameras of an adjacent cluster.
(82) In one implementation, the cluster aggregation system 604 may utilize the received location information for each user pattern and determine user patterns received from different cluster processing systems that overlap and/or represent the same user. As discussed above, the field of view of cameras within adjacent clusters may overlap to aid in monitoring the location of a user as they move through the materials handling facility. When the user moves between clusters, both cluster processing systems will provide user pattern location information for a period of time. The cluster aggregation system 604 receives this information and determines that the two user patterns are to be associated with a single user.
(101) In addition to generating a point cloud, the example process 900 may also identify any pixels that are overlapping and/or assigned to a same horizontal coordinate, as in 906. As discussed above, the field of view of one or more cameras within a cluster may overlap. The overlapping fields of view may result in pixel information from corresponding image data being assigned to a same horizontal location within the materials handling facility. However, because the cameras are potentially different distances from the overlapping horizontal locations, the depth information (distance of an object from the camera) may be different. The example process 900 resolves the overlap by selecting one set of the pixel information as representative of the physical location. For example, the example process may select the pixel information that has a larger vertical component. Alternatively, the example process may select the pixel information that has a smaller vertical component. In still another example, the example process 900 may select pixel information that is an average of the vertical component between the overlapping pixel information. In still another example, the example process 900 may consider the vertical component of adjacent pixels and select the pixel information having a vertical component that is closest to adjacent pixels.
(110) Upon receipt of location information and session identifiers, the example process determines and resolves any overlaps in location information that are identified in locations of overlapping clusters. Similar to cameras within a cluster, cameras at the perimeter of adjacent clusters may have overlapping fields of view. When a user moves between clusters, they will be identified by each cluster for the portion of the overlap and, as such, each cluster processing system will provide a session identifier and location information for the detected user pattern. Because the location information from each cluster may vary slightly due to the cameras being at different distances/perspectives, the example process 1000 determines overlapping and/or similar location information for user patterns reported from different clusters and determines if the two reported session identifiers and location information correspond to the same user. For example, the example process 1000 may consider location information and session identifiers from prior points in time to determine a trajectory of a user and, based on that trajectory, determine if the two overlapping location information correspond to the same user.
(111) For each session identifier or resolved session identifier and corresponding location information, the example process may associate the session identifier with a user identifier, as in 1006. The user identifier may be any unique identifier that is utilized to monitor the location of the user. For example, the user identifier may be the user identifier determined or established as part of the example process 700. Because the cluster aggregation system and the corresponding example process 1000 generate a complete view of users throughout the materials handling facility, the example process 1000 continually receives location information for the user pattern of the user from the time the user is identified. Utilizing this information, the example process can maintain information as to which session identifiers generated by the cluster processing systems correspond to which user identifiers.
(112) In addition to determining and resolving overlapping location information for a user, the example process 1000 may determine an anticipated trajectory for each user based on the currently received location information and previously received location information for the user, as in 1008. For example, if a series of reported location information for a session identifier identifies that the user is moving down an aisle, the example process 1000 may determine that the user is likely to continue moving in that direction down the aisle.
(169) If it is determined that there is no occlusion, a no occlusion notification is returned, as in 1904. However, if it is determined that an occlusion exists near the event location, image information may be obtained from other cameras that have overlapping fields of view with the event location, as in 1906. As discussed above, the cameras may be position so that the field of view of each camera overlaps. If an event occurs at an event location in which two or more camera fields of view overlap, the image information from the adjacent cameras may be obtained. In addition to using image information from other overhead cameras, in some implementations, image information may be obtained from cameras at other positions within the materials handling facility. For example, image information (e.g., color values, depth information) from cameras positioned within or on inventory shelves may be obtained and utilized.
(170) Utilizing the image information from the camera associated with the area that includes the event location and image information obtained from other cameras, a full point cloud of the area around the event location may be generated, as in 1908. For example, multiple point clouds representative of different periods of time for the event location may be generated from different fields of view and combined to establish a complete point cloud that may provide additional information.
(171) Because the other cameras have different points of reference and different fields of view, the fields of view of the other camera(s) may provide additional information that can be used to resolve the occlusion. For example, while the field of view from the camera associated with the area of the event location may be occluded by a user leaning over another user, an adjacent camera and/or a camera positioned on the shelf facing into the aisle may not be occluded. By utilizing the information from multiple cameras, a point cloud may be generated that considers the depth information and/or color value information from each of the multiple cameras.
Emphasis provided.
Regarding Claim 1, before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to have provided overlapping fields of view to track items/articles/products,
as taught by Jiang, in Whitelaw’s checkout terminal system for the purpose of increasing security by adding various cameras in peripheral areas of the store where the point of sale checkout systems are located, which are connected with each other and pass video information between them for processing at particular cameras, so as to track items as they are processed at adjacent checkout counters.
Regarding Claim 2, Whitelaw does not expressly teach wherein the first peripheral viewing area of at least one of the first cameras is located within the second checkout station.
Regarding Claim 2, Whitelaw does not expressly teach, but Landers ‘130 teaches wherein the first peripheral viewing area of at least one of the first edge cameras (202) is located within the second checkout station (205, 300, 400). Note that paragraph 65 of Landers ‘130 teaches the network including edge servers.
Regarding Claim 3, Whitelaw does not expressly teach wherein the one first edge camera receives information from the plurality of first edge cameras and the at least one second edge camera.
Regarding Claim 3, Whitelaw does not expressly teach, but Carranza teaches wherein the one first edge camera, i.e., smart cameras (220a-220c, 902, 1150) receives information from the plurality of first edge cameras (220a-220c, 902, 1150) and the at least one second edge camera (220a-220c, 902, 1150) as mentioned at paragraphs 49-50, for example, which states as follows.
[0049] The smart cameras 220a-c each include one or more processors 221, memory elements 222, communication interfaces 223, and vision sensors 224 (e.g., cameras). The vision sensors 224 can include any type of sensors that can be used to capture or generate visual representations of their surrounding environment, such as cameras, depth sensors, ultraviolet (UV) sensors, laser rangefinders (e.g., light detection and ranging (LIDAR)), infrared (IR) sensors, electro-optical/infrared (EO/IR) sensors, and so forth. In particular, the vision sensors 224 are used to generate video streams associated with the environment in which an associated smart camera 220 is deployed. In various embodiments, a smart camera 220 may store video streams in memory 222, process video streams using its own processor 221, and/or transmit video streams and/or associated metadata over communication interface 223 to another component for processing and/or storage purposes (e.g., surveillance orchestration device 210).
[0050] The respective components of surveillance system 200 may be used to implement the intelligent camera orchestration functionality described further throughout this disclosure. Moreover, in various embodiments, the underlying components and functionality of surveillance system 200, surveillance orchestration device 210, and/or smart cameras 220 may be combined, separated, and/or distributed across any number of devices or components.
Emphasis provided.
See also paragraphs 57-59, 90, 114, 115, 117, 119 and 123.
Regarding Claim 4, Whitelaw teaches wherein the information includes images, i.e., cameras (121-124) as mentioned in paragraph 28, and noting the cameras in paragraphs 96 and 97, for example.
Regarding Claim 5, see the rejection of Claims 1-4, above.
Regarding Claim 6, see the rejection of Claim 4, above.
Regarding Claim 7, Whitelaw does not expressly teach wherein each of the plurality of first edge cameras and each of the plurality of second edge cameras are nodes within the plurality of nodes on the vision mesh network and in communication with each other.
Regarding Claim 7, Whitelaw does not expressly teach, but Carranza teaches wherein each of the plurality of first edge cameras (220a-220c, 902, 1150) and each of the plurality of second edge cameras (220a-220c, 902, 1150) are nodes within the plurality of nodes on the vision mesh network and in communication with each other, as mentioned in first sentence of paragraphs 91, 95 and 99, as well as at illustrated in figures 8 and 9, and in communication with each other, as mentioned at paragraph 90, second sentence, i.e., “a number of IoT devices 804 may communicate with a gateway 854, and with each other through the gateway 854” and paragraph 102, first sentence, i.e., “[c]ommunications from any IoT device 902 may be passed along a convenient path (e.g., a most convenient path) between any of the IoT devices 902 to reach the gateways 904”.
Note that it has been held that mere duplication of the essential working parts of a device involves only routine skill in the art. See St. Regis Paper Co. v. Bemis Co., 193 USPQ 8.
Regarding Claim 8, see the rejection of Claim 1, above, noting that it is considered a matter of design choice as to how many checkout counters and how many sets of cameras based upon the amount of customer item transaction throughput desired to be processed.
Regarding Claim 9, Whitelaw teaches further comprising two to four more
checkout stations (100) at the first location, as mentioned at paragraph 69, last sentence, which states “[i]n some embodiments, the steps of FIG. 8 may be at least partially executed by a central server that exchanges data with a plurality of checkout terminals”;.
Regarding Claim 10, see the rejection of Claim 7, above.
Regarding Claim 11, Whitelaw teaches, wherein the first primary viewing area
includes a target, i.e., noting that each camera (121-124, 153, 154, 1110, 1120, 1130, 1140, 1250, 1260, 1270, 1280, 1290) have fields of view (1111, 1121, 1131, 1141, 1251, 1261, 1271, 1281, 1291), as mentioned at paragraphs 96 and 97, for example.
Regarding Claim 12, Whitelaw teaches wherein the target includes a scanner
platter (120), as mentioned in paragraph 28, a scale (120), a scanner (134), as mentioned in paragraph 29, for example, a shopping cart, a handbasket, a bagging area or a payment area, i.e., card reader (132) as mentioned at paragraph 29 and as illustrated in figures 11-17, for example.
Regarding Claim 13, see the rejection of Claim 1.
Regarding Claim 14, see the rejection of Claim 1.
Regarding Claim 15, see the rejection of Claims 1-3.
Regarding Claim 16, see the rejection of Claim 1.
Regarding Claim 17, see the rejection of Claim 4.
Regarding Claim 18, see the rejection of Claim 1.
Regarding Claim 19, see the rejection of Claim 1.
Regarding Claim 20, see the rejection of Claim 1.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant is encouraged to contact the Examiner should there be any questions about this rejection or in an endeavor to explore potential amendments or potential allowable subject matter.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Glaser ‘086 is cited as teaching a self checkout system with imaging system (100), as illustrated in figure 2 and as mentioned at col. 9, lines 7-31, and an environmental object graph (EOG) modeling system, as mentioned at col. 7, lines 21-48.
Latapie ‘800 is cited as teaching smart cameras (302) including a mesh network as mentioned at col. 2, lines 40-55, also mentioning the mesh network is an internet of things (IoT) network with edge computing/fog computing as mentioned in col. 2, lines 56-67.
Carranza ‘171 is cited as teaching mutli-camera tracking (MCMCT) as mentioned in col. 2, lines 32-43 and smart cameras (220a-c with one or more processors (221) and memory elements (222), communication interfaces (223) and vision sensors (224), as mentioned at col. 9, lines 8-25 and as illustrated in figure 2 and IoT networks with fog devices as mentioned at col. 16, lines 1-14.
Tshouva ‘619 is cited as teaching internet-connected and networked connected devices, as mentioned at paragraph 3, with IoT internet of things as mentioned at paragraph 4, mentioned as including cameras and sensors as IoT things at paragraph 31, for example.
Crain ‘243 is cited as teaching edge camera clients at col 5, lines 7-11, a camera subsystem (202) with a camera server (225) and several camera clients (220) as well as cameras (205) as illustrated in figures 2a, 2b and 2b, for example.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEFFREY ALAN SHAPIRO whose telephone number is (571)272-6943. The examiner can normally be reached Monday-Friday generally between 8:30AM and 6:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anita Y Coupe can be reached at 571-270-3614. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JEFFREY A SHAPIRO/Primary Examiner, Art Unit 3619
September 22, 2026
Prosecution Insights