DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/19/2025 has been entered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Disposition of Claims
Claims 1-21 are pending in the instant application. Claim 12 has been cancelled. Claim 21 has been added. Claims 1, 3, 6-11, 13, and 16-21 have been amended. The rejection of the pending claims is hereby made non-final.
Response to Remarks
103
Applicant’s arguments and amendments have been considered by the examiner, but are not found to be persuasive. Applicant argues that the applied prior art of record, in particular Chaubard, fails to teach or suggest the use of a region-based convolutional neural network in the detection of out of stock items. The examiner respectfully disagrees. The examiner submits that Chaubard discloses wherein the out-of-stock detection system receives image data of a product display area from one or more cameras periodically, as described above. The out-of-stock detection system, in this embodiment, identifies product labels in the image data and generates bounding boxes. The system generates a set of individual product bounding boxes that each identify an individual product within the display location corresponding to a single bounding box. Accordingly, the system compares the number of individual products within the display location from the image to the number of product associated with the fully stocked product display area of the template image for each display location within the product display area. In response to determining that the number of individual products within the display location does not match the number of product associated with the fully stocked product display area of the template image, the system generating an out-of-stock notification for the product corresponding to the product display location to a store client device (see at least paragraph [0035] to Chaubard). Chaubard further teaches wherein in the deployment stage, the cameras 100 capture an image an hour, for example, and for each image, for loops over each template, crops the region corresponding to each bounding box and forward passes that template over the captured image to predict whether the product corresponding to each bounding box is in stock, out of stock, low (e.g., 1-3 products left for that SKU), bad template (e.g., the camera shifted or a stocker changed the shelf unit organization dramatically), Planogram compliance issue (e.g., wrong product is there), or occlusion (e.g., the area of the shelf unit is blocked by an individual, cart, etc.). Then the out-of-stock detection system may compile a list of action items to send to a team of store associate stockers (see at least paragraph [0036] to Chaubard et al). This can be accomplished by the product-detection model generating bounding boxes for each product and determines a likelihood that the product-detection model's prediction is correct. The product-detection model can be a convolutional neural network that has been trained using labeled training data via Stochastic Gradient Descent based on the template’s images. Additionally, the product detection module 150 uses the product-detection model to compare images received from the one or more cameras 100 to template images captured by the store client device 150 or to higher resolution images captured by the cameras 100 in the onboarding process (see at least paragraph [0049] to Chaubard). It is clear to one of ordinary skill in the art that the applied prior art reference Chaubard clearly discloses the selection of regions and the usage of a convolutional neural network to identify out of stock conditions. For at least the reasoning provided above, the rejection of the pending claims is hereby maintained.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-11 and 13-21 are rejected under 35 U.S.C. 103 as being unpatentable over Chaubard (US 2020/0005225) in view of Adato et al (US 2019/0236531) and further in view of Yang et al (US 2017/0076195).
Regarding claim 1, the prior art discloses a system comprising: one or more processors (see at least paragraph [0061] to Chaubard “processor”); and one or more non-transitory computer-readable media storing computing instructions that, when executed on the one or more processors, perform: receiving a plurality of images from one or more devices, the images corresponding to a store shelf of a physical retail store (see at least paragraph [0020] to Chaubard “the out-of-stock detection system may obtain information for a particular shelf unit 105 from the planogram of the store that identifies what products should be on that particular shelf unit based on the camera ID for a particular camera 100 associated with the template image or images”); processing the shelf image in the encoded string format with a region-based convolutional neural network using pre-trained weights to output an output image that identifies (see at least paragraph [0049] to Chaubard) positions in the planogram that correspond to an out-of-stock detection for an item of the store shelf (see at least paragraph [0036] to Chaubard, wherein in the deployment stage, the cameras 100 capture an image an hour, for example, and for each image, for loops over each template, crops the region corresponding to each bounding box and forward passes that template over the captured image to predict whether the product corresponding to each bounding box is in stock, out of stock, low (e.g., 1-3 products left for that SKU), bad template (e.g., the camera shifted or a stocker changed the shelf unit organization dramatically), Planogram compliance issue (e.g., wrong product is there), or occlusion (e.g., the area of the shelf unit is blocked by an individual, cart, etc.). Then the out-of-stock detection system may compile a list of action items to send to a team of store associate stockers); generating a first alert, the first alert including an indication of coordinates of the out-of-stock detection for the item of the store that corresponds to the coordinates (see at least paragraphs [0028] and [0046] to Chaubard “The camera may also send a SMS based alert with the current state of a shelf unit”);
Chaubard does not appear to explicitly disclose transmitting the first alert to a first electronic device of a first employee to provide more inventory of the item on the store shelf;
Determining whether back room inventory for the item exists at the physical retail store;
When the back-room inventory for the item is determined to not exist, generating a second alert; and
Transmitting the second alert to a second device of a second employee to order more of the back-room inventory for the item.
However, Adato et al discloses a system and method for comparing planogram compliance to checkout data, further comprising transmitting the first alert to a first electronic device of a first employee to provide more inventory of the item on the store shelf (see at least paragraph [0384] to Adato et al, wherein a first action may refer to server 135 causing real-time automated alerts when products may be out of shelf, when pricing may be inaccurate, when intended promotions may be absent, and/or when there may be issues with planogram compliance, among others);
Determining whether back room inventory for the item exists at the physical retail store (see at least paragraph [0691] to Adato et al, wherein systems 100, 3300 may further rely on: manual input (e.g. from a user such as a shift manager), inventory information (e.g. from backroom or nearby warehouses)) ;
When the back-room inventory for the item is determined to not exist, generating a second alert (see at least paragraph [0136] to Adato et al, wherein inventory data 246 that may be used to determine if additional products should be ordered from suppliers 115; and
Transmitting the second alert to a second device of a second employee to order more of the back-room inventory for the item (see at least paragraph [0497] to Adato et al, wherein when the employee did not perform the task of returning the misplaced product to its correct display location, image processing unit 130 may be configured to cause issuance of another user-notification. This may alert the user and remind the user to perform the task).
Chaubard and Adato et al, in combination, fail to teach or suggest further comprising combining the plurality of images to generate a shelf image corresponding to the store shelf; encoding the shelf image into an encoded string format, wherein encoding the shelf image into the first processing format further comprises: responsive to converting the shelf image into the encoded string format, processing the encoded string format of the shelf image via an application programming interface (API) based on horizontal facing quantities and vertical facing quantities of a planogram.
However, Yang et al discloses a system and method of distributed neural networks for scalable real-time analytics, further comprising combining the plurality of images to generate a shelf image corresponding to the store shelf; encoding the shelf image into an encoded string format, wherein encoding the shelf image into the first processing format further comprises: responsive to converting the shelf image into the encoded string format, processing the encoded string format of the shelf image via an application programming interface (API) based on horizontal facing quantities and vertical facing quantities of a planogram (see at least paragraphs [0044-0045] to Yang et al).
The examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). The examiner submits that the combination of the teaching of the system and method of on shelf image based out of stock detection, as disclosed by Chaubard and the system and method of comparing planogram compliance to checkout data, as taught by Adato et al, further in view of the system and method of distributed neural networks for scalable real-time analytics, as disclosed by Yang et al, in order to enable store employees to monitor and replenish local stock and initiate order fulfillment based on real time inventory data, could have been readily and easily implemented, with a reasonable expectation of success. As such, the aforementioned combination is found to be obvious to try, given the state of the art at the time of filing.
Regarding claim 2, the prior art discloses the system of claim 1, wherein the one or more devices comprise at least one of: a shelf- scanning robot, a drone, or a camera (see at least paragraph [0014] to Chaubard).
Regarding claim 3, the prior art discloses the system of claim 1, wherein combining the plurality of images further comprises combining the plurality of images based on the planogram indicating where items of the physical retail store are to be located, the planogram comprising the horizontal-facing quantities and vertical-facing quantities (see at least paragraph [0020] to Chaubard “Alternatively, the out-of-stock detection system may identify a portion of the planogram corresponding to a particular shelf unit 105 and identify the products 110 on the shelf unit 105 by overlaying the planogram onto the image shelf unit 105 by, for example, spatially matching bounding boxes”).
Regarding claim 4, the prior art discloses the system of claim 1, wherein each of the plurality of images comprise metadata corresponding to a sequential order in which each of the plurality of images was captured (see at least paragraph [0023] to Chaubard “ Accordingly, once these images are received by the system, the system may optionally stitch the images into a single large image”).
Regarding claim 5, the prior art discloses the system of claim 1, wherein encoding the shelf image into the encoded string format further comprises: converting the shelf image to the encoded string format (see at least paragraph [0055] to Chaubard “ The user interface module 180 may provide application programming interface (API) functionality to send data directly to native client device operating systems, such as IOS®, ANDROID™, WEBOS®, or RIM® “).
Regarding claim 6, the prior art discloses the system of claim 1, wherein the region-based convolutional neural network comprises at least one of: a Masked Region-Based Convolutional Neural Network or a Single Shot Detector (SSD) (see at least paragraph [0049] to Chaubard “The product-detection model can be a convolutional neural network that has been trained using labeled training data via Stochastic Gradient Descent based on the templates images”).
Regarding claim 7, the prior art discloses the system of claim 1, further comprising calibrating the region-based convolutional neural network using location loss and class loss (see at least paragraph [0049] to Chaubard “In some embodiments, multiple models are trained where one model detects empty shelves and another model detects partly empty shelves”) wherein the location loss comprises where the item (that is determined to be out of stock) is located on the store shelf, and wherein the class loss comprises where the item that is determined to be out of stock is (see at least paragraph [0738] to Adato et al).
Regarding claim 8, the prior art discloses the system of claim 1, wherein the computing instructions, when executed on the one or more processors, further perform: training the region-based convolutional neural network using a first set of training data corresponding to a portion of items in the physical retail store; and processing the shelf image without retraining the region-based convolutional neural network (see at least paragraph [0049] to Chaubard).
Regarding claim 9, the prior art discloses the system of claim 1, wherein the processing the shelf image in the encoded string format with the region-based convolutional neural network also outputs a probability of a presence or absence of an out-of-stock detection (see at least paragraphs [0049] and [0051] to Chaubard “out-of-stock detection module 175”).
Regarding claim 10, the prior art discloses the system of claim 1, wherein processing the shelf image in the encoded string format with the region-based convolutional neural network further comprises: when processing the shelf image, using the region-based convolutional neural network to implement a reduction of search space and to segment the shelf image using k-means clustering in color (RGB) space (see at least paragraph [0703] to Adato et al, wherein he first set of images may be analyzed with a machine learning model trained using training examples to determine first product turnover data from a set of images. In another example, the first set of images may be analyzed with an artificial neural network configured to determine first product turnover data from a set of images and paragraph [0049] to Chaubard).
Claims 11 and 13-21 each contain recitations substantially similar to those addressed above and, therefore, are likewise rejected.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
The examiner has considered all references listed on the Notice of References Cited, PTO-892.
The examiner has considered all references cited on the Information Disclosure Statement submitted by Applicant, PTO-1449.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TALIA F CRAWLEY whose telephone number is (571)270-5397. The examiner can normally be reached on Monday thru Thursday; 8:30 AM-4:30 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Fahd A Obeid can be reached on 571-270-3324. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TALIA F CRAWLEY/Primary Examiner, Art Unit 3627