DETAILED ACTION
Notice of AIA Status
The present application is being examined under the AIA the first inventor to file provisions.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/29/2024, 07/29/2025, 11/25/2025, and 04/23/2026 were considered and placed in the applicant’s file.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1 along with dependent claims 2-7 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Claim 1 is drawn to at least one computer readable media encoded with instructions which, when executed, cause the system to perform operations, where the at least one computer readable media encoded with instructions which, when executed, cause the system to perform operations as defined in the specification in paragraph [0029-0032]. The specification at paragraph [0029] states “The computing device 102 further has one or more computer-readable media such as the memory 108. The memory 108 includes any quantity of media associated with or accessible by the computing device 102…The memory 108 can include read-only memory and/or memory wired into an analog computing device.” Moreover, paragraph [0033] states “the memory 108 stores data, such as one or more applications. The applications, when executed by the processor 106, operate to perform functionality on the computing device 102.” It has been understood by the office that the at least one computer readable media is the same as "computer program medium" and "computer usable medium" and that these are forms of memory which are statutory (e.g., ROM, RAM, hard drive, a hard disk, an integrated circuit assembly, magnetic tape, cartridge, optical disk, a CD or DVD, or other fixed or removable medium that is read by, written to or accessed by media drive 512). As these examples illustrate, the memory 108 can include a computer usable storage medium having stored thereon computer software or data, but also transitory propagating signals, per se. Therefore, claim 1 includes both transitory, and non-statutory subject matter. As a result, the claim is explicitly defined to encompass both transitory and non-transitory, which can be a signal, carrier wave, etc. Therefore, claim 1 fail(s) to fall within at least one of the four categories of patent eligible subject matter. See MPEP 2106.
The Office respectfully recommends the Applicant to amend claim 1 limitation “a computer-readable medium storing instructions that are operative upon execution by a processor to” to reflect the limitation “a non-transitory computer-readable medium storing instructions that are operative upon execution by a processor to”.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.
Claims 1, 8, and 15 recites limitations that use words like “means” (or “step”) or similar terms with functional language and do invoke 35 U.S.C. 112(f):
Claim 1; recites the limitation, “obtain an image of an item storage structure captured by an image capture device using a multi-object cluster detection model.” [Line 5-6].
Claim 8; recites the limitation, “obtaining, from an image capture device, an image of an item storage structure using a multi-object cluster detection model.” [Line 5-6].
Claim 15; recites the limitation, “obtaining an image of an item storage structure captured by an image capture device using a multi-object cluster detection model,” [Line 4-5].
Because these claim limitation(s) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
After a careful analysis, as disclosed above, and a careful review of the specification the following limitations in claims 1, 8, and 15:
(i) “a multi-object cluster detection model” (Fig. 1 and 3, #302. Paragraph [0020-0021 0049 0071 and 0072]- This enables utilization of a single trained multi-object cluster detection model to detect and recognize multiple different types of objects in multiple clusters within a given location captured in one or more images of the location. the disclosure enable a trained deep learning convolutional neural network (CNN) model to analyze a plurality of images of a location, such as an item storage area, to detect clusters of different types of items and/or the number of different types of items in an area more accurately for use in inventory, locating items within the store, updating planograms, restocking shelves, identifying incorrectly placed (misplaced) items, etc. the cluster detection manager 130 includes one or more machine learning (ML) model(s) 136. The ML model(s) 136 include a CV multi-object cluster detection model which is trained to detect unique different types of items and/or different clusters of items. The cluster detection manager 130, in some embodiments, obtains an image of an item storage structure captured by an image capture device using a multi- object cluster detection model. The cluster detection manager 130 in this example is implemented on the computing device 102. However, the embodiments are not limited to implementing the cluster detection manager on a computing device. In other embodiments, the cluster detection manager is implemented on a cloud server, such as, but not limited to, the cloud server 118. In Figure 3, the multi-object cluster detection model 302 is illustrated as a black box within the cluster detection manager #130. (wherein the multi-object cluster detection model #302 has sufficient structure associated with it cloud server/computer.)).
If applicant does not intend to have these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claims 1, 4-5, 7-8, 11-12, 14-15, 17-18 and 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by BRONICKI (US 20220261831 A1), hereinafter referenced as BRONICKI.
Regarding claim 1, BRONICKI explicitly teaches system (Fig. 1, #100 called a system. Paragraph [0112]-BRONICKI discloses reference is now made to FIG. 1, which shows an example of a system 100 for analyzing information collected from retail stores 105. System 100 may represent a computer-based system. System 100 may include or be connected to various network computing resources. In paragraph [0114]-BRONICKI discloses system 100 may include an image processing unit 130 to execute the analysis of images captured by the one or more capturing devices 125. Image processing unit 130 may include a server 135 operatively connected to a database 140. Please also see Fig. 2, 19, and 22 and read paragraph [0328 and 0344]) for multi-object cluster detection with improved accuracy (Fig. 21. Paragraph [0118]-BRONICKI discloses server 135 may be coupled to one or more physical or virtual storage devices such as database 140. Server 135 may access database 140 to detect and/or identify products. In paragraph [0337]-BRONICKI discloses FIG. 21 is a schematic diagram illustrating a sensor fusion technique for identifying the products on a retail shelf, consistent with the present disclosure. In paragraph [0359]-BRONICKI discloses by integrating the multiple parameters and features extracted from on-shelf sensor data 2100 and image data 2150, processing device 2202 may determine the product type accurately. Please also see Fig. 1, 19, and 22, and read paragraph [0120, 0168, 0232, 0401, 0470]), the system comprising:
a computer-readable medium (Fig. 2, #226 called a memory device. Paragraph [0129]) storing instructions (Fig. 2. Paragraph [0129]-BRONICKI discloses memory device 226 may include graphical user interface instructions 230; image processing instructions 232; sensor processing instructions 234; web browsing instructions 236; and other software instructions 238 to facilitate other processes and functions. Each of the above identified instructions and applications may correspond to a set of instructions for performing one or more functions. In paragraph [0130]-BRONICKI discloses memory device 226 may store database 140. Database 140 may include product type model data 240 (e.g., an image representation, a list of features, a model obtained by training machine learning algorithm using training examples, an artificial neural network, and more) that may be used to identify products in received images; contract-related data 242 (e.g., planograms, promotions data, etc.) that may be used to determine if the placement of products on the store shelves. Please also read paragraph [0123]) that are operative upon execution by a processor (Fig. 2 #202 called a processing device. Paragraph [0122]-BRONICKI discloses processing device 202, shown in FIG. 2, may include at least one processor configured to execute computer programs, applications, methods, processes, or other software to execute particular instructions associated with embodiments described in the present disclosure. Further in paragraph [0123]-BRONICKI discloses the methods and processes disclosed herein may be performed by server 135 as a result of processing device 202 executing one or more sequences of one or more instructions contained in a non-transitory computer-readable storage medium. Please also read paragraph [0424]) to:
obtain an image (Fig. 1. In paragraph [0114]-BRONICKI discloses system 100 may include an image processing unit 130 to execute the analysis of images captured by the one or more capturing devices 125. Image processing unit 130 may receive raw or processed data from capturing device 125 (wherein image processing unit 130 may use any suitable image analysis technique including, for example, object recognition, object detection, image segmentation, feature extraction, optical character recognition (OCR), object-based image analysis, shape region techniques, edge detection techniques, pixel-based detection, artificial neural networks, convolutional neural networks, etc.). In paragraph [0142]-BRONICKI discloses some non-limiting examples of image data may include images, grayscale images, color images, 2D images, 3D images, videos, 2D videos, 3D videos, frames, footages, data derived from other image data, and so forth (wherein an image may be, for example, image #1800 shown in Fig. 18A and image data #2150 shown in Fig. 21). Please also see Fig. 18A and 21-22, and read paragraph [0350]) of an item storage structure (Fig. 21. Paragraph [0350]-BRONICKI discloses the images may include a representation of at least part of the retail shelf and at least one of the one or more products (wherein an item storage structure may be, for example, a shelf, displays, containers, drawer units, bin units, baskets, pegboards, etc.). Please also see Fig. 18A and 33, and read paragraph [0297, 0402 and 0475)) captured by an image capture device (Fig. 1. Paragraph [0316]-BRONICKI discloses image sensor 1910 may be mounted (for example, to store shelves, walls, ceilings, floors, refrigerators, checkout stations, displays, dispensers, rods which may be connected to other objects in the retail store, and so forth) and configured to take images of the items hanging on peg-hooks 1810 (wherein an image capture device may be image capturing device #125 and image sensor #1910 or #2245, for example). Further in paragraph [0350]-BRONICKI discloses processing device 2202 may use the data generated by image sensors to identify the products displayed on a retail shelf Processing device 2202 may then analyze the images (i.e., image data 2150 shown in FIG. 21) captured by image sensors 2245 to determine the presence (or absence) of one or more products on the retail shelf, the product type associated with the one or more products, and other information related to the one or more products. Please also see Fig. 19 and 21-22, and read paragraph [0145, 0325 and 0350]) using a multi-object cluster detection model (Fig. 1. In paragraph [0118]-BRONICKI discloses server 135 may access database 140 to detect and/or identify products. The detection may occur through analysis of features in the image using an algorithm and stored data. The identification may occur through analysis of product features in the image according to stored product models. The term “product model” refers to any type of algorithm or stored product data that a processor may access or execute to enable the identification of a particular product associated with the product model. The product model may include a description of visual and contextual properties of the particular product (e.g., the shape, the size, the colors, the texture, the brand name, the price, the logo, text appearing on the particular product, the shelf associated with the particular product, adjacent products in a planogram, the location within the retail store, etc.) (wherein trained machine learning models may be used for product detection/identification, the multi-object cluster detection model may be, for example, a product recognition model #2160, and product models may comprise an inference model, such as a clustering model, a deep neural network, a convolutional neural network, a recurrent neural network, etc.). Please also see Fig. 19-22, and read paragraph [0139-0140, 0328, 0352 and 0363]);
detect a plurality of clusters of items of interest associated with a plurality of object types within the image (Fig. 21. Paragraph [0335]-BRONICKI discloses visual data produced by the off-shelf cameras, may be used to identify the presence (or absence) of a product and/or determine the product type associated with the product (wherein “identifying a product” refers to the operations for detecting the presence/absence of a product, determining the type, condition, and quality of a product, determining the quantity of the same product type and determining the arrangement of the products on the retail shelf, etc.). In paragraph [0337]-BRONICKI discloses image data 2150 may be input to one or more product recognition models 2160. The output of product recognition models 2160 may include product type information 2190 regarding the products displayed on the retail shelf, and other information related to at least part of the products displayed on the retail shelf. Please also see Fig. 18-19, 22 and 29A-B, and read paragraph [0316-0326 and 360]), the plurality of clusters comprising a first cluster of items associated with a first object type and a second cluster of items associated with a second object type (Fig. 22. Paragraph [0362]-BRONICKI discloses processing device 2202 may analyze image data 2150 of a retail shelf and determine that the products on the shelf belong to a first product type. Processing device 2202 may analyze on-shelf sensor data 2100 associated with the retail shelf to determine that the items on the shelf belong to a second product type. Processing device 2200 may analyze additional information (i.e., information other than on-shelf sensor data 2100 and image data 2150) to determine the product type associated with the products. The additional information may include historic information regarding the retail shelf (e.g., the product types previously displayed on the retail shelf), place of the retail shelf, planogram information, product types of nearby products, shelf labels, height of the retail shelf, 3D images. Processing device 2202 may determine, based on the analysis of the additional information, that the products on the shelf belong to a third product type that differs from both the first and second product types. Please also see Fig. 11D, 18G, and 29A-B, and read paragraph [0256, 0320-0335, 0358, 0364]);
generate a labeled image comprising a plurality of indicators within an overlay associated with the image (Fig. 29A-B. Paragraph [0249]-BRONICKI discloses the images may be used to construct a store plan representing at least one product characteristic relative to a display structure. A store plan may include a planogram, a realogram, a three-dimensional (3D) model of the retail store, or the like, or a combination thereof. A realogram may be a virtual copy of shelves in a store. The images may be used to construct a store plan representing at least one product characteristic relative to a display structure associated with a retail environment (such as a store shelf or area of one or more shelves) (wherein the store plan may be used to determine planogram compliance and to detect/identify products using product characteristics, such as product presence/absence, product type, product size, product color, product shape, total and individual product quantities, product density, product homogeneity (i.e. the determination and count of one or more different unique product types), product combinations, product arrangements, product configurations, etc.). Please also see Fig. 11D, 18G, and 25, and read paragraph [0251, 283, 0319-0335, 0362, 0380, 0383 and 0480]), the plurality of indicators comprising a first cluster indicator associated with the first cluster of items within the image and a second cluster indicator associated with the second cluster of items within the image (Fig. 29A-B. Paragraph [0403]-BRONICKI discloses as shown in FIG. 27A, image 2700 may include at least part of a shelf 2702. Shelf 2702 may be a shelving unit at least partially dedicated to dairy products within a retail store. Shelf 2702 may include milk products 2710, cream products 2720, butter products 2730, and egg products 2740. Shelf 2702 and products 2710, 2720, 2730, and 2740 are provided. Server 135 may be configured to analyze image to determine a placement of products on the shelf (or part of the shelf). Image processing unit 130 may use various image analysis techniques to detect products 2710, 2720, 2730, and 2740 on shelf 2702 (wherein the generated labeled image may be, for example, the following: a planogram compliance image displayed in GUI 1140 shown in Fig. 11D, which contains visual indicators for correct, misplaced or missing items; a distribution map of a pegboard #20 shown in Fig. 18G, which contains shapes, colors and symbols, such as hatched symbols to indicate the quantity of items; an augmented virtual reality display area 2510 shown in Fig. 25, which contains destination overlay indicators (e.g. a symbol, icon, outline, color, image distortion, or other visual indicator) corresponding to a product name, location, and/or identifier, and/or; image #2700 shown in Fig. 29A, which contains visual indicators surrounding similar product groupings)). Please also see Fig. 11D, 18G, and 25, and read paragraph [0283, 0319-0335, 0362, 0380, 0383 and 0480]); and
identify a number of unique clusters of items associated with a location corresponding to the plurality of clusters detected within the labeled image using the plurality of indicators (Fig. 29A-B. Paragraph [0213]-BRONICKI discloses determining, based on the at least one pattern associated with a detected product and the at least one empty space, at least one aspect of planogram compliance (wherein the aspect of planogram compliance may include the presence or absence of particular products (or brands), locations of products on the shelves, quantities of products within particular areas (e.g., identifying stacked or clustered products)). Further in paragraph [0214]-BRONICKI discloses the at least one aspect may include product homogeneity, and may further include counting occurrences where a product of the second type is placed on an area of the store shelf associated with the first type of product. By accessing a memory including base patterns (or any other type of pattern associated with product types, such as product models), the at least one processor may detect different products and product types. Such information may be used, for example, to monitor whether a certain region of a shelf includes an appropriate or intended product or product type. Such information may also be useful in determining whether products or product types have been mixed (e.g., product homogeneity). Regarding planogram compliance, detection of different products and their relative locations on a shelf may aid in determining whether a product homogeneity value, ratio, etc. has been achieved. At least one processor may count occurrences where a product of a second type is placed on an area of the store shelf associated with a product of a first type. Please also see Fig. 11D and 18G, and read paragraph [0178, 0319-0335, 0362, 0366, 0403-0408 and 0492-0498]).
PNG
media_image1.png
636
1269
media_image1.png
Greyscale
Fig. 29A. An image #2700 representing product characteristics generated through the detection, identification and labeling of unique product clusters (e.g. #2710, #2720, etc.) using trained neural networks (e.g. product recognition models #2150). Image #2700 may be further assessed for compliance against a target planogram (see Fig. 29B), and intermediate planograms may be generated representing product arrangements between adjustments.
Regarding claim 4, BRONICKI explicitly teaches the system of claim 1, BRONICKI further teaches wherein the instructions are further operative to:
validate item identification of at least one item associated with the location using the identified number of unique clusters (Fig. 11D. Paragraph [0233]-BRONICKI discloses server 135 may cause real-time automated alerts when products are out of shelf (or near out of shelf), when there are issues with planogram compliance, among others. GUI 1140 may include a first display area 1142 for showing a display of a video stream captured by output device 145C (e.g., a real-time display or a near real-time display) with augmented markings indicting a status of planogram compliance for each product (e.g., correct place, misplaced, not in planogram, empty, and so forth). GUI 1140 may also include a second display area 1144 for showing a summary of the planogram compliance for all the products identified in the video stream captured by output device 145C. Please also see Fig. 18G and 29A-B, and read paragraph [0249-0251, 0322-0324, 0362, 0266, 0403-0408]).
Regarding claim 5, BRONICKI explicitly teaches the system of claim 1, BRONICKI further teaches wherein the instructions are further operative to:
analyze the image (Fig. 22. Paragraph [0110]-BRONSKI discloses embodiments include analyzing images to detect and identify different products. The term “detecting a product” may broadly refer to determining an existence of the product. The system may determine the existence of a plurality of distinct products displayed on a store shelf. By detecting the plurality of products, the system may acquire different details relative to the plurality of products (e.g., how many products on a store shelf are associated with a same product type), but it does not necessarily gain knowledge of the type of product. The term “identifying a product” may refer to determining a unique identifier associated with a specific type of product that allows inventory managers to uniquely refer to each product type in a product catalogue). Please also read paragraph [0358-0359]) using a set of parameters for distinguishing unique items (Fig. 22. Paragraph [0118]-BRONICKI discloses the detection may occur through analysis of features in the image using an algorithm and stored data. The identification may occur through analysis of product features in the image according to stored product models. The term “product model” refers to any type of algorithm or stored product data that a processor may access or execute to enable the identification of a particular product associated with the product model. A product model may be used by server 135 to identify more than one type of products, such as, when two or more product models are used in combination to enable identification of a product. A first product model may be used by server 135 to identify a product category (such models may apply to multiple product types, e.g., shampoo, soft drinks, etc.), and a second product model may be used by server 135 to identify the product type, product identity, or other characteristics associated with a product. Please also read paragraph [0358-0364]), the set of parameters comprising size, shape, color, and proximity (Fig. 29A-B. Paragraph [0110]-BRONICKI discloses the term “identifying a product” may refer to determining a unique identifier associated with a specific brand of products the identification may be made based at least in part on visual characteristics of the product (e.g., size, shape, logo, text, color, etc.). In paragraph [0118]-BRONICKI discloses the product model may include a description of visual and contextual properties of the particular product (e.g., the shape, the size, the colors, the texture, the brand name, the price, the logo, text appearing on the particular product, the shelf associated with the particular product, adjacent products in a planogram, the location within the retail store, etc.). Please also read paragraph [0111, 0319-0335 and 0362]); and
detect a first cluster comprising at least one instance of a first type of item and a second cluster comprising at least one instance of a second type of item (), wherein the first type of item has at least one different attribute than the second type of item, an attribute comprising at least one of a size, shape, and color (Fig. 29A. Paragraph [0403]-BRONICKI discloses image 2700 may include at least part of a shelf 2702. Shelf 2702 may include milk products 2710, cream products 2720, butter products 2730, and egg products 2740. Shelf 2702 and products 2710, 2720, 2730, and 2740. Server 135 may be configured to analyze image to determine a placement of products on the shelf (or part of the shelf). Image processing unit 130 may use various image analysis techniques to detect products 2710, 2720, 2730, and 2740 on shelf 2702 (wherein each product set has a different size, shape and/or color). Please also see 11D and 18G, and read paragraph [0362]).
Regarding claim 7, BRONICKI explicitly teaches the system of claim 1, BRONICKI further teaches wherein the instructions are further operative to:
map a current location of each unique cluster of items in the plurality of clusters of items to a location of a portion of the item storage structure (Fig. 27A. Paragraph [0402]-BRONICKI discloses a shelf may refer to any structure used for displaying products in a retail environment. In paragraph [0297]-BRONICKI discloses the above-described methods and systems regarding the “store shelf” or “shelf” may also be used to monitor and detect changes of the items hanging on the pegboards and peg-hooks (wherein an item storage structure is a shelf, pegboard and/or storage area, such as pegboard #20 and shelf #2702). Please also see Fig. 18A-G and read paragraph [0319, 0350 and 0452]) in a database (Fig. 1, #140 called a database. Paragraph [0118]-BRONICKI discloses as depicted in FIG. 1, server 135 may be coupled to one or more physical or virtual storage devices such as database 140. Server 135 may access database 140 to detect and/or identify products. In paragraph [0119]-BRONICKI discloses database 140 may include one or more memory devices that store data and instructions used to perform one or more features of the disclosed embodiments) using image data associated with the labeled image (Fig. 11D. Paragraph [0112]-BRONICKI discloses the system may retrieve different types of contextual information from captured image data and/or from other data sources. Contextual information may include a location of the product in the retail store. Further in paragraph [0233]-BRONICKI discloses shown in FIG. 11D, GUI 1140 may include a first display area 1142 for showing a display of a video stream captured by output device 145C (e.g., a real-time display or a near real-time display) with augmented markings indicting a status of planogram compliance for each product (e.g., correct place, misplaced, not in planogram, empty, and so forth). Please also see Fig. 19G, and 29A-B, and read paragraph [0283, 0320-0324, 0362, 0366, 0403-0407]).
Regarding claim 8, BRONICKI explicitly teaches the method for multi-object cluster detection with improved accuracy (Fig. 21. Paragraph [0337]-BRONICKI discloses FIG. 21 is a schematic diagram illustrating a sensor fusion technique for identifying the products on a retail shelf, consistent with the present disclosure. On-shelf sensor data 2100 and image data 2150 may be input to one or more product recognition models 2160. The output of product recognition models 2160 may include product type information 2190 regarding the products displayed on the retail shelf, and other information related to at least part of the products displayed on the retail shelf (such as the condition of the products, the quality (e.g., freshness) of the products, the quantity of the products, the arrangement of the products, etc. In paragraph [0359]-BRONICKI discloses by integrating the multiple parameters and features extracted from on-shelf sensor data 2100 and image data 2150, processing device 2202 may determine the product type accurately. Please also see Fig. 1, 19, and 22, and read paragraph [0120, 0168, 0232, 0401, 0470]), the method comprising:
obtaining, from an image capture device (Fig. 1. Paragraph [0316]-BRONICKI discloses image sensor 1910 may be mounted (for example, to store shelves, walls, ceilings, floors, refrigerators, checkout stations, displays, dispensers, rods which may be connected to other objects in the retail store, and so forth) and configured to take images of the items hanging on peg-hooks 1810 (wherein an image capture device may be image capturing device #125, image sensor #1910 or #2245, for example). Further in paragraph [0350]-BRONICKI discloses processing device 2202 may use the data generated by image sensors to identify the products displayed on a retail shelf Processing device 2202 may then analyze the images (i.e., image data 2150 shown in FIG. 21) captured by image sensors 2245 to determine the presence (or absence) of one or more products on the retail shelf, the product type associated with the one or more products, and other information related to the one or more products. Please also see Fig. 19 and 21-22, and read paragraph [0145, 0325 and 0350]), an image (Fig. 1. In paragraph [0114]-BRONICKI discloses system 100 may include an image processing unit 130 to execute the analysis of images captured by the one or more capturing devices 125. Image processing unit 130 may receive raw or processed data from capturing device 125 (wherein image processing unit 130 may use any suitable image analysis technique including, for example, object recognition, object detection, image segmentation, feature extraction, optical character recognition (OCR), object-based image analysis, shape region techniques, edge detection techniques, pixel-based detection, artificial neural networks, convolutional neural networks, etc.). In paragraph [0142]-BRONICKI discloses some non-limiting examples of image data may include images, grayscale images, color images, 2D images, 3D images, videos, 2D videos, 3D videos, frames, footages, data derived from other image data, and so forth (wherein an image may be, for example, image #1800 shown in Fig. 18A and image data #2150 shown in Fig. 21). Please also see Fig. 18A and 21-22, and read paragraph [0350]) of an item storage structure (Fig. 21. Paragraph [0350]-BRONICKI discloses the images may include a representation of at least part of the retail shelf and at least one of the one or more products (wherein an item storage structure may be, for example, a shelf, displays, containers, drawer units, bin units, baskets, pegboards, etc.). Please also see Fig. 18A and 33, and read paragraph [0297, 0402 and 0475)) using a multi-object cluster detection model (Fig. 1. In paragraph [0118]-BRONICKI discloses server 135 may access database 140 to detect and/or identify products. The detection may occur through analysis of features in the image using an algorithm and stored data. The identification may occur through analysis of product features in the image according to stored product models. The term “product model” refers to any type of algorithm or stored product data that a processor may access or execute to enable the identification of a particular product associated with the product model. The product model may include a description of visual and contextual properties of the particular product (e.g., the shape, the size, the colors, the texture, the brand name, the price, the logo, text appearing on the particular product, the shelf associated with the particular product, adjacent products in a planogram, the location within the retail store, etc.) (wherein trained machine learning models may be used for product detection/identification, the multi-object cluster detection model may be, for example, a product recognition model #2160, and product models may comprise an inference model, such as a clustering model, a deep neural network, a convolutional neural network, a recurrent neural network, etc.). Please also see Fig. 19-22, and read paragraph [0139-0140, 0328, 0352 and 0363]);
detecting a plurality of clusters of items associated with a plurality of object types within the image (Fig. 21. Paragraph [0335]-BRONICKI discloses visual data produced by the off-shelf cameras, may be used to identify the presence (or absence) of a product and/or determine the product type associated with the product (wherein “identifying a product” refers to the operations for detecting the presence/absence of a product, determining the type, condition, and quality of a product, determining the quantity of the same product type and determining the arrangement of the products on the retail shelf, etc.). In paragraph [0337]-BRONICKI discloses image data 2150 may be input to one or more product recognition models 2160. The output of product recognition models 2160 may include product type information 2190 regarding the products displayed on the retail shelf, and other information related to at least part of the products displayed on the retail shelf. Please also see Fig. 18-19, 22 and 29A-B, and read paragraph [0316-0326 and 360]), the plurality of clusters of items comprising a first cluster of items associated with a first object type and a second cluster of items associated with a second object type (Fig. 22. Paragraph [0362]-BRONICKI discloses processing device 2202 may analyze image data 2150 of a retail shelf and determine that the products on the shelf belong to a first product type. Processing device 2202 may analyze on-shelf sensor data 2100 associated with the retail shelf to determine that the items on the shelf belong to a second product type. Processing device 2200 may analyze additional information (i.e., information other than on-shelf sensor data 2100 and image data 2150) to determine the product type associated with the products. The additional information may include historic information regarding the retail shelf (e.g., the product types previously displayed on the retail shelf), place of the retail shelf, planogram information, product types of nearby products, shelf labels, height of the retail shelf, 3D images. Processing device 2202 may determine, based on the analysis of the additional information, that the products on the shelf belong to a third product type that differs from both the first and second product types. Please also see Fig. 11D, 18G, and 29A-B, and read paragraph [0256, 0320-0335, 0358, 0364]);
generating a labeled image comprising a plurality of indicators within an overlay associated with the image (Fig. 29A-B. Paragraph [0249]-BRONICKI discloses the images may be used to construct a store plan representing at least one product characteristic relative to a display structure. A store plan may include a planogram, a realogram, a three-dimensional (3D) model of the retail store, or the like, or a combination thereof. A realogram may be a virtual copy of shelves in a store. The images may be used to construct a store plan representing at least one product characteristic relative to a display structure associated with a retail environment (such as a store shelf or area of one or more shelves) (wherein the store plan may be used to determine planogram compliance and to detect/identify products using product characteristics, such as product presence/absence, product type, product size, product color, product shape, total and individual product quantities, product density, product homogeneity (i.e. the determination and count of one or more different unique product types), product combinations, product arrangements, product configurations, etc.). Please also see Fig. 11D, 18G, and 25, and read paragraph [0251, 283, 0319-0335, 0362, 0380, 0383 and 0480]), the plurality of indicators comprising a first cluster indicator associated with the first cluster of items within the image and a second cluster indicator associated with the second cluster of items within the image (Fig. 29A-B. Paragraph [0403]-BRONICKI discloses as shown in FIG. 27A, image 2700 may include at least part of a shelf 2702. Shelf 2702 may be a shelving unit at least partially dedicated to dairy products within a retail store. Shelf 2702 may include milk products 2710, cream products 2720, butter products 2730, and egg products 2740. Shelf 2702 and products 2710, 2720, 2730, and 2740 are provided. Server 135 may be configured to analyze image to determine a placement of products on the shelf (or part of the shelf). Image processing unit 130 may use various image analysis techniques to detect products 2710, 2720, 2730, and 2740 on shelf 2702 (wherein the generated labeled image may be, for example, the following: a planogram compliance image displayed in GUI 1140 shown in Fig. 11D, which contains visual indicators for correct, misplaced or missing items; a distribution map of a pegboard #20 shown in Fig. 18G, which contains shapes, colors and symbols, such as hatched symbols to indicate the quantity of items; an augmented virtual reality display area 2510 shown in Fig. 25, which contains destination overlay indicators (e.g. a symbol, icon, outline, color, image distortion, or other visual indicator) corresponding to a product name, location, and/or identifier, and/or; image #2700 shown in Fig. 29A, which contains visual indicators surrounding similar product groupings)). Please also see Fig. 11D, 18G, and 25, and read paragraph [0283, 0319-0335, 0362, 0380, 0383 and 0480]); and
identifying a number of unique clusters of items associated with a location corresponding to the plurality of clusters of items detected within the labeled image using the plurality of indicators (Fig. 29A-B. Paragraph [0213]-BRONICKI discloses determining, based on the at least one pattern associated with a detected product and the at least one empty space, at least one aspect of planogram compliance (wherein the aspect of planogram compliance may include the presence or absence of particular products (or brands), locations of products on the shelves, quantities of products within particular areas (e.g., identifying stacked or clustered products)). Further in paragraph [0214]-BRONICKI discloses the at least one aspect may include product homogeneity, and may further include counting occurrences where a product of the second type is placed on an area of the store shelf associated with the first type of product. By accessing a memory including base patterns (or any other type of pattern associated with product types, such as product models), the at least one processor may detect different products and product types. Such information may be used, for example, to monitor whether a certain region of a shelf includes an appropriate or intended product or product type. Such information may also be useful in determining whether products or product types have been mixed (e.g., product homogeneity). Regarding planogram compliance, detection of different products and their relative locations on a shelf may aid in determining whether a product homogeneity value, ratio, etc. has been achieved. At least one processor may count occurrences where a product of a second type is placed on an area of the store shelf associated with a product of a first type. Please also see Fig. 11D and 18G, and read paragraph [0178, 0319-0335, 0362, 0366, 0403-0408 and 0492-0498]).
Regarding claim 11, BRONICKI explicitly teaches the method of claim 8, BRONICKI further teaches further comprising:
validating item identification of at least one item associated with the location using the identified number of unique clusters (Fig. 11D. Paragraph [0233]-BRONICKI discloses server 135 may cause real-time automated alerts when products are out of shelf (or near out of shelf), when there are issues with planogram compliance, among others. GUI 1140 may include a first display area 1142 for showing a display of a video stream captured by output device 145C (e.g., a real-time display or a near real-time display) with augmented markings indicting a status of planogram compliance for each product (e.g., correct place, misplaced, not in planogram, empty, and so forth). GUI 1140 may also include a second display area 1144 for showing a summary of the planogram compliance for all the products identified in the video stream captured by output device 145C. Please also see Fig. 18G and 29A-B, and read paragraph [0249-0251, 0322-0324, 0362, 0266, 0403-0408]).
Regarding claim 12, BRONICKI explicitly teaches the method of claim 8, BRONICKI further teaches further comprising:
analyzing the image (Fig. 21. Paragraph [0110]-BRONSKI discloses embodiments include analyzing images to detect and identify different products. As used herein, the term “detecting a product” may broadly refer to determining an existence of the product. For example, the system may determine the existence of a plurality of distinct products displayed on a store shelf. By detecting the plurality of products, the system may acquire different details relative to the plurality of products (e.g., how many products on a store shelf are associated with a same product type), but it does not necessarily gain knowledge of the type of product. In contrast, the term “identifying a product” may refer to determining a unique identifier associated with a specific type of product that allows inventory managers to uniquely refer to each product type in a product catalogue) (wherein the image, may be for example, image data #2150 shown in Fig. 21). Please also read paragraph [0358-0359]) using a set of parameters for distinguishing unique items (Fig. 22. Paragraph [0118]-BRONICKI discloses the detection may occur through analysis of features in the image using an algorithm and stored data. The identification may occur through analysis of product features in the image according to stored product models. The term “product model” refers to any type of algorithm or stored product data that a processor may access or execute to enable the identification of a particular product associated with the product model. A product model may be used by server 135 to identify more than one type of products, such as, when two or more product models are used in combination to enable identification of a product. A first product model may be used by server 135 to identify a product category (such models may apply to multiple product types, e.g., shampoo, soft drinks, etc.), and a second product model may be used by server 135 to identify the product type, product identity, or other characteristics associated with a product. Please also read paragraph [0358-0364]), the set of parameters comprising size, shape, color, and proximity (Fig. 29A-B. Paragraph [0110]-BRONICKI discloses the term “identifying a product” may refer to determining a unique identifier associated with a specific brand of products the identification may be made based at least in part on visual characteristics of the product (e.g., size, shape, logo, text, color, etc.). In paragraph [0118]-BRONICKI discloses the product model may include a description of visual and contextual properties of the particular product (e.g., the shape, the size, the colors, the texture, the brand name, the price, the logo, text appearing on the particular product, the shelf associated with the particular product, adjacent products in a planogram, the location within the retail store, etc.). Please also read paragraph [0111, 0319-0335 and 0362]); and
detecting a first cluster comprising at least one instance of a first type of item and a second cluster comprising at least one instance of a second type of item (Fig. 22. Paragraph [0362]-BRONICKI discloses processing device 2202 may analyze image data 2150 of a retail shelf and determine that the products on the shelf belong to a first product type. Processing device 2202 may analyze on-shelf sensor data 2100 associated with the retail shelf to determine that the items on the shelf belong to a second product type. Processing device 2202 may determine, based on the analysis of the additional information, that the products on the shelf belong to a third product type that differs from both the first and second product types. The first product type (determined based on image data 2150) may be a salad bowl, while the second product type (determined based on on-shelf sensor data 2100) may be a cooking pot), wherein the first type of item has at least one of a different size, shape, and color than the second type of item (Fig. 29A, illustrates a store plan/map generated from, such as a planogram or shelf view, with detected/identified product groups #2710, #2720, #2730 and #2740. Paragraph [0403]-BRONICKI discloses image 2700 may include at least part of a shelf 2702. Shelf 2702 may include milk products 2710, cream products 2720, butter products 2730, and egg products 2740. Shelf 2702 and products 2710, 2720, 2730, and 2740. Server 135 may be configured to analyze image to determine a placement of products on the shelf (or part of the shelf). Image processing unit 130 may use various image analysis techniques to detect products 2710, 2720, 2730, and 2740 on shelf 2702 (wherein each product set has a different size, shape and/or color). Please also see Fig. 11D, 18G, and 29A-B, and read paragraph [0256, 0320-0335, 0358, 0364]).
Regarding claim 14, BRONICKI explicitly teaches the method of claim 8, BRONICKI further teaches further comprising:
mapping a current location of each unique cluster of items in the plurality of clusters of items to a location of a portion of the item storage structure (Fig. 27A. Paragraph [0402]-BRONICKI discloses a shelf may refer to any structure used for displaying products in a retail environment. In paragraph [0297]-BRONICKI discloses the above-described methods and systems regarding the “store shelf” or “shelf” may also be used to monitor and detect changes of the items hanging on the pegboards and peg-hooks (wherein an item storage structure is a shelf, pegboard and/or storage area, such as pegboard #20 and shelf #2702). Please also see Fig. 18A-G and read paragraph [0319, 0350 and 0452]) in a database (Fig. 1, #140 called a database. Paragraph [0118]-BRONICKI discloses as depicted in FIG. 1, server 135 may be coupled to one or more physical or virtual storage devices such as database 140. Server 135 may access database 140 to detect and/or identify products. In paragraph [0119]-BRONICKI discloses database 140 may include one or more memory devices that store data and instructions used to perform one or more features of the disclosed embodiments) using image data associated with the labeled image (Fig. 11D. Paragraph [0112]-BRONICKI discloses the system may retrieve different types of contextual information from captured image data and/or from other data sources. Contextual information may include a location of the product in the retail store. Further in paragraph [0233]-BRONICKI discloses shown in FIG. 11D, GUI 1140 may include a first display area 1142 for showing a display of a video stream captured by output device 145C (e.g., a real-time display or a near real-time display) with augmented markings indicting a status of planogram compliance for each product (e.g., correct place, misplaced, not in planogram, empty, and so forth). Please also see Fig. 19G, and 29A-B, and read paragraph [0283 and 0403-0407]).
Regarding claim 15, BRONICKI explicitly teaches one or more computer storage devices (Fig. 1-2, #226 called a memory device, respectively Paragraph [0129]. In paragraph [0112]-BRONICKI discloses system 100 may represent a computer-based system that may include memory devices.) having computer-executable instructions stored thereon (Fig. 2. Paragraph [0129]-BRONICKI discloses memory device 226 may include graphical user interface instructions 230; image processing instructions 232; sensor processing instructions 234; web browsing instructions 236; and other software instructions 238 to facilitate other processes and functions. Each of the above identified instructions and applications may correspond to a set of instructions for performing one or more functions. In paragraph [0130]-BRONICKI discloses memory device 226 may store database 140. Database 140 may include product type model data 240 (e.g., an image representation, a list of features, a model obtained by training machine learning algorithm using training examples, an artificial neural network, and more) that may be used to identify products in received images; contract-related data 242 (e.g., planograms, promotions data, etc.) that may be used to determine if the placement of products on the store shelves. Please also read paragraph [0123]), which, upon execution by a computer, cause the computer to perform operations (Fig. 2 #202 called a processing device. Paragraph [0122]-BRONICKI discloses processing device 202, shown in FIG. 2, may include at least one processor configured to execute computer programs, applications, methods, processes, or other software to execute particular instructions associated with embodiments described in the present disclosure. Further in paragraph [0123]-BRONICKI discloses the methods and processes disclosed herein may be performed by server 135 as a result of processing device 202 executing one or more sequences of one or more instructions contained in a non-transitory computer-readable storage medium. Please also read paragraph [0424]) comprising:
obtaining an image (Fig. 1. In paragraph [0114]-BRONICKI discloses system 100 may include an image processing unit 130 to execute the analysis of images captured by the one or more capturing devices 125. Image processing unit 130 may receive raw or processed data from capturing device 125 (wherein image processing unit 130 may use any suitable image analysis technique including, for example, object recognition, object detection, image segmentation, feature extraction, optical character recognition (OCR), object-based image analysis, shape region techniques, edge detection techniques, pixel-based detection, artificial neural networks, convolutional neural networks, etc.). In paragraph [0142]-BRONICKI discloses some non-limiting examples of image data may include images, grayscale images, color images, 2D images, 3D images, videos, 2D videos, 3D videos, frames, footages, data derived from other image data, and so forth (wherein an image may be, for example, image #1800 shown in Fig. 18A and image data #2150 shown in Fig. 21). Please also see Fig. 18A and 21-22, and read paragraph [0350]) of an item storage structure (Fig. 21. Paragraph [0350]-BRONICKI discloses the images may include a representation of at least part of the retail shelf and at least one of the one or more products (wherein an item storage structure may be, for example, a shelf, displays, containers, drawer units, bin units, baskets, pegboards, etc.). Please also see Fig. 18A and 33, and read paragraph [0297, 0402 and 0475)) captured by an image capture device (Fig. 1. Paragraph [0316]-BRONICKI discloses image sensor 1910 may be mounted (for example, to store shelves, walls, ceilings, floors, refrigerators, checkout stations, displays, dispensers, rods which may be connected to other objects in the retail store, and so forth) and configured to take images of the items hanging on peg-hooks 1810 (wherein an image capture device may be image capturing device #125, image sensor #1910 or #2245, for example). Further in paragraph [0350]-BRONICKI discloses processing device 2202 may use the data generated by image sensors to identify the products displayed on a retail shelf Processing device 2202 may then analyze the images (i.e., image data 2150 shown in FIG. 21) captured by image sensors 2245 to determine the presence (or absence) of one or more products on the retail shelf, the product type associated with the one or more products, and other information related to the one or more products. Please also see Fig. 19 and 21-22, and read paragraph [0145, 0325 and 0350]) using a multi-object cluster detection model (Fig. 1. In paragraph [0118]-BRONICKI discloses server 135 may access database 140 to detect and/or identify products. The detection may occur through analysis of features in the image using an algorithm and stored data. The identification may occur through analysis of product features in the image according to stored product models. The term “product model” refers to any type of algorithm or stored product data that a processor may access or execute to enable the identification of a particular product associated with the product model. The product model may include a description of visual and contextual properties of the particular product (e.g., the shape, the size, the colors, the texture, the brand name, the price, the logo, text appearing on the particular product, the shelf associated with the particular product, adjacent products in a planogram, the location within the retail store, etc.) (wherein trained machine learning models may be used for product detection/identification, the multi-object cluster detection model may be, for example, a product recognition model #2160, and product models may comprise an inference model, such as a clustering model, a deep neural network, a convolutional neural network, a recurrent neural network, etc.). Please also see Fig. 19-22, and read paragraph [0139-0140, 0328, 0352 and 0363]);
detecting a plurality of clusters of items of interest associated with a plurality of object types within the image (Fig. 21. Paragraph [0335]-BRONICKI discloses visual data produced by the off-shelf cameras, may be used to identify the presence (or absence) of a product and/or determine the product type associated with the product (wherein “identifying a product” refers to the operations for detecting the presence/absence of a product, determining the type, condition, and quality of a product, determining the quantity of the same product type and determining the arrangement of the products on the retail shelf, etc.). In paragraph [0337]-BRONICKI discloses image data 2150 may be input to one or more product recognition models 2160. The output of product recognition models 2160 may include product type information 2190 regarding the products displayed on the retail shelf, and other information related to at least part of the products displayed on the retail shelf. Please also see Fig. 18-19, 22 and 29A-B, and read paragraph [0316-0326 and 360]), the plurality of clusters of items comprising a first cluster of items associated with a first object type and a second cluster of items associated with a second object type (Fig. 22. Paragraph [0362]-BRONICKI discloses processing device 2202 may analyze image data 2150 of a retail shelf and determine that the products on the shelf belong to a first product type. Processing device 2202 may analyze on-shelf sensor data 2100 associated with the retail shelf to determine that the items on the shelf belong to a second product type. Processing device 2200 may analyze additional information (i.e., information other than on-shelf sensor data 2100 and image data 2150) to determine the product type associated with the products. The additional information may include historic information regarding the retail shelf (e.g., the product types previously displayed on the retail shelf), place of the retail shelf, planogram information, product types of nearby products, shelf labels, height of the retail shelf, 3D images. Processing device 2202 may determine, based on the analysis of the additional information, that the products on the shelf belong to a third product type that differs from both the first and second product types. Please also see Fig. 11D, 18G, and 29A-B, and read paragraph [0256, 0320-0335, 0358, 0364]);
generating a labeled image comprising a plurality of indicators within an overlay associated with the image (Fig. 29A-B. Paragraph [0249]-BRONICKI discloses the images may be used to construct a store plan representing at least one product characteristic relative to a display structure. A store plan may include a planogram, a realogram, a three-dimensional (3D) model of the retail store, or the like, or a combination thereof. A realogram may be a virtual copy of shelves in a store. The images may be used to construct a store plan representing at least one product characteristic relative to a display structure associated with a retail environment (such as a store shelf or area of one or more shelves) (wherein the store plan may be used to determine planogram compliance and to detect/identify products using product characteristics, such as product presence/absence, product type, product size, product color, product shape, total and individual product quantities, product density, product homogeneity (i.e. the determination and count of one or more different unique product types), product combinations, product arrangements, product configurations, etc.). Please also see Fig. 11D, 18G, and 25, and read paragraph [0251, 283, 0319-0335, 0362, 0380, 0383 and 0480]), the plurality of indicators comprising a first cluster indicator associated with the first cluster of items within the image and a second cluster indicator associated with the second cluster of items within the image (Fig. 29A-B. Paragraph [0403]-BRONICKI discloses as shown in FIG. 27A, image 2700 may include at least part of a shelf 2702. Shelf 2702 may be a shelving unit at least partially dedicated to dairy products within a retail store. Shelf 2702 may include milk products 2710, cream products 2720, butter products 2730, and egg products 2740. Shelf 2702 and products 2710, 2720, 2730, and 2740 are provided. Server 135 may be configured to analyze image to determine a placement of products on the shelf (or part of the shelf). Image processing unit 130 may use various image analysis techniques to detect products 2710, 2720, 2730, and 2740 on shelf 2702 (wherein the generated labeled image may be, for example, the following: a planogram compliance image displayed in GUI 1140 shown in Fig. 11D, which contains visual indicators for correct, misplaced or missing items; a distribution map of a pegboard #20 shown in Fig. 18G, which contains shapes, colors and symbols, such as hatched symbols to indicate the quantity of items; an augmented virtual reality display area 2510 shown in Fig. 25, which contains destination overlay indicators (e.g. a symbol, icon, outline, color, image distortion, or other visual indicator) corresponding to a product name, location, and/or identifier, and/or; image #2700 shown in Fig. 29A, which contains visual indicators surrounding similar product groupings)). Please also see Fig. 11D, 18G, and 25, and read paragraph [0283, 0319-0335, 0362, 0380, 0383 and 0480]); and
identifying a number of unique clusters of items associated with a location corresponding to the plurality of clusters of items detected within the labeled image using the plurality of indicators (Fig. 29A-B. Paragraph [0213]-BRONICKI discloses determining, based on the at least one pattern associated with a detected product and the at least one empty space, at least one aspect of planogram compliance (wherein the aspect of planogram compliance may include the presence or absence of particular products (or brands), locations of products on the shelves, quantities of products within particular areas (e.g., identifying stacked or clustered products)). Further in paragraph [0214]-BRONICKI discloses the at least one aspect may include product homogeneity, and may further include counting occurrences where a product of the second type is placed on an area of the store shelf associated with the first type of product. By accessing a memory including base patterns (or any other type of pattern associated with product types, such as product models), the at least one processor may detect different products and product types. Such information may be used, for example, to monitor whether a certain region of a shelf includes an appropriate or intended product or product type. Such information may also be useful in determining whether products or product types have been mixed (e.g., product homogeneity). Regarding planogram compliance, detection of different products and their relative locations on a shelf may aid in determining whether a product homogeneity value, ratio, etc. has been achieved. At least one processor may count occurrences where a product of a second type is placed on an area of the store shelf associated with a product of a first type. Please also see Fig. 11D and 18G, and read paragraph [0178, 0319-0335, 0362, 0366, 0403-0408 and 0492-0498]).
Regarding claim 17, BRONICKI explicitly teaches the one or more computer storage devices of claim 15, BRONICKI further teaches wherein the operations further comprise:
validating item identification of at least one item associated with the location using the identified number of unique clusters (Fig. 11D. Paragraph [0233]-BRONICKI discloses server 135 may cause real-time automated alerts when products are out of shelf (or near out of shelf), when there are issues with planogram compliance, among others. GUI 1140 may include a first display area 1142 for showing a display of a video stream captured by output device 145C (e.g., a real-time display or a near real-time display) with augmented markings indicting a status of planogram compliance for each product (e.g., correct place, misplaced, not in planogram, empty, and so forth). GUI 1140 may also include a second display area 1144 for showing a summary of the planogram compliance for all the products identified in the video stream captured by output device 145C. Please also see Fig. 18G and 29A-B, and read paragraph [0249, 0322-0324, 0362, 0403-0408 and 0492-0498]).
Regarding claim 18, BRONICKI explicitly teaches the one or more computer storage devices of claim 15, BRONICKI further teaches wherein the operations further comprise:
analyzing the image (Fig. 22. Paragraph [0110]-BRONSKI discloses embodiments include analyzing images to detect and identify different products. The term “detecting a product” may broadly refer to determining an existence of the product. The system may determine the existence of a plurality of distinct products displayed on a store shelf. By detecting the plurality of products, the system may acquire different details relative to the plurality of products (e.g., how many products on a store shelf are associated with a same product type), but it does not necessarily gain knowledge of the type of product. The term “identifying a product” may refer to determining a unique identifier associated with a specific type of product that allows inventory managers to uniquely refer to each product type in a product catalogue). Please also read paragraph [0358-0359]) using a set of parameters for distinguishing unique items (Fig. 22. Paragraph [0118]-BRONICKI discloses the detection may occur through analysis of features in the image using an algorithm and stored data. The identification may occur through analysis of product features in the image according to stored product models. The term “product model” refers to any type of algorithm or stored product data that a processor may access or execute to enable the identification of a particular product associated with the product model. A product model may be used by server 135 to identify more than one type of products, such as, when two or more product models are used in combination to enable identification of a product. A first product model may be used by server 135 to identify a product category (such models may apply to multiple product types, e.g., shampoo, soft drinks, etc.), and a second product model may be used by server 135 to identify the product type, product identity, or other characteristics associated with a product. Please also read paragraph [0358-0364]), the set of parameters comprising size, shape, color, and proximity (Fig. 29A-B. Paragraph [0110]-BRONICKI discloses the term “identifying a product” may refer to determining a unique identifier associated with a specific brand of products the identification may be made based at least in part on visual characteristics of the product (e.g., size, shape, logo, text, color, etc.). In paragraph [0118]-BRONICKI discloses the product model may include a description of visual and contextual properties of the particular product (e.g., the shape, the size, the colors, the texture, the brand name, the price, the logo, text appearing on the particular product, the shelf associated with the particular product, adjacent products in a planogram, the location within the retail store, etc.). Please also read paragraph [0111, 0319-0335 and 0362]); and
detecting a first cluster comprising at least one instance of a first type of item and a second cluster comprising at least one instance of a second type of item (), wherein the first type of item has at least one of a different size, shape, and color than the second type of item (Fig. 29A. Paragraph [0403]-BRONICKI discloses image 2700 may include at least part of a shelf 2702. Shelf 2702 may include milk products 2710, cream products 2720, butter products 2730, and egg products 2740. Shelf 2702 and products 2710, 2720, 2730, and 2740. Server 135 may be configured to analyze image to determine a placement of products on the shelf (or part of the shelf). Image processing unit 130 may use various image analysis techniques to detect products 2710, 2720, 2730, and 2740 on shelf 2702 (wherein each product set has a different size, shape and/or color). Please also see 11D and 18G, and read paragraph [0362]).
Regarding claim 20, BRONICKI explicitly teaches the one or more computer storage devices of claim 15, BRONICKI further teaches wherein the operations further comprise:
mapping a current location of each unique cluster of items in the plurality of clusters of items to a location of a portion of the item storage structure (Fig. 27A. Paragraph [0402]-BRONICKI discloses a shelf may refer to any structure used for displaying products in a retail environment. In paragraph [0297]-BRONICKI discloses the above-described methods and systems regarding the “store shelf” or “shelf” may also be used to monitor and detect changes of the items hanging on the pegboards and peg-hooks (wherein an item storage structure is a shelf, pegboard and/or storage area, such as pegboard #20 and shelf #2702). Please also see Fig. 18A-G and read paragraph [0319, 0350 and 0452]) in a database (Fig. 1, #140 called a database. Paragraph [0118]-BRONICKI discloses as depicted in FIG. 1, server 135 may be coupled to one or more physical or virtual storage devices such as database 140. Server 135 may access database 140 to detect and/or identify products. In paragraph [0119]-BRONICKI discloses database 140 may include one or more memory devices that store data and instructions used to perform one or more features of the disclosed embodiments) using image data associated with the labeled image (Fig. 11D. Paragraph [0112]-BRONICKI discloses the system may retrieve different types of contextual information from captured image data and/or from other data sources. Contextual information may include a location of the product in the retail store. Further in paragraph [0233]-BRONICKI discloses shown in FIG. 11D, GUI 1140 may include a first display area 1142 for showing a display of a video stream captured by output device 145C (e.g., a real-time display or a near real-time display) with augmented markings indicting a status of planogram compliance for each product (e.g., correct place, misplaced, not in planogram, empty, and so forth). Please also see Fig. 19G, and 29A-B, and read paragraph [0283, 0320-0324, 0362, 0366, 0403-0407]).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-3, 9-10 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over BRONICKI (US 20220261831 A1), hereinafter referenced as BRONICKI in view of CHAUBARD et al. (US 20200005225 A1), hereinafter referenced as CHAUBARD.
Regarding claim 2, BRONICKI explicitly teaches the system of claim 1, BRONICKI further teaches wherein the instructions are further operative to:
determine whether the identified number of unique clusters of items (Fig. 18G. Paragraph [0317]-BRONICKI discloses different peg-hooks 1810 may be pre-assigned to and reserved for hanging different product types. Memory device 1914 may store a predetermined peg-hook map indicating which peg-hooks 1810 are used for hanging what product types (i.e., corresponding relationship between peg-hooks 1810 and the product types). Based on the peg-hook map, processing device 1902 may determine the product type of the items hanging on each peg-hook. In paragraph [0322]-BRONICKI discloses processing device 1902 may generate a distribution map of the items hanging on a pegboard 1800. The distribution map may include information showing the quantities and product types of the items hanging on peg-hooks 1810. FIG. 18G shows an exemplary distribution map of the items hanging on pegboard #20 in retail store 105. Processing device 1902 may generate the distribution map based on the product type and item quantity determined for each peg-hook 1810) equals an expected number of unique clusters of items (Fig. 18G. Paragraph [0324]-BRONICKI disclose processing device 1902 may analyze the one or more captured images, for example the processing device may extract features from the captured images, compare the extracted features to the product recognition models, and may determine the product type or quantity based on the comparison. Processing device 1902 may further determine whether the product type or quantity complies with the planogram (e.g., whether the item added to the peg-hook 1810 have the correct product type, or whether the quantity of the items remaining on the peg-hook 1810 is too low). Please also see Fig. 11D and 29A-B, and read paragraph [0403-0408, 0414-0415, and 0492-0498]); and
Although BRONICKI explicitly teaches responsive to the identified number of unique clusters of items (Fig. 18G. Paragraph [0332]-BRONICKI discloses processing device 1902 may determine the product type of the added or removed items. Processing device 1902 may also determine the quantity of the added or removed items. Processing device 1902 may further cause one or more other actions based on the determined quantity and/or product type), generate a notification indicating at least one incorrectly placed item at the location (Fig. 18G. Paragraph [0332]-BRONICKI discloses if it is determined that the quantity of the items hanging on first peg-hook 1810 is low, processing device 1902 may generate a notification indicating a need to restock first peg-hook 1810. If it is determined that the items added to first peg-hook 1810 are not the product type designated for first peg-hook 1810, processing device 1902 may provide a warning regarding the mismatch and prompt a user to move the items to other peg-hooks 1810 that are associated with the correct product type. Please also read paragraph [0233]).
BRONICKI fails to explicitly teach responsive to the identified number of unique clusters of items exceeding the expected number of unique clusters of items, generate a notification.
However, CHAUBARD explicitly teaches responsive to the identified number of unique clusters of items (Fig. 4. Paragraph [0027]-CHAUBARD discloses the out-of-stock detection system obtains 405 first image data of a product display area in a store captured by one or more cameras 100 (wherein the out-of-stock system comprises a neural network trained via Stochastic Gradient Descent based on the templates images). Based on the position of the product labels 115, the system generates 410 a bounding box 120. The system then identifies 415 the product 110 associated with each bounding box 120. In paragraph [0032]-CHAUBARD discloses the out-of-stock detection system identifies a number of individual product bounding boxes within each bounding box 120 identified for the product based on the product label 115 (wherein product identification may include extracting actual product information and matching products with a planogram that contains product bounding boxes and visual indicators). With the product 110 identified for each bounding box 120, the system stores 420 the identified product 110 with the corresponding bounding box 120) exceeding the expected number of unique clusters of items (Fig. 4. Paragraph [0033]-CHAUBARD discloses the product-detection model generates individual product bounding boxes 125 for each bounding box 120 and compares the number of individual products 110 within each display location from the image to the number of product associated with the fully stocked shelf unit of the template image and determines that the number of “Soda” products identified from the image does not match the number associated with the template image for shelf unit 105b), generate a notification (Fig. 18G. Paragraph [0033]-CHAUBARD discloses in response to determining that the number of individual products within the display location does not match the number of product associated with the fully stocked product display area of the template image, the out-of-stock detection system generates an out-of-stock notification for “soda” to a store client device. In paragraph [0036]-CHAUBARD discloses the cameras 100 capture an image an hour, for example, and for each image, for loops over each template, crops the region corresponding to each bounding box and forward passes that template over the captured image to predict whether the product corresponding to each bounding box is in stock, out of stock, low (e.g., 1-3 products left for that SKU), Planogram compliance issue (e.g., wrong product is there). This list of action items is provided via an application associated with the out-of-stock detection system in the store in “Stocker Action Tools” which could be delivered to the store associates via text or an application notification).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of BRONICKI of having a system for multi-object cluster detection, wherein the instructions are further operative to: determine whether the identified number of unique clusters of items equals an expected number of unique clusters of items, with the teachings of CHAUBARD of having and responsive to the identified number of unique clusters of items exceeding the expected number of unique clusters of items, generate a notification indicating at least one incorrectly placed item at the location.
Wherein having BRONICKI’s system having and responsive to the identified number of unique clusters of items exceeding the expected number of unique clusters of items, generate a notification indicating at least one incorrectly placed item at the location.
The motivation behind the modification would have been to obtain a system that improves the detection of planogram compliance in retail images, since both BRONICKI and CHAUBARD concern retail inventory management. Wherein BRONICKI’s systems and methods provide improved efficiency, convenience, and functionality over prior art for planogram compliance and inventory management systems., while CHAUBARD systems and methods provide improvements for detection of planogram compliance. Please see BRONICKI (US 20220261831 A1), Abstract and Paragraph [0370 and 0401] and CHAUBARD et al. (US 20200005225 A1), Abstract and Paragraph [0014].
Regarding claim 3, BRONICKI explicitly teaches the system of claim 2, although BRONICKI further teaches wherein the instructions are further operative to: responsive to the expected number of unique clusters of items, generate an item out notification indicating at least one out-of-stock item.
BRONICKI fails to explicitly teach wherein the instructions are further operative to: responsive to the expected number of unique clusters of items exceeding the identified number of unique clusters of items, generate an item out notification indicating at least one out-of-stock item.
However, CHAUBARD explicitly teaches wherein the instructions are further operative to: responsive to the expected number of unique clusters of items (Fig. 4. Paragraph [0027]-CHAUBARD discloses the out-of-stock detection system obtains 405 first image data of a product display area in a store captured by one or more cameras 100 (wherein the out-of-stock system comprises a neural network trained via Stochastic Gradient Descent based on the templates images). Based on the position of the product labels 115, the system generates 410 a bounding box 120. The system then identifies 415 the product 110 associated with each bounding box 120. In paragraph [0032]-CHAUBARD discloses the out-of-stock detection system identifies a number of individual product bounding boxes within each bounding box 120 identified for the product based on the product label 115 (wherein product identification may include extracting actual product information and matching products with a planogram that contains product bounding boxes and visual indicators). With the product 110 identified for each bounding box 120, the system stores 420 the identified product 110 with the corresponding bounding box 120) exceeding the identified number of unique clusters of items (Fig. 4. Paragraph [0033]-CHAUBARD discloses the product-detection model generates individual product bounding boxes 125 for each bounding box 120 and compares the number of individual products 110 within each display location from the image to the number of product associated with the fully stocked shelf unit of the template image and determines that the number of “Soda” products identified from the image does not match the number associated with the template image for shelf unit 105b. In paragraph [0033]-CHAUBARD discloses in response to determining that the number of individual products within the display location does not match the number of product associated with the fully stocked product display area of the template image, the out-of-stock detection system generates an out-of-stock notification for “soda” to a store client device), generate an item out notification indicating at least one out-of-stock item (Fig. 18G. Paragraph In paragraph [0036]-CHAUBARD discloses the cameras 100 capture an image an hour, for example, and for each image, for loops over each template, crops the region corresponding to each bounding box and forward passes that template over the captured image to predict whether the product corresponding to each bounding box is in stock, out of stock, low (e.g., 1-3 products left for that SKU), Planogram compliance issue (e.g., wrong product is there). This list of action items is provided via an application associated with the out-of-stock detection system in the store in “Stocker Action Tools” which could be delivered to the store associates via text or an application notification).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of BRONICKI of having a system for multi-object cluster detection, with the teachings of CHAUBARD of having wherein the instructions are further operative to: responsive to the expected number of unique clusters of items exceeding the identified number of unique clusters of items, generate an item out notification indicating at least one out-of-stock item.
Wherein having BRONICKI’s system having wherein the instructions are further operative to: responsive to the expected number of unique clusters of items exceeding the identified number of unique clusters of items, generate an item out notification indicating at least one out-of-stock item.
The motivation behind the modification would have been to obtain a system that improves the detection of planogram compliance in retail images, since both BRONICKI and CHAUBARD concern retail inventory management. Wherein BRONICKI’s systems and methods provide improved efficiency, convenience, and functionality over prior art for planogram compliance and inventory management systems., while CHAUBARD systems and methods provide improvements for detection of planogram compliance. Please see BRONICKI (US 20220261831 A1), Abstract and Paragraph [0370 and 0401] and CHAUBARD et al. (US 20200005225 A1), Abstract and Paragraph [0014].
Regarding claim 9, BRONICKI explicitly teaches the method of claim 8, BRONICKI further teaches further comprising:
determining whether the identified number of unique clusters of items (Fig. 18G. Paragraph [0317]-BRONICKI discloses different peg-hooks 1810 may be pre-assigned to and reserved for hanging different product types. Memory device 1914 may store a predetermined peg-hook map indicating which peg-hooks 1810 are used for hanging what product types (i.e., corresponding relationship between peg-hooks 1810 and the product types). Based on the peg-hook map, processing device 1902 may determine the product type of the items hanging on each peg-hook. In paragraph [0322]-BRONICKI discloses processing device 1902 may generate a distribution map of the items hanging on a pegboard 1800. The distribution map may include information showing the quantities and product types of the items hanging on peg-hooks 1810. FIG. 18G shows an exemplary distribution map of the items hanging on pegboard #20 in retail store 105. Processing device 1902 may generate the distribution map based on the product type and item quantity determined for each peg-hook 1810) falls within a threshold range of unique clusters of items (Fig. 18G. Paragraph [0324]-BRONICKI disclose processing device 1902 may analyze the one or more captured images, for example the processing device may extract features from the captured images, compare the extracted features to the product recognition models, and may determine the product type or quantity based on the comparison. Processing device 1902 may further determine whether the product type or quantity complies with the planogram (e.g., whether the item added to the peg-hook 1810 have the correct product type, or whether the quantity of the items remaining on the peg-hook 1810 is too low). Please also see Fig. 11D and 29A-B, and read paragraph [0233, 0403-0408, 0414-0415, and 0492-0498]); and
Although BRONICKI explicitly teaches responsive to the identified number of unique clusters of items (Fig. 18G. Paragraph [0332]-BRONICKI discloses processing device 1902 may determine the product type of the added or removed items. Processing device 1902 may also determine the quantity of the added or removed items. Processing device 1902 may further cause one or more other actions based on the determined quantity and/or product type), generating a notification indicating at least one incorrectly placed item at the location (Fig. 18G. Paragraph [0332]-BRONICKI discloses if it is determined that the quantity of the items hanging on first peg-hook 1810 is low, processing device 1902 may generate a notification indicating a need to restock first peg-hook 1810. If it is determined that the items added to first peg-hook 1810 are not the product type designated for first peg-hook 1810, processing device 1902 may provide a warning regarding the mismatch and prompt a user to move the items to other peg-hooks 1810 that are associated with the correct product type. Please also see Fig. 11D and 29A-B, and read paragraph [0120, 0233, 0369, 0487 and 0492-0498]).
BRONICKI fails to explicitly teach and responsive to the identified number of unique clusters of items exceeding the threshold range of unique clusters of items, generating a notification indicating at least one incorrectly placed item at the location.
However, CHAUBARD explicitly teaches and responsive to the identified number of unique clusters of items (Fig. 4. Paragraph [0027]-CHAUBARD discloses the out-of-stock detection system obtains 405 first image data of a product display area in a store captured by one or more cameras 100 (wherein the out-of-stock system comprises a neural network trained via Stochastic Gradient Descent based on the templates images). Based on the position of the product labels 115, the system generates 410 a bounding box 120. The system then identifies 415 the product 110 associated with each bounding box 120. In paragraph [0032]-CHAUBARD discloses the out-of-stock detection system identifies a number of individual product bounding boxes within each bounding box 120 identified for the product based on the product label 115 (wherein product identification may include extracting actual product information and matching products with a planogram that contains product bounding boxes and visual indicators). With the product 110 identified for each bounding box 120, the system stores 420 the identified product 110 with the corresponding bounding box 120) exceeding the threshold range of unique clusters of items (Fig. 4. Paragraph [0033]-CHAUBARD discloses the product-detection model generates individual product bounding boxes 125 for each bounding box 120 and compares the number of individual products 110 within each display location from the image to the number of product associated with the fully stocked shelf unit of the template image and determines that the number of “Soda” products identified from the image does not match the number associated with the template image for shelf unit 105b. In paragraph [0033]-CHAUBARD discloses in response to determining that the number of individual products within the display location does not match the number of product associated with the fully stocked product display area of the template image, the out-of-stock detection system generates an out-of-stock notification for “soda” to a store client device), generating a notification indicating at least one incorrectly placed item at the location (Fig. 18G. Paragraph [0036]-CHAUBARD discloses for each image, for loops over each template, crops the region corresponding to each bounding box and forward passes that template over the captured image to predict whether the product corresponding to each bounding box is in stock, out of stock, low (e.g., 1-3 products left for that SKU), Planogram compliance issue (e.g., wrong product is there). This list of action items is provided via an application associated with the out-of-stock detection system in the store in “Stocker Action Tools” which could be delivered to the store associates via text or an application notification. Therefore, it would be obvious to a person of ordinary skill in the art to generate a notification for an incorrectly placed item when the expected number of unique cluster items exceeds the expected number. CHAUBARD teaches assessing planogram compliance, and determining whether the number of detected item bounding boxes match an expected number and whether items are either misplaced or out-of-stock. Moreover, out-of-stock and misplacement events are often related and/or simultaneously triggered when the wrong item is misplaced and results in the number of items exceeding an expected number. Thus, it would be obvious to generate misplaced notification when the number of items exceeds the expected amount because this would improve the ability to assess product distribution and planogram compliance).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of BRONICKI of having a method for multi-object cluster detection with improved accuracy, further comprising: determining whether the identified number of unique clusters of items falls within a threshold range of unique clusters of items, with the teachings of CHAUBARD of having and responsive to the identified number of unique clusters of items exceeding the threshold range of unique clusters of items, generating a notification.
Wherein having BRONICKI’s method having and responsive to the identified number of unique clusters of items exceeding the threshold range of unique clusters of items, generating a notification indicating at least one incorrectly placed item at the location.
The motivation behind the modification would have been to obtain a method that improves the detection of planogram compliance in retail images, since both BRONICKI and CHAUBARD concern retail inventory management. Wherein BRONICKI’s systems and methods provide improved efficiency, convenience, and functionality over prior art for planogram compliance and inventory management systems., while CHAUBARD systems and methods provide improvements for detection of planogram compliance. Please see BRONICKI (US 20220261831 A1), Abstract and Paragraph [0370 and 0401] and CHAUBARD et al. (US 20200005225 A1), Abstract and Paragraph [0014].
Regarding claim 10, BRONICKI in view of CHAUBARD explicitly teaches the method of claim 9, further comprising:
responsive to the identified number of unique clusters of items falling below the threshold range (Fig. 18G. Paragraph [0332]-BRONICKI discloses processing device 1902 may determine the product type of the added or removed items. Processing device 1902 may also determine the quantity of the added or removed items. Processing device 1902 may further cause one or more other actions based on the determined quantity and/or product type), generating an alert indicating at least one out-of-stock item (Fig. 18G. Paragraph [0332]-BRONICKI discloses if it is determined that the quantity of the items hanging on first peg-hook 1810 is low, processing device 1902 may generate a notification indicating a need to restock first peg-hook 1810. If it is determined that the items added to first peg-hook 1810 are not the product type designated for first peg-hook 1810, processing device 1902 may provide a warning regarding the mismatch and prompt a user to move the items to other peg-hooks 1810 that are associated with the correct product type. Please also see Fig. 11D and 29A-B, and read paragraph [0233, 0403-0408, 0414-0415, and 0492-0498]).
Regarding claim 16, BRONICKI explicitly teaches the one or more computer storage devices of claim 15, BRONICKI fails to explicitly teach wherein the operations further comprise:
responsive to the identified number of unique clusters of items exceeding an expected number of unique clusters of items, generating a first notification indicating at least one incorrectly placed item at the location; and responsive to the expected number of unique clusters of items exceeding the identified number of unique clusters of items, generating a second notification indicating at least one out-of-stock item.
However, CHAUBARD explicitly teaches wherein the operations further comprise:
responsive to the identified number of unique clusters of items (Fig. 4. Paragraph [0027]-CHAUBARD discloses the out-of-stock detection system obtains 405 first image data of a product display area in a store captured by one or more cameras 100 (wherein the out-of-stock system comprises a neural network trained via Stochastic Gradient Descent based on the templates images). Based on the position of the product labels 115, the system generates 410 a bounding box 120. The system then identifies 415 the product 110 associated with each bounding box 120. In paragraph [0032]-CHAUBARD discloses the out-of-stock detection system identifies a number of individual product bounding boxes within each bounding box 120 identified for the product based on the product label 115 (wherein product identification may include extracting actual product information and matching products with a planogram that contains product bounding boxes and visual indicators). With the product 110 identified for each bounding box 120, the system stores 420 the identified product 110 with the corresponding bounding box 120) exceeding an expected number of unique clusters of items (Fig. 4. Paragraph [0033]-CHAUBARD discloses the product-detection model generates individual product bounding boxes 125 for each bounding box 120 and compares the number of individual products 110 within each display location from the image to the number of product associated with the fully stocked shelf unit of the template image and determines that the number of “Soda” products identified from the image does not match the number associated with the template image for shelf unit 105b), generating a first notification indicating at least one incorrectly placed item at the location (Fig. 18G. Paragraph [0036]-CHAUBARD discloses for each image, for loops over each template, crops the region corresponding to each bounding box and forward passes that template over the captured image to predict whether the product corresponding to each bounding box is in stock, out of stock, low (e.g., 1-3 products left for that SKU), Planogram compliance issue (e.g., wrong product is there). This list of action items is provided via an application associated with the out-of-stock detection system in the store in “Stocker Action Tools” which could be delivered to the store associates via text or an application notification. Therefore, it would be obvious to a person of ordinary skill in the art to generate a notification for an incorrectly placed item when the expected number of unique cluster items exceeds the expected number. CHAUBARD teaches assessing planogram compliance, and determining whether the number of detected item bounding boxes match an expected number and whether items are either misplaced or out-of-stock. Moreover, out-of-stock and misplacement events are often related and/or simultaneously triggered when the wrong item is misplaced and results in the number of items exceeding an expected number. Thus, it would be obvious to generate misplaced notification when the number of items exceeds the expected amount because this would improve the ability to assess product distribution and planogram compliance); and
responsive to the expected number of unique clusters of items exceeding the identified number of unique clusters of items, generating a second notification indicating at least one out-of-stock item (Fig. 4. Paragraph [0033]-CHAUBARD discloses the product-detection model generates individual product bounding boxes 125 for each bounding box 120 and compares the number of individual products 110 within each display location from the image to the number of product associated with the fully stocked shelf unit of the template image and determines that the number of “Soda” products identified from the image does not match the number associated with the template image for shelf unit 105b. Further in paragraph [0036]-CHAUBARD discloses for each image, predict whether the product corresponding to each bounding box is in stock, out of stock, low (e.g., 1-3 products left for that SKU), Planogram compliance issue (e.g., wrong product is there). This list of action items is provided via an application associated with the out-of-stock detection system in the store in “Stocker Action Tools” which could be delivered to the store associates via text or an application notification).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of BRONICKI of having one or more computer storage devices having computer-executable instructions stored thereon, with the teachings of CHAUBARD of having wherein the operations further comprise: responsive to the identified number of unique clusters of items exceeding an expected number of unique clusters of items, generating a first notification indicating at least one incorrectly placed item at the location; and responsive to the expected number of unique clusters of items exceeding the identified number of unique clusters of items, generating a second notification indicating at least one out-of-stock item.
Wherein having BRONICKI’s computer storage devices having wherein the operations further comprise: responsive to the identified number of unique clusters of items exceeding an expected number of unique clusters of items, generating a first notification indicating at least one incorrectly placed item at the location; and responsive to the expected number of unique clusters of items exceeding the identified number of unique clusters of items, generating a second notification indicating at least one out-of-stock item.
The motivation behind the modification would have been to obtain computer storage devices that improves the detection of planogram compliance in retail images, since both BRONICKI and CHAUBARD concern retail inventory management. Wherein BRONICKI’s systems and methods provide improved efficiency, convenience, and functionality over prior art for planogram compliance and inventory management systems., while CHAUBARD systems and methods provide improvements for detection of planogram compliance. Please see BRONICKI (US 20220261831 A1), Abstract and Paragraph [0370 and 0401] and CHAUBARD et al. (US 20200005225 A1), Abstract and Paragraph [0014].
Claims 6, 13 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over BRONICKI (US 20220261831 A1), hereinafter referenced as BRONICKI in view of CALHOON et al. (US 20200234394 A1), hereinafter referenced as CALHOON.
Regarding claim 6, BRONICKI explicitly teaches the system of claim 1, BRONICKI further teaches wherein the instructions are further operative to:
detect (Fig. 22. Paragraph [0110]-BRONSKI discloses embodiments include analyzing images to detect and identify different products. As used herein, the term “detecting a product” may broadly refer to determining an existence of the product. For example, the system may determine the existence of a plurality of distinct products displayed on a store shelf. By detecting the plurality of products, the system may acquire different details relative to the plurality of products (e.g., how many products on a store shelf are associated with a same product type), but it does not necessarily gain knowledge of the type of product. In contrast, the term “identifying a product” may refer to determining a unique identifier associated with a specific type of product that allows inventory managers to uniquely refer to each product type in a product catalogue). Please also read paragraph [0358-0359]), by the multi-object cluster detection model (Fig. 1. Paragraph [0118]-BRONICKI discloses the detection may occur through analysis of features in the image using an algorithm and stored data. The identification may occur through analysis of product features in the image according to stored product models. The term “product model” refers to any type of algorithm or stored product data that a processor may access or execute to enable the identification of a particular product associated with the product model. The product model may include a description of visual and contextual properties of the particular product (e.g., the shape, the size, the colors, the texture, the brand name, the price, the logo, text appearing on the particular product, the shelf associated with the particular product, adjacent products in a planogram, the location within the retail store, etc.). A first product model may be used by server 135 to identify a product category (such models may apply to multiple product types, e.g., shampoo, soft drinks, etc.), and a second product model may be used by server 135 to identify the product type, product identity, or other characteristics associated with a product. Please also read paragraph [0111, 0319-0335 and 0362]), a cluster of items comprising at least one instance of an item (Fig. 29A. Paragraph [0362]-BRONICKI discloses processing device 2202 may analyze image data 2150 of a retail shelf and determine that the products on the shelf belong to a first product type. Processing device 2202 may analyze on-shelf sensor data 2100 associated with the retail shelf to determine that the items on the shelf belong to a second product type. Processing device 2200 may analyze additional information (i.e., information other than on-shelf sensor data 2100 and image data 2150) to determine the product type associated with the products. The additional information may include, but is not limited to, historic information regarding the retail shelf (e.g., the product types previously displayed on the retail shelf), place of the retail shelf in a retail store (e.g., whether the retail shelf is located in an aisle for condiments or in a section of the store for home cleaning products), planogram information, product types of nearby products, shelf labels, height of the retail shelf, 3D images), wherein each instance of an item in the cluster of items is located within a predetermined proximity to at least one other item having similar size, shape, and color (Fig. 29A. Paragraph [0403]-BRONICKI discloses image 2700 may include at least part of a shelf 2702. Shelf 2702 may include milk products 2710, cream products 2720, butter products 2730, and egg products 2740. Shelf 2702 and products 2710, 2720, 2730, and 2740. Server 135 may be configured to analyze image to determine a placement of products on the shelf (or part of the shelf). Image processing unit 130 may use various image analysis techniques to detect products 2710, 2720, 2730, and 2740 on shelf 2702 (wherein each item in a product set is located within a close proximity to other products with similar attributes (e.g. shape, size and color). Please also see Fig. 11D and 18G, and read paragraph [0319-0335]);
identify a location tag on at least a portion of the item storage structure associated with the detected cluster of items (Fig. 1. Paragraph [0111]-BRONICKI discloses the characteristic of the product may assist in classifying and identifying the product. The characteristic of the product may be associated a logo or text associated with the product (e.g., on a product label), and more. In paragraph [0477]-BRONICKI discloses product label 3306 may be attached to a part of a shelf or to other objects (such as displays, containers, etc.) in the retail store that is associated with the product (e.g., stocks the product). A product label may include any combination of a logo, a barcode, a quick response (QR) code, a word, a character, a shape, a color, a placement location of the label relative to the product, or any other information detectable by an image sensor);
store the determined location of the cluster of items (Fig. 2. Paragraph [0112]-BRONICKI discloses the system may retrieve different types of contextual information from captured image data and/or from other data sources. Other examples of types of contextual information may include a location of the product in the retail store. Please also read paragraph [0118, 0213 and 0233]) in a database (Fig. 2, #140 called a database. Paragraph [0130]-BRONICKI discloses memory device 226 may store database 140. Database 140 may include product type model data 240 (e.g., an image representation, a list of features, a model obtained by training machine learning algorithm using training examples, an artificial neural network, and more) that may be used to identify products in received images; contract-related data 242 (e.g., planograms, promotions data, etc.) that may be used to determine the placement of products on the store shelves. Please also read paragraph [0123, 0279-0280]).
BRONICKI fails to explicitly teach determine the location of the cluster of items using the location tag; and
However, CALHOON explicitly teaches determine the location of the cluster of items (Fig. 3A. Paragraph [0011]-CALHOON discloses FIG. 2A shows an image of a supermarket shelf. In paragraph [0082]-CALHOON discloses imagery captured by a robot can be analyzed to recognize a shelved product by its visual appearance. Image fingerprints can be derived from product artwork, and compared against reference fingerprint data for each item in the store (or for each item in the aisle, or shelving unit, where the robot is capturing imagery), to find a match, indicating the product identity (wherein other visual product recognition techniques can also be used, including color histograms, trained neural networks, a shelf label, RFID identification). In paragraph [0049]-CALHOON discloses referring to FIG. 3A, the left edge of the Campbell's Chicken Noodle Soup shelf label serves as the left edge of an overlaid dashed bounding box that extends eight inches to the right, and the full height of the shelf space. Further in paragraph [0069]-CALHOON discloses overlay data or colors can be projected onto shelved products) using the location tag (Fig. 3A. Paragraph [0096]-CALHOON discloses the identifiers encoded by the watermark blocks can be associated with metadata using a data structure, such as a table or database—stored in the robot memory or remotely (e.g., in a store server memory, or in the cloud), or both. In paragraph [0098]-CALHOON discloses traditional planogram data is all in the table. With such a data structure, and the captured imagery, the robot can sense which products are out of place, or out of stock, and generate a punch list for attention by a store clerk to correct same. In paragraph [0120]-CALHOON discloses location markings can be provided on store shelves. Exemplary markings are watermark or barcode patterns that convey planogram coordinates, e.g., a shelving unit number, a shelf number, and/or a distance from the left edge of the shelving unit. In paragraph [0121]-CALHOON discloses markings can take the form of adhesive labels. In paragraph [0124]-CALHOON discloses in addition to identifying planogram location, such markings can also aid in understanding pose (including scale) of products resting on the shelf).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of BRONICKI of having a system for multi-object cluster detection, wherein the instructions are further operative to: detect, by the multi-object cluster detection model, a cluster of items comprising at least one instance of an item, wherein each instance of an item in the cluster of items is located within a predetermined proximity to at least one other item having similar size, shape, and color; identify a location tag on at least a portion of the item storage structure associated with the detected cluster of items; and store the determined location of the cluster of items in a database, with the teachings of CALHOON of having determine the location of the cluster of items using the location tag.
Wherein having BRONICKI’s system having determine the location of the cluster of items using the location tag.
The motivation behind the modification would have been to obtain a system that improves the detection of planogram compliance in retail images, since both BRONICKI and CALHOON concern retail inventory management. Wherein BRONICKI’s systems and methods provide improved efficiency, convenience, and functionality over prior art for planogram compliance and inventory management systems, while CALHOON systems and methods provide improvements for detection of multiple instances of objects and planogram compliance. Please see BRONICKI (US 20220261831 A1), Abstract and Paragraph [0370 and 0401] and CALHOON et al. (US 20200234394 A1), Abstract and Paragraph [0003-0006, 0067 and 0098].
Regarding claim 13, BRONICKI explicitly teaches the method of claim 8, BRONICKI further teaches further comprising:
detecting, by the multi-object cluster detection model, a cluster of items having similar size, shape, and color, wherein each instance of an item in the cluster of items is located within a predetermined proximity to at least one other item having the similar size, shape, and color (Fig. 29A-B. Paragraph [0403]-BRONICKI discloses image 2700 may include at least part of a shelf 2702. Shelf 2702 may include milk products 2710, cream products 2720, butter products 2730, and egg products 2740. Shelf 2702 and products 2710, 2720, 2730, and 2740. Server 135 may be configured to analyze image to determine a placement of products on the shelf (or part of the shelf). Image processing unit 130 may use various image analysis techniques to detect products 2710, 2720, 2730, and 2740 on shelf 2702 (wherein each item in a product set is located within a close proximity to other products with similar attributes (e.g. shape, size and color). Please also see Fig. 11D and 18G, and read paragraph [0110, 0118, and 0319-0335]);
and storing the location of the detected cluster of items (Fig. 2. Paragraph [0112]-BRONICKI discloses the system may retrieve different types of contextual information from captured image data and/or from other data sources. Other examples of types of contextual information may include a location of the product in the retail store. Please also read paragraph [0118, 0213 and 0233]) in a database (Fig. 2, #140 called a database. Paragraph [0130]-BRONICKI discloses memory device 226 may store database 140. Database 140 may include product type model data 240 (e.g., an image representation, a list of features, a model obtained by training machine learning algorithm using training examples, an artificial neural network, and more) that may be used to identify products in received images; contract-related data 242 (e.g., planograms, promotions data, etc.) that may be used to determine the placement of products on the store shelves. Please also read paragraph [0123, 0279-0280]);
identifying a location tag on at least a portion of the item storage structure associated with the cluster of items (Fig. 1. Paragraph [0111]-BRONICKI discloses the characteristic of the product may assist in classifying and identifying the product. The characteristic of the product may be associated a logo or text associated with the product (e.g., on a product label), and more. In paragraph [0477]-BRONICKI discloses product label 3306 may be attached to a part of a shelf or to other objects (such as displays, containers, etc.) in the retail store that is associated with the product (e.g., stocks the product). A product label may include any combination of a logo, a barcode, a quick response (QR) code, a word, a character, a shape, a color, a placement location of the label relative to the product, or any other information detectable by an image sensor).
BRONICKI fails to explicitly teach determining the location of the cluster of items using the location tag.
However, CALHOON explicitly teaches determining the location of the cluster of items (Fig. 3A. Paragraph [0011] FIG. 2A shows an image of a supermarket shelf. In paragraph [0082]-CALHOON discloses imagery captured by a robot can be analyzed to recognize a shelved product by its visual appearance. Image fingerprints can be derived from product artwork, and compared against reference fingerprint data for each item in the store (or for each item in the aisle, or shelving unit, where the robot is capturing imagery), to find a match, indicating the product identity (wherein other visual product recognition techniques can also be used, including color histograms, trained neural networks, a shelf label, RFID identification). In paragraph [0049]-CALHOON discloses referring to FIG. 3A, the left edge of the Campbell's Chicken Noodle Soup shelf label serves as the left edge of an overlaid dashed bounding box that extends eight inches to the right, and the full height of the shelf space. Further in paragraph [0069]-CALHOON discloses overlay data or colors can be projected onto shelved products) using the location tag (Fig. 3A. Paragraph [0096]-CALHOON discloses the identifiers encoded by the watermark blocks can be associated with metadata using a data structure, such as a table or database—stored in the robot memory or remotely (e.g., in a store server memory, or in the cloud), or both. In paragraph [0098]-CALHOON discloses traditional planogram data is all in the table. With such a data structure, and the captured imagery, the robot can sense which products are out of place, or out of stock, and generate a punch list for attention by a store clerk to correct same. In paragraph [0120]-CALHOON discloses location markings can be provided on store shelves. Exemplary markings are watermark or barcode patterns that convey planogram coordinates, e.g., a shelving unit number, a shelf number, and/or a distance from the left edge of the shelving unit. In paragraph [0121]-CALHOON discloses markings can take the form of adhesive labels. In paragraph [0124]-CALHOON discloses in addition to identifying planogram location, such markings can also aid in understanding pose (including scale) of products resting on the shelf).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of BRONICKI of having a method for multi-object cluster detection, detecting, by the multi-object cluster detection model, a cluster of items having similar size, shape, and color, wherein each instance of an item in the cluster of items is located within a predetermined proximity to at least one other item having the similar size, shape, and color; identifying a location tag on at least a portion of the item storage structure associated with the cluster of items; and storing the location of the detected cluster of items in a database., with the teachings of CALHOON of having determining the location of the cluster of items using the location tag.
Wherein having BRONICKI’s method having determining the location of the cluster of items using the location tag.
The motivation behind the modification would have been to obtain a method that improves the detection of planogram compliance in retail images, since both BRONICKI and CALHOON concern retail inventory management. Wherein BRONICKI’s systems and methods provide improved efficiency, convenience, and functionality over prior art for planogram compliance and inventory management systems, while CALHOON systems and methods provide improvements for detection of multiple instances of objects and planogram compliance. Please see BRONICKI (US 20220261831 A1), Abstract and Paragraph [0370 and 0401] and CALHOON et al. (US 20200234394 A1), Abstract and Paragraph [0003-0006, 0067 and 0098].
Regarding claim 19, BRONICKI explicitly teaches the one or more computer storage devices of claim 15, BRONICKI further teaches wherein the operations further comprise:
detecting (Fig. 22. Paragraph [0110]-BRONSKI discloses embodiments include analyzing images to detect and identify different products. As used herein, the term “detecting a product” may broadly refer to determining an existence of the product. For example, the system may determine the existence of a plurality of distinct products displayed on a store shelf. By detecting the plurality of products, the system may acquire different details relative to the plurality of products (e.g., how many products on a store shelf are associated with a same product type), but it does not necessarily gain knowledge of the type of product. In contrast, the term “identifying a product” may refer to determining a unique identifier associated with a specific type of product that allows inventory managers to uniquely refer to each product type in a product catalogue). Please also read paragraph [0358-0359]), by the multi-object cluster detection model (Fig. 1. Paragraph [0118]-BRONICKI discloses the detection may occur through analysis of features in the image using an algorithm and stored data. The identification may occur through analysis of product features in the image according to stored product models. The term “product model” refers to any type of algorithm or stored product data that a processor may access or execute to enable the identification of a particular product associated with the product model. The product model may include a description of visual and contextual properties of the particular product (e.g., the shape, the size, the colors, the texture, the brand name, the price, the logo, text appearing on the particular product, the shelf associated with the particular product, adjacent products in a planogram, the location within the retail store, etc.). A first product model may be used by server 135 to identify a product category (such models may apply to multiple product types, e.g., shampoo, soft drinks, etc.), and a second product model may be used by server 135 to identify the product type, product identity, or other characteristics associated with a product. Please also read paragraph [0111, 0319-0335 and 0362]), a cluster of items comprising at least one instance of an item (Fig. 22. Paragraph [0362]-BRONICKI discloses processing device 2202 may analyze image data 2150 of a retail shelf and determine that the products on the shelf belong to a first product type. Processing device 2202 may analyze on-shelf sensor data 2100 associated with the retail shelf to determine that the items on the shelf belong to a second product type. Processing device 2200 may analyze additional information (i.e., information other than on-shelf sensor data 2100 and image data 2150) to determine the product type associated with the products. The additional information may include, but is not limited to, historic information regarding the retail shelf (e.g., the product types previously displayed on the retail shelf), place of the retail shelf in a retail store (e.g., whether the retail shelf is located in an aisle for condiments or in a section of the store for home cleaning products), planogram information, product types of nearby products, shelf labels, height of the retail shelf, 3D images. Please also see Fig. 11D, 18G and 29A-B), wherein each instance of an item in the cluster of items is located within a predetermined proximity to at least one other item having a same size, shape, and color (Fig. 29A. Paragraph [0403]-BRONICKI discloses image 2700 may include at least part of a shelf 2702. Shelf 2702 may include milk products 2710, cream products 2720, butter products 2730, and egg products 2740. Shelf 2702 and products 2710, 2720, 2730, and 2740. Server 135 may be configured to analyze image to determine a placement of products on the shelf (or part of the shelf). Image processing unit 130 may use various image analysis techniques to detect products 2710, 2720, 2730, and 2740 on shelf 2702 (wherein each item in a product set is located within a close proximity to other products with similar attributes (e.g. shape, size and color)). Please also see Fig. 18G, and read paragraph [0319-0335]);
identifying a location tag on at least a portion of the item storage structure associated with the cluster of items (Fig. 1. Paragraph [0111]-BRONICKI discloses the characteristic of the product may assist in classifying and identifying the product. The characteristic of the product may be associated a logo or text associated with the product (e.g., on a product label), and more. In paragraph [0477]-BRONICKI discloses product label 3306 may be attached to a part of a shelf or to other objects (such as displays, containers, etc.) in the retail store that is associated with the product (e.g., stocks the product). A product label may include any combination of a logo, a barcode, a quick response (QR) code, a word, a character, a shape, a color, a placement location of the label relative to the product, or any other information detectable by an image sensor);
storing the determined location of the cluster of items (Fig. 2. Paragraph [0112]-BRONICKI discloses the system may retrieve different types of contextual information from captured image data and/or from other data sources. Other examples of types of contextual information may include a location of the product in the retail store. Please also read paragraph [0118, 0213 and 0233]) in a database (Fig. 2, #140 called a database. Paragraph [0130]-BRONICKI discloses memory device 226 may store database 140. Database 140 may include product type model data 240 (e.g., an image representation, a list of features, a model obtained by training machine learning algorithm using training examples, an artificial neural network, and more) that may be used to identify products in received images; contract-related data 242 (e.g., planograms, promotions data, etc.) that may be used to determine the placement of products on the store shelves. Please also read paragraph [0123, 0279-0280]).
BRONICKI fails to explicitly teach determining the location of the cluster of items using the location tag.
However, CALHOON explicitly teaches determining the location of the cluster of items (Fig. 3A. Paragraph [0011] FIG. 2A shows an image of a supermarket shelf. In paragraph [0082]-CALHOON discloses imagery captured by a robot can be analyzed to recognize a shelved product by its visual appearance. Image fingerprints can be derived from product artwork, and compared against reference fingerprint data for each item in the store (or for each item in the aisle, or shelving unit, where the robot is capturing imagery), to find a match, indicating the product identity (wherein other visual product recognition techniques can also be used, including color histograms, trained neural networks, a shelf label, RFID identification). In paragraph [0049]-CALHOON discloses referring to FIG. 3A, the left edge of the Campbell's Chicken Noodle Soup shelf label serves as the left edge of an overlaid dashed bounding box that extends eight inches to the right, and the full height of the shelf space. Further in paragraph [0069]-CALHOON discloses overlay data or colors can be projected onto shelved products) using the location tag (Fig. 3A. Paragraph [0096]-CALHOON discloses the identifiers encoded by the watermark blocks can be associated with metadata using a data structure, such as a table or database—stored in the robot memory or remotely (e.g., in a store server memory, or in the cloud), or both. In paragraph [0098]-CALHOON discloses traditional planogram data is all in the table. With such a data structure, and the captured imagery, the robot can sense which products are out of place, or out of stock, and generate a punch list for attention by a store clerk to correct same. In paragraph [0120]-CALHOON discloses location markings can be provided on store shelves. Exemplary markings are watermark or barcode patterns that convey planogram coordinates, e.g., a shelving unit number, a shelf number, and/or a distance from the left edge of the shelving unit. In paragraph [0121]-CALHOON discloses markings can take the form of adhesive labels. In paragraph [0124]-CALHOON discloses in addition to identifying planogram location, such markings can also aid in understanding pose (including scale) of products resting on the shelf).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of BRONICKI of having one or more computer storage devices, wherein the operations further comprise: detecting, by the multi-object cluster detection model, a cluster of items comprising at least one instance of an item, wherein each instance of an item in the cluster of items is located within a predetermined proximity to at least one other item having a same size, shape, and color; identifying a location tag on at least a portion of the item storage structure associated with the cluster of items; and storing the determined location of the cluster of items in a database. with the teachings of CALHOON of having determining the location of the cluster of items using the location tag.
Wherein having BRONICKI’s computer storage devices having determining the location of the cluster of items using the location tag.
The motivation behind the modification would have been to obtain computer storage devices that improves the detection of planogram compliance in retail images, since both BRONICKI and CALHOON concern retail inventory management. Wherein BRONICKI’s systems and methods provide improved efficiency, convenience, and functionality over prior art for planogram compliance and inventory management systems, while CALHOON systems and methods provide improvements for detection of multiple instances of objects and planogram compliance. Please see BRONICKI (US 20220261831 A1), Abstract and Paragraph [0370 and 0401] and CALHOON et al. (US 20200234394 A1), Abstract and Paragraph [0003-0006, 0067 and 0098].
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant`s disclosure.
PATIL et al. (US 20230274226 A1)- The disclosed system and method relate to automatically detecting empty spaces on retail store shelves, identifying the missing product(s) and causing the space to be replenished or restocked. For example, stores may use shelf-mounted imaging devices to capture images of shelves across the aisle from the imaging devices. The images captured by the imaging devices may be pre-processed to de-warp, de-skew images and stitch together multiple images in order to retrieve an image that captures a full width of a shelf. The pre-processed images can then be used to detect products on the shelf, identify the detected products. For example, the captured image may be compared against a reference background image using a background modeling algorithm to identify empty spaces and mis-shelved items within the shelf.............................. Please see Fig. 5-6 and [0082-0101]. Abstract.
ADATO et al. (US 20190213752 A1)- A system for processing images captured in a retail store is provided. The system may include at least one processor configured to receive an image depicting a store shelf having at least one bottle displayed thereon and analyze the image to detect a representation in the image of the at least one bottle. The at least one bottle may have an outline design. The at least one processor is also configured to identify in the image two outline elements being associated with the outline design of the at least one bottle. Each of the two outline elements may have a specific length. The at least one processor may further be configured to determine a size of the at least one bottle based on a comparison of the lengths of the two outline elements............................ Please see Fig. 30A-C and 44. Abstract.
SKAFF et al. (US 20220303445 A1)- An image capture system for tracking target objects includes a movable base that supports one or more image capture modules, each of the image capture modules including a camera module, a first light module and a second light module. The camera module is mounted on a first plane of a mount bracket, the first light module is mounted on a second plane of the mounting bracket, and a second light module is mounted on a third plane of the mounting bracket. The first and second planes define a first angle, the first and third planes define a second angle, and the second angle is larger than the first angle. An output of the first light module and a light output of the second light module is selected based on an operating condition of the camera module............................... Please see Fig. 4. Para. [0072-0083]. Abstract.
VERMA et al. (US 20200394598 A1)- A computer-implemented method includes: determining, by a computer device, initial locations of products in a repository; determining, by the computer device, an initial arrangement of the products based on the initial locations; determining, by the computer device, current locations of the products in the repository; determining, by the computer device and based on the initial locations and the current locations, that one of the products is moved away from its home location; determining, by the computer device and based on the determining the one of the products is moved away from its home location, that the one of the products is misplaced based on predefined conditions being satisfied; and generating, by the computer device, an alert based on the determining the one of the products is misplaced............................. Please see para. [0058-0060]. Abstract.
KUMAR et al. (US 20160371634 A1)- The system and method for recognizing and counting products within images is disclosed. The products are recognized by carrying out a nearest-neighbor search in the template feature space using a k-d tree and the product count is then obtained by using a maximum feature repeatability index for each identified feature. The system also obtains product arrangement by fitting bounding boxes around each identified product. The count of boxes thus obtained may also provide the exact number of discrete products visible in an image. A second stage of grid-based search is also carried out in the neighborhood of each detected product to detect new products that might have been missed out in the previous step. This detection is based on a confidence measure including information like histogram matching and spatial location of products. This system is also useful in verifying planogram compliance for a given product................................ Please see Fig. 13 and para. [0036 and 0042]. Abstract.
Karmakar et al. (US 20220415012 A1)- This application relates to automated processes for determining item placement compliance within retail locations. For example, a computing device may obtain an image of a fixture within a store. The image may be captured by a camera with a field of view directed at the fixture. The computing device may apply a segmentation process to the image to determine a portion of the image. Further, the computing device may determine a correlation between the portion of the image and each of a plurality of item image templates. Each item image template may include an image of an item the retail location sells in the retail location. The computing device may determine, based on the correlations, one of the plurality of item image templates and its corresponding item. The computing device may then determine whether the item should be located at the fixture based on a planogram................................ Please see Fig. 5-9. Abstract.
Schumacher et al. (US 20210216952 A1)- A smart store management tool and methods are provided for enabling a user to design and stock an entire store. The smart store management tool may include a store builder that enables the user to use familiar techniques to create a layout of the entire retail store, as well as organize locations of gondolas and inventory within the store. A whole-store view comprising the smart store management tool may be configured to show locations and current stock of inventory within the store and enables the user to view the current status of inventory at specific locations within the store. An inventory analysis view comprising the smart store management tool may be configured to show the locations and current stock of inventory on a specific gondola or shelf that is selected by the user in the whole-store view............................... Please see Paragraph 1 and 4. Abstract.
SAVVIDES et al. (US 20250045692 A1)- Disclosed herein is a system and method of identifying misplaced products on a retail shelf using a feature extractor trained to extract features from images of products on the shelf and output identifying information regarding the product in the product image. The extracted features are compared to extracted features in a product library and a best fit is obtained. A misplaced product is identified if the identifying information produced by the feature extractor fails to match the identifying information associated with the best fit features from the product library............................... Please see Fig. 2-5 Abstract.
Ramamurthy et al. (US 20240135317 A1)- A system for image-based inventory determination including a cart. The cart includes a camera and a cart computing system in communication with the camera. The cart computing system includes a cart processor and a cart memory storing instructions that, when executed by the cart processor, cause the cart computing system to capture and store a set of images from the camera; process the set of images to identify an image of the set of images that displays a section of shelving having both a left vertical beam and a right vertical beam; and transmit the identified image to a remote server................................. Please see para. [0012 and 0035-0042]. Abstract.
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to Aaron Bonansinga whose telephone number is (703) 756-5380 The examiner can normally be reached on Monday-Friday, 9:00 a.m. - 6:00 p.m. ET.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s
supervisor, Chineyere Wills-Burns can be reached by phone at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AARON TIMOTHY BONANSINGA/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights