DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s “Amendment” filed on 01/30/2026 has been considered.
Claims 16-27 are amended. Claims 16-27 remain pending in this application and an action on the merits follow.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/13/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 16 and 27 recites the limitation " for each detected electronic shelf label whose identified unique area is determined to correspond to a detected empty area ". There is insufficient antecedent basis for this limitation in the claim.
Claims 20 and 21 recites the limitation " a detected empty area ". There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 16-27 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No. 2017/0337508 to Bogolea et al., in view of U.S. Patent Application Publication No. 2016/0171707 to Schwartz.
With regard to claims 16 and 27, Bogolea discloses a computer-implemented method for determining out-of-stock articles in a gondola of a sales area including a plurality of electronic shelf labels, wherein each electronic shelf label corresponds to a unique area of the gondola associated with a unique product slot field, the method comprising:
obtaining an image of the gondola acquired by an imaging device (paragraph 13, a robotic system to capture images of products arranged on shelves throughout a retail space (e.g., a grocery store));
detecting empty areas in the obtained image (paragraphs 81 and 107, the system determines that a particular slot is empty of product. In yet another variation shown in FIG. 4, the system can detect an “empty” slot devoid of product based on features extracted from an image recorded by the robotic system.);
detecting electronic shelf labels in the obtained image (paragraph 74, the system: implements computer vision techniques to detect a product label on a shelf within the image);
for each detected electronic shelf label: identifying, in the obtained image, an area of the gondola unique to the detected electronic shelf label (paragraphs 56 and 74, the system: implements computer vision techniques to detect a product label on a shelf within the image. The system can also: read a barcode, extract a SKU or other product identifier, and/or read a price from a first shelf tag detected in the first image. If data read from the first shelf tag align to data specified in the planogram, the system can define a first slot position relative to the first shelf tag in the image); and
determining whether the identified area unique to the detected electronic shelf label corresponds to a detected empty area (paragraphs 56, 74, 88, and 127, In this example, the system can also: read a barcode, extract a SKU or other product identifier, and/or read a price from a first shelf tag detected in the first image; confirm that these data read from the first shelf tag align to data assigned to the adjacent slot (e.g., above and to the right of the first shelf tag) by the planogram. If data read from the first shelf tag align to data specified in the planogram, the system can define a first slot position relative to the first shelf tag in the image. The system can also flag a slot container in the stock map as empty or containing an incorrect product. Each 2D overlay defines a vectorized line map delineating discrete slots in the shelving structure, containing a numerical indicator of a proportion of total product facings (e.g., ½, 50%) assigned to each slot in the shelving structure, and color-coded flags for empty);
for each detected electronic shelf label whose identified unique area is determined to correspond to a detected empty area: determining the product slot field unique to the detected electronic shelf label (paragraphs 56, 74, 88, and 127); and
retrieving, from a realogram database, an associated with the determined product slot field; and deducing that an article associated with the retrieved article identifier is out-of-stock (paragraph 74, 76, 81, and 88, the system can then determine the status of a product arranged on the shelf in the corresponding slot in Block S152 based directly on product facing count and product identifier data appearing on a product label applied to a shelf. Alternatively, the system can retrieve stocking requirements for the slot by passing the product identifier read from the product label into a product location database and compare these stocking requirements to data tagged to a template image matched to the slot region in the image to determine the stocking status of the slot, as described below. The system can therefore aggregate a set of template images—from the template image database—for comparison with the image based on: SKUs of products designated for stocking within this aisle or on this shelving structure. The system can aggregate a set of template images—for subsequent comparison to the image to identify the presence or status of products stocked on the shelving structure. The system can also flag a slot container in the stock map as empty);
however, Bogolea does not disclose detecting empty areas in the obtained image by color recognition with respect to a predetermined pattern on a top surface, on a back, or on a row of shelves of the gondola/by detecting zones of lower luminosity.
However, Schwartz teaches detecting empty areas in the obtained image by color recognition with respect to a predetermined pattern on a top surface, on a back, or on a row of shelves of the gondola/by detecting zones of lower luminosity (For example, a superpixel is a compact part of a digital image, which is larger than a normal pixel, where each part includes pixels of approximately the same color and brightness. At 608, the empty space module 207 determines the presence and location of empty space under shelves. In some embodiments, the empty space module 207 identifies out of stock areas by determining areas lacking objects (for example, dark, homogenously colored areas and/or areas corresponding to the back of shelves such as pegboard). FIG. 20 is a flow diagram of an example method 608 for finding empty space under shelves in a realogram image by using a segmentation algorithm, such as GrabCut, to segment out the shelf background from the shelf fronts and products. At 2002 the empty space module 207 identifies empty areas under a shelf, for example by choosing the darkest superpixels that are directly under a shelf (and are not in the bounding box of a recognized product). the empty space module 207 seeds the dark pixels directly beneath a shelf with “out of stock” superpixels. The techniques include an image recognition system to receive a realogram image including a plurality of organized objects and to identify areas where products are “out of stock”., abstract, paragraphs 62, 74 and 163).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Bogolea to include, detecting empty areas in the obtained image by color recognition with respect to a predetermined pattern on a top surface, on a back, or on a row of shelves of the gondola/by detecting zones of lower luminosity, as taught in Schwartz, in order to analyze images of a plurality of organized objects to identify objects (Schwartz, paragraph 2).
With regard to claim 17, Bogolea discloses further comprising outputting a signal alert based on detected electronic shelf label whose identified unique area corresponds to a detected empty area not being associated with an article identifier in the realogram database (paragraphs 15 and 112, generate a global restocking list containing a set of prompts or tasks for correcting missing, misplaced, or misoriented products throughout the store).
With regard to claim 18, Bogolea discloses further comprising detecting unsatisfactory front views in the identified area unique to the detected electronic shelf label based on a detected electronic shelf label whose identified unique area is determined not to correspond to a detected empty area (paragraphs 13, 27, 74 and 76, identify deviations between actual product arrangement throughout the store and target product presentation requirements defined in a planogram of the store. identify the presence or status of products stocked on the shelving structure. The computer system can process images collected during the scan cycle to generate a graph, map, or table of current product placement on shelves in the store and/or to generate a task list of misplaced or misoriented products to correct. Examiner notes that misoriented products image/view captured in the field of view of the camera can be considered as “detecting unsatisfactory front views in the identified unique area”).
With regard to claim 19, Bogolea discloses each electronic shelf label is associated in a labels database with a single article identifier, and, based on a given electronic shelf label corresponding to a unique area associated with a unique product slot field being associated in the labels database with a specific article identifier, the corresponding product slot field is associated in the realogram database with the specific article identifier, wherein the realogram database is updated in real-time to comply with the labels database (Fig. 2, paragraph 15 and 133, a SKU assigned to a particular slot. The system can accommodate this manual change in real-time by detecting a unit of the second product in the slot assigned the first product, checking that the first product is not in stock at the store, and temporarily updating the planogram to assign the second product to the slot. Examiner notes that a specific article identifier (a second product) is associated with the slot assigned, updating the planogram to assign the second product to the slot, which is considered as “based on a given electronic shelf label corresponding to a unique area associated with a unique product slot field being associated in the labels database with a specific article identifier, the corresponding product slot field is associated in the realogram database with the specific article identifier, wherein the realogram database is updated in real-time to comply with the labels database”).
With regard to claim 20, Bogolea discloses detecting unsatisfactory front views in the identified unique area comprises:
determining the product slot field unique to the detected electronic shelf label whose identified unique area does not correspond to a detected empty area (paragraphs 56, 74, 88, and 127);
retrieving, from the realogram database, the article identifier stored in association with the determined product slot field unique to the detected electronic shelf label whose identified unique area does not correspond to a detected empty area (paragraph 74, the system can then determine the status of a product arranged on the shelf in the corresponding slot in Block S152 based directly on product facing count and product identifier data appearing on a product label applied to a shelf.);
retrieving, from an article image database, an expected front view of the article identified by the retrieved article identifier (paragraph 76, In Block S140, the system can assemble this set of template images that includes images of various sides, lighting conditions, orientations, graphics release, etc. from product packaging of each product identified in Block S130. Examiner notes that images of various sides of each product identified can be considered as “retrieving… an expected front view of the article identified by the retrieved article identifier”);
calculating a similarity rate between the real view of the article in the obtained image and the expected front view of the article identified by the retrieved article identifier (paragraph 84, the system can calculate a score (e.g., a “confidence score,” a “similarity score”) for a match between a subregion of the image and a template image in the set of template images representing products assigned to the first shelf. ); and
outputting at least one of a list of determined product slot fields or a list of article identifiers for which it is determined, from the value of the similarity rate, that the front view is unsatisfactory (paragraphs 83-85, the system can: implement optical character recognition, object detection, or other computer vision techniques to extract text and/or object geometry from the first subregion of the image; and then rank, prioritize, or filter the set of template images for comparison to the first subregion according to similarities between text shown on and/or the geometry of products represented in the template images and in the first subregion of the image. identify an object represented in the first subregion as a first product in response to the similarity score between the first subregion and a first template image—representing the first product—exceeding all other similarity scores for template images in the set and exceeding a preset threshold similarity score. Examiner note that even though the orientation of products does not match to the stock requirements (i.e., the front view is unsatisfactory), the system can: implement optical character recognition, object detection, or other computer vision techniques to extract text and/or object geometry from the first subregion of the image to compare images until a match is found, which is considered as “outputting at least one of … article identifiers for which it is determined…that the front view is unsatisfactory”.).
With regard to claim 21, Bogolea discloses further comprising: retrieving from the realogram database expected facing information for an identified area unique to a detected electronic shelf label that does not correspond to a detected empty area (Fig. 1, paragraph 40, template target orientation: right side or front. the target location and target orientation defined in a nearest waypoint); and checking, by image recognition, compliance between the expected facing information and a real facing information in the identified area unique to the detected electronic shelf label that does not correspond to a detected empty area in the obtained image (paragraphs 40 and 102, the system can: compare the actual position and orientation of the robotic system at the time the raw image was captured (e.g., as stored in image metadata) to the target location and target orientation defined in a nearest waypoint. The system can implement any other method or technique to count product facings on a shelf shown in an image and to handle deviations between actual and target product facings specified in the planogram).
With regard to claim 22, Bogolea discloses the expected facing information is a number of consecutive lines of a same article displayed in the gondola (Fig, 1, paragraphs 74 and 81, facing count a set of template images tagged with facing count. the system can execute Blocks S150 and S152 to determine whether: the correct number of facings of a product are showing).
With regard to claim 23, Bogolea discloses further comprising displaying to a user, via a graphical interface, at least one of a representation of the gondola, along with visual signals highlighting empty areas of the gondola or a visual alert that at least one of article identifiers or product slot fields associated with detected empty areas of the gondola must be re-stocked (paragraphs 109 and 111, the system can generate a restocking prompt for filling an empty slot with a number of units of a particular product—assigned to the slot by the planogram—in response to identifying the slot as empty. The system can then compile restocking prompts for various slots—across multiple shelves, shelving segments, and/or shelving structures throughout the store—exhibiting deviation from the planogram into a global electronic restocking list in Block S160.).
With regard to claim 24, Bogolea discloses detecting electronic shelf labels in the obtained image further comprises performing pattern recognition with respect to a predetermined set of possible electronic shelf label shapes (paragraph 87, pattern matching, shape recognition).
With regard to claim 25, Bogolea substantially discloses the claimed invention, however, Bogolea does not disclose further comprising detecting rows of electronic shelf labels, wherein detecting a row of electronic shelf labels comprises detecting an alignment of detected electronic shelf labels and wherein identifying the area unique to a detected electronic shelf label comprises identifying an area of the acquired image situated between the detected electronic shelf label and a consecutive detected electronic shelf label along a detected row of the electronic shelf labels.
However, Schwartz teaches detecting rows of electronic shelf labels, wherein detecting a row of electronic shelf labels comprises detecting an alignment of detected electronic shelf labels and wherein identifying the area unique to a detected electronic shelf label comprises identifying an area of the acquired image situated between the detected electronic shelf label and a consecutive detected electronic shelf label along a detected row of the electronic shelf labels (The shelf/label detection module 205 determines whether a superpixels is a superpixel shelf feature based on finding groups of four nearby superpixels at horizontal edges. when labels are close together, multiple labels may be detected as a combined label region instead of as individual, separate labels. If there are multiple prices that are horizontally separable in a single label region, the shelf/label detection module 205 splits that label region. In one embodiment, the shelf/label detection module splits the combined label region into separate labels based on the median width of labels determined from label detection. In another embodiment, the shelf/label detection module 205 splits the combined label region based on the location and/or size of price bounding boxes, paragraphs 76 and 125).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Bogolea to include, detecting rows of electronic shelf labels, wherein detecting a row of electronic shelf labels comprises detecting an alignment of detected electronic shelf labels and wherein identifying the area unique to a detected electronic shelf label comprises identifying an area of the acquired image situated between the detected electronic shelf label and a consecutive detected electronic shelf label along a detected row of the electronic shelf labels, as taught in Schwartz, in order to analyze images of a plurality of organized objects to identify objects (Schwartz, paragraph 2).
With regard to claim 26, Bogolea discloses a non-transitory computer-readable medium storing code instructions which, when executed by a processor, cause the processor to implement the method according to claim 16 (fig. 1, paragraph 19).
Response to Arguments
Applicants' arguments filed on 01/30/2026 have been fully considered but they are not fully persuasive especially in light of the previously references applied in the rejections.
Applicants remark that “based on the proposed amendment, applicant respectfully submits that the cited references fail to disclose all of the features of claims 16 and 27”.
Examiner directs Applicants' attention to the office action above.
Conclusion
Please refer to form 892 for cited references.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication from the examiner should be directed to Ariel Yu whose telephone number is 571-270-3312. The examiner can normally be reached on Monday-Friday 9:00am-5:00pm EST.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Obeid Fahd A can be reached on 571-270-3324. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ARIEL J YU/Primary Examiner, Art Unit 3627