DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant has filed a first After Final on 19 February 2026 and followed by, on 13 March 2026 RCE, a Second After Final both of which have been entered.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1 and 12 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. See the new application of Takenaka (US 2018/0122062 A1) responsive to the recent amendments and arguments.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-15 are rejected under 35 U.S.C. 103 as being unpatentable over Mok (US 20210027452 A1) and Takenaka (US 2018/0122062 A1).
Claim 1
In regards to claim 1, Mok discloses a monitoring device for monitoring placement of an article, {Mok discloses a variety of computer-vision based product compliance verification methods including methods for monitoring articles/products on a store shelf for compliance with a planogram specifying placements (position and/or orientation) of various articles on the store shelves to detect misplacements of such articles as per Figs. 1, 3, 8 (copied below), [0001], [0015]-[0016}. As such, Mok monitors the same type of articles for correct product placement fully consistent with the instant specification as well as the more broadly recited claims}, the monitoring device comprising:
a processor; and a memory including a program that, when executed by the processor, causes the processor to perform operations {see [0092]-[0101]},
the operations including
acquiring an image in which the article is imaged when the article is placed {Fig. 3 including camera(s) 206 acquiring images of articles (products) 304 placed on a store shelf (product display 302, [0029], [0033]-[0036]};
setting a display section in the image, the display section being a section in which the article is to be placed {Mok employs a planogram that defines the particular locations for each of the products on the store shelf 302, [0075]-[0078]. As shown in Fig. 8, system 100 identifies/sets a region 806 (e.g. any of regions 806-1 through 806-3 shown in fig. 8 that is associated with and used to verify compliance with criteria 804 including article locations. In other words, the system set/designates regions 806 of the image in which each product is expected according to the planogram, [0077]-[0078]};
PNG
media_image1.png
604
854
media_image1.png
Greyscale
setting an intermediate section adjacent to the display section in the image, the intermediate section being used for determining suspect placement of the article {the product/article placement criteria include tolerances for each of the product placement regions 806 including “acceptable distances from ideal locations” and a margin for error for placement as per [0001], [0015], [0019], [0066], [0075]-[0078], [0088] such that an article whose placement is outside the tolerance range (intermediate section) is a “suspect placement of the article” as claimed. The plain meaning of the phrases “suspect placement” and “suspected of being misplaced” is a broad term that includes an article that is actually misplaced-- just as a person who actually stole a car is also suspected of stealing the car. In other words, suspected misplacement is a general category that also includes misplacement. As such, Mok discloses a two level article placement determination: correctly placed and misplaced (which includes suspected misplacement as noted above). Even further, Mok discloses these two categories/levels of positional error by employing both a “display section” and “intermediate section”};
detecting an article area in the image, the article area being an image area of the article {See [0078] including defining/setting a region of interest where a particular product is identified or recognized, [0080]-[0081] including initial matching to determine where each product is located in the image frame. See also Fig. 9 including step 906, [0087]-[0090]}; and
determining whether the article is misplaced and whether the article is suspected as being misplaced in a correlation with the display section based on a positional correlation between the article area and the intermediate section in a case where at least a part of the article area does not overlap the display section
{see above cites including [0001], [0015], [0019], [0066], [0075]-[0078], [0088] and particularly the tolerances and margins of error for detecting correct placement and suspected misplacement wherein when the detected article area is evaluated relative to the “display section”/expected area for product detection 806. Mok’s system 100 determines that the article is misplaced when the detected article area is partially outside and not fully within the expected region 806; in other words the detected article area may partially extend into the tolerance/margin of error region such that, depending on the product width and size of the margin of error, can result in the first degree of overlap (with the expected area) being 40% while the second degree of overlap (with the margin of error) being 60% thus indicating “suspected” misplacement within the broadly worded claim.
See also [0083] indicating differentiated compliance issues indicating categories of product misalignment with the planogram and [0085]-[0090] thus further disclosing concepts of “suspected misplacement”. For example, Mok discloses therein detecting whether products have been apparently switched on the shelf which is a differentiated compliance issue based on incorrect product placement such that product switch detection may also qualify as article misplacement while products detected within the margin of error are “suspected” misplacements. As such, Mok clearly discloses the concept of differentiated product placement designations that meet the broadly worded claim language}
determining that the article is correctly placed in a first case where a first positional correlation between (i) the article area and (ii) the display section combined with the intermediate section satisfies a first predetermined condition, and that the article is misplaced in a second case where the first positional correlation does not satisfy the first predetermined condition;
{see above cites wherein the placement criteria correspond to the first predetermined condition which determine correct placement and misplacement}
determining whether the article is misplaced and whether the article is suspected as being misplaced in a correlation with the display section based on a positional correlation between the article area and the intermediate section in a case where at least a part of the article area does not overlap the display section
{see above cites and explanations including that determining misplacement also determines suspected misplacement particularly in view of the broadly worded claim lanaguage}.
Takenaka is an analogous reference from the same field of planogram compliance determination including setting a state monitoring area on the captured image of a product, detecting a product with a product detector 33 and evaluating a display state of the products using various indices relating to display disorder including product misplacements.
Takenaka also teaches
in the first case of determining that the article is correctly placed, determining that placement of the article is suspect in a third case where a second positional correlation between (i) the article area, (ii) the display section, and (iii) the intermediate section satisfies a second predetermined condition; and
{See abstract, Figs. 1-3, 5, 6 including display state evaluator 34, product present detector 45, alignment degree calculator 47 and occupancy calculator 49.
Product detection unit 33 detects the position of the product in the state monitoring area based on an image of the state monitoring area, [0078]-[0086] while display state evaluation unit 34 evaluations the display state of the product in the state monitoring area including alignment degree calculator 47 to determine three differentiated states of alignment (appropriate state, mild disorder state, and severe disorder state. See also Figs. 6, 10, [0109]-[0117], [0066], [0139] discussing the alignment degree determination process in further detail. In sum, Takenaka determines an alignment degree which is an evaluation index indicating various degrees or levels to which products on store shelf are positionally aligned including (aligned, “first case”), mild disorder state (suspected of being misplaced, “second case”), and severe disorder state (misaligned), “third case”.}
in the second case and the third case, outputting a notification that distinguishes between whether the article is misplaced and whether the placement of the article is suspect.
{see above mappings and explanations concluding that Takenaka determines an alignment degree which is an evaluation index indicating various degrees or levels to which products on store shelf are positionally aligned including (aligned, “first case”), mild disorder state (suspected of being misplaced, “second case”), and severe disorder state (misaligned), “third case”. Further as to outputting notification, see Figs. 1, 6 including monitor/display 7 and printer 8 and display information generator 37. See also
Figs. 14, 15 output notification display, [0080]-[0093], [0139] teaching outputting notifications that distinguish between three different states/cases/stages of positional alignment referred to as appropriate display state (aligned), mild disorder state (suspected of being misplaced), and severe disorder state (misaligned)}
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Mok which already discloses a two-level determination of correct article placement (aligned and misaligned wherein “misaligned” includes suspect placements) and the use of threshold and tolerances for such determinations, and wherein Mok also already discloses determining that the article is correctly placed in a first case where a first positional correlation between (i) the article area and (ii) the display section combined with the intermediate section satisfies a first predetermined condition, and that the article is misplaced in a second case where the first positional correlation does not satisfy the first predetermined condition such that the determination is a three-level determination of correct article placement, referred to as appropriate display state (aligned), mild disorder state (suspected of being misplaced), and severe disorder state (misaligned) as taught by Takenaka and such that the combination thus includes in the first case of determining that the article is correctly placed, determining that placement of the article is suspect in a third case where a second positional correlation between (i) the article area, (ii) the display section, and (iii) the intermediate section satisfies a second predetermined condition and, in the second case and the third case, outputting a notification that distinguishes between whether the article is misplaced and whether the placement of the article is suspect as also taught by Takenaka because doing so provides a more granular determination of product placement compliance with the planogram thus enabling more efficient and granular evaluation of such compliance as well as more detailed and granular notifications to better enable staff to remedy the placement issues, because there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 2
In regards to claim 2, Mok discloses wherein the processor determines that the article is correctly placed in the first case when the article area entirely overlaps a combined section obtained by combining the display section and the intermediate section {see above cites including [0001], [0015], [0019], [0066], [0075]-[0078], [0088] and particularly the tolerances and margins of error for detecting correct placement wherein when the detected article area is misplaced relative to the “display section” expected area for product detection 806. Mok’s system 100 determines that the article is not misplaced (correctly placed) when the detected article area overlaps or is otherwise within the combined section obtained by combining the region 806 with the intermediate section (tolerance/margin of error region). In other words, the detected object area is within the expected region 806 and within the tolerance/margin of error surrounding the region 806}.
Claim 3
In regards to claim 3, Mok discloses wherein the processor determines that the article is correctly placed in the first case when at least a part of the article area overlaps the combined section obtained by combining the display section and the intermediate section
{see above cites including [0001], [0015], [0019], [0066], [0075]-[0078], [0088] and particularly the tolerances and margins of error for detecting correct placement wherein when the detected article area is misplaced relative to the “display section” expected area for product detection 806. Mok’s system 100 determines that the article is not misplaced (correctly placed) when at least part (e.g. the whole detected article area) overlaps the combined section obtained by combining the region 806 with the intermediate section (tolerance/margin of error region). In other words, the detected object area is fully within the expected region 806 and within the tolerance/margin of error surrounding the region 806}.
Claim 4
In regards to claim 4, Mok discloses wherein the processor determines whether the placement of the article is suspect in the third case based on a degree of overlap of the article area with respect to the display section and the intermediate section
{see above cites including [0001], [0015], [0019], [0066], [0075]-[0078], [0088] and particularly the tolerances and margins of error for detecting correct placement wherein when the detected article area is misplaced relative to the “display section” expected area for product detection 806. Mok’s system 100 determines that the article is misplaced (and suspected of misplacement) based on a degree of overlap of the detected article area with respect to the region 806 (display section) and the intermediate section (tolerance/margin of error region). In other words, the detected object is misplaced (and suspected of misplacement) when there is no overlap (degree of overlap = 0) of the detected object region with the expected region 806 or tolerance/margin of error surrounding the region 806}.
Claim 5
In regards to claim 5, Mok discloses wherein the processor determines that the placement of the article area is not suspect in the third case when a first degree of overlap is equal to or greater than a second degree of overlap, the first degree of overlap being a degree of overlap of the article area with respect to the display section, the second degree of overlap being a degree of overlap of the article area with respect to the intermediate section
{see above cites including [0001], [0015], [0019], [0066], [0075]-[0078], [0088] and particularly the tolerances and margins of error for detecting correct placement wherein when the detected article area is not misplaced (not suspect) relative to the “display section” expected area for product detection 806. Mok’s system 100 determines that the article is not misplaced (not suspect) when the detected article area is fully within the expected region 806 and does not extend into the tolerance/margin of error region in which case the first degree of overlap is 100% while the second degree of overlap is 0% thus indicating correct placement (not misplaced)}.
Claim 6
In regards to claim 6, Mok discloses wherein the processor determines that the placement of the article is suspect in the third case when a first degree of overlap is smaller than a second degree of overlap, the first degree of overlap being a degree of overlap of the article area with respect to the display section, the second degree of overlap being a degree of overlap of the article area with respect to the intermediate section
{see the mapping of claims 1 and 5 above which also addresses the limitations of claim 6}.
Claim 7
In regards to claim 7, Mok discloses wherein when at least two article areas are included in the intermediate section, the processor determines that the at least two article areas are misplaced {see above cites while noting that Mok’s process is clearly applied to detecting misplacement of plural articles as illustrated in Fig. 8 and operates in an iterative fashion for each expected area for product detection 806. Moreover, each display section 806 includes an associated tolerance/margin of error area. As such, when one or two articles articles/products and their corresponding detection areas are within the intermediate section (tolerance/margin of error area around each display section 806) and extend beyond this intermediate section then Mok’s system determined that the article areas are misplaced}.
Claim 8
In regards to claim 8, Mok discloses wherein when the processor detects that the article area of the article to be placed in the display section and a second article area of a second article to be placed in a second display section adjacent to the display section are swapped, the processor determines that the article is misplaced regardless of the first positional correlation between (i) the article area and (ii) the display section combined with the intermediate section {see [0039], [0083] which determine whether products have been switched/swapped, Mok’s system determine that the product is misplaced regardless of the positional correlation because it is based on product recognition (wrong article/product) in expected position including the intermediate/margin of error section}.
Claim 9
In regards to claim 9, Mok discloses wherein the processor sets a first width of the intermediate section overlapping a second display section adjacent to the display section to not exceed a width of the second display section
{see above cites including [0001], [0015], [0019], [0066], [0075]-[0078], [0088] and particularly the tolerances and margins of error (intermediate section) for detecting correct placement. Fig. 8 illustrates adjacent display sections 806-1, 806-2, and 806-3. Given the close spacing (mirroring close spacing of products on a store shelf), the width of the tolerance/margin of error of display section 806-2 would not exceed a width of the adjacent display section 806-3 because otherwise a larger width of the tolerance/margin of error would thus supplant or otherwise interfere with the successful detection of a product within the adjacent area 806-3}.
Claim 9
In regards to claim 9, Mok discloses wherein the processor sets a first width of the intermediate section overlapping a second display section adjacent to the display section to not exceed a width of the second display section
{see above cites including [0001], [0015], [0019], [0066], [0075]-[0078], [0088] and particularly the tolerances and margins of error (intermediate section) for detecting correct placement. Fig. 8 illustrates adjacent display sections 806-1, 806-2, and 806-3 each having a tolerance/margin of error region (intermediate section) for successful detection of product placement}. In this alternative rejection of claim 9 it is assumed, in arguendo, that the relative width feature of claim 9 is not disclosed by Mok.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified the relative size of Mok’s object detection areas (display sections) 806 and corresponding tolerances/margins of error regions (intermediate sections) around each area 806 such that Mok’s system sets a first width of the intermediate section overlapping a second display section adjacent to the display section to not exceed a width of the second display section because Mok motivates flexibly arranging the location and size of the areas 806 to match a Planogram wherein the expected product arrangement and corresponding sizes and placements of the areas 806 may also be determined by scanning a verified product display in [0079] wherein such verified product displays vary widely in retail stores and/or because the sizes and relative of the detection areas and corresponding tolerance areas are merely routine optimization. “[W]here the general conditions of a claim are disclosed in the prior art, it is not inventive to discover the optimum or workable ranges by routine experimentation.” In re Aller, 220 F.2d 454, 456, 105 USPQ 233, 235 (CCPA 1955). See also MPEP 2144.05(II)(A). Here, at least the general conditions are disclosed because Mok discloses variable sizes for the detection areas 806 and the corresponding tolerance areas to accommodate different product placement schemes and desired accuracy level represented by the size of the tolerance area relative to the size of the detection area, wherein different conventional product layouts (e.g. a pyramid of cans) may require more positional precision than others (e.g. shelf of rather formless potato chip bags whose size/shape may vary as opposed to identical cans).
Claim 10
In regards to claim 10, Mok discloses wherein the processor sets the first width of the intermediate section overlapping the second display section to be
{see above cites including [0001], [0015], [0019], [0066], [0075]-[0078], [0088] and particularly the tolerances and margins of error (intermediate section) for detecting correct placement. Fig. 8 illustrates adjacent display sections 806-1, 806-2, and 806-3 each having a tolerance/margin of error region (intermediate section) for successful detection of product placement}.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified the relative size of Mok’s object detection areas (display sections) 806 and corresponding tolerances/margins of error regions (intermediate section) around each area 806 such that Mok’s system sets the first width of the intermediate section overlapping the second display section to be equal to the second width of the second display section because Mok motivates flexibly arranging the location and size of the areas 806 to match a Planogram wherein the expected product arrangement and corresponding sizes and placements of the areas 806 may also be determined by scanning a verified product display in [0079] and/or because the sizes and relative of the detection areas and corresponding tolerance areas are merely routine optimization. “[W]here the general conditions of a claim are disclosed in the prior art, it is not inventive to discover the optimum or workable ranges by routine experimentation.” In re Aller, 220 F.2d 454, 456, 105 USPQ 233, 235 (CCPA 1955). See also MPEP 2144.05(II)(A). Here, at least the general conditions are disclosed because Mok discloses variable sizes for the detection areas 806 and the corresponding tolerance areas to accommodate different product placement schemes and desired accuracy represented by the size of the tolerance area relative to the size of the detection area, wherein different conventional product layouts (e.g. a pyramid of cans) may require more positional precision than others (e.g. shelf of potato chip bags whose size/shape may vary as opposed to identical cans).
Claim 11
In regards to claim 11, Mok discloses wherein the processor determines a second display section corresponding to the article area, and outputs a determination result of misplacement for the article in accordance with a determination result of misplacement for the display section corresponding to the article area {Fig. 9 including steps 908-910 that generate and output determination results (compliance verification data) to a monitoring system, [0088]-[0090]}.
Claims 12 and 13
The rejection of device claim 1 above applies mutatis mutandis to the corresponding limitations of method claim 12 and computer readable medium claim 13 while noting that the rejection above cites to both device and method disclosures. For the computer readable storage medium storing program limitations of claim 10 see [0092]-[0094]}.
Claim 14
In regards to claim 1, Mok is not relied upon to disclose but Takenaka teaches
wherein the operations further include: outputting a notification when the placement of the article is suspect in the third case {see above mapping for claim 1 including Figs. 1, 6 including monitor/display 7 and printer 8 and display information generator 37 as well as Figs. 6, 10, [0080]-[0093], [0109]-[0117], [0066], [0139] outputting notifications that distinguish between three different states/cases/stages of positional alignment referred to as appropriate display state (aligned), mild disorder state (suspected of being misplaced), and severe disorder state (misaligned)}.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Mok which already discloses a two-level determination of correct article placement (aligned and misaligned wherein “misaligned” includes suspect placements) and the use of threshold and tolerances for such determinations, and wherein Mok also already discloses determining that the article is correctly placed in a first case where a first positional correlation between (i) the article area and (ii) the display section combined with the intermediate section satisfies a first predetermined condition, and that the article is misplaced in a second case where the first positional correlation does not satisfy the first predetermined condition such that the determination is a three-level determination of correct article placement, referred to as appropriate display state (aligned), mild disorder state (suspected of being misplaced), and severe disorder state (misaligned) as taught by Takenaka and such that the combination thus includes in the first case of determining that the article is correctly placed, determining that placement of the article is suspect in a third case where a second positional correlation between (i) the article area, (ii) the display section, and (iii) the intermediate section satisfies a second predetermined condition and, in the second case and the third case, outputting a notification that distinguishes between whether the article is misplaced and whether the placement of the article is suspect, and outputting a notification when the placement of the article is suspect in the third case as also taught by Takenaka because doing so provides a more granular determination of product placement compliance with the planogram thus enabling more efficient and granular evaluation of such compliance as well as more detailed and granular notifications to better enable staff to remedy the placement issues, because there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 15
In regards to claim 15, Mok is not relied upon to disclose but Takenaka teaches
wherein the operations further include: outputting a different notification for each reason in which the placement of the article is determined as suspect in the third case {the display state evaluator 34 also determines display state placement notifications for other reasons including volume degree 46 and face-up degree 48, and occupancy 49 which are also output as notifications via display information generator 37 and monitor 7. See [0080]-[0093], [0103]-[0121], [0139], Figs. 14-15}.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Mok which already discloses a two-level determination of correct article placement (aligned and misaligned wherein “misaligned” includes suspect placements) and the use of threshold and tolerances for such determinations, and wherein Mok also already discloses determining that the article is correctly placed in a first case where a first positional correlation between (i) the article area and (ii) the display section combined with the intermediate section satisfies a first predetermined condition, and that the article is misplaced in a second case where the first positional correlation does not satisfy the first predetermined condition such that the determination is a three-level determination of correct article placement, referred to as appropriate display state (aligned), mild disorder state (suspected of being misplaced), and severe disorder state (misaligned) as taught by Takenaka and such that the combination thus includes in the first case of determining that the article is correctly placed, determining that placement of the article is suspect in a third case where a second positional correlation between (i) the article area, (ii) the display section, and (iii) the intermediate section satisfies a second predetermined condition and, in the second case and the third case, outputting a notification that distinguishes between whether the article is misplaced and whether the placement of the article is suspect, wherein the operations further include: outputting a different notification for each reason in which the placement of the article is determined as suspect in the third case as also taught by Takenaka because doing so provides a more granular determination of product placement compliance with the planogram thus enabling more efficient and granular evaluation of such compliance as well as more detailed and granular notifications and differentiated reasons for the third case to better enable staff to remedy the placement issues, because there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Bogolea (US 20200013007) discloses a system for detection mis-oriented products on a shelf that employs detection areas and tolerance ranges similar to those in Mok and which read on the pending claims. See Figs. 1, 4, [0058]-[0060], [0097]-[0100].
Zaremski (US 2017/0177969 A1) discloses a planogram compliance system that determines a distance score metric for each product based on distance between expected and actual locations and misplacement degrees of individual products as per [0089]-[0102].
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael R Cammarata whose telephone number is (571)272-0113. The examiner can normally be reached M-Th 7am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at 5712-272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL ROBERT CAMMARATA/Primary Examiner, Art Unit 2667
Prosecution Insights