Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefore, subject to the conditions and requirements of this title.
Claims 1-5 and 9-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception, in particular an Abstract Idea falling under at least the (c) mental processes grouping (concepts performed in the human mind including an observation, evaluation, judgement, opinion), not ‘integrated into a practical application’ at Prong Two of Step 2A and without ‘significantly more’ at Step 2B.
Regarding claims 1-5 and 9-20:
Step 1: The claims in question are directed to a method for “receiving” image data, “identifying” items in an image, “creating” a map based on items in the image, “determining” differences between the current map and past maps, and “determining” missing items between maps (Step 1: Yes).
Step 2A Prong One: This part of the eligibility analysis evaluates whether the claim recites a judicial exception. As explained in MPEP 2106.04, subsection II, a claim “recites” a judicial exception when the judicial exception is “set forth” or “described” in the claim. Representative claims 1, 16, and 19 recite at a high level of generality –
1) “receiving…image data, 2) identifying…specific items in the image, 3) creating…a current map based on the specific items in the image, 4) determining…differences between the current map and one or more historical maps, 5) determining…that one or more of the items from the one or more…maps are not present” which is at least a mental process under the Abstract Ideas grouping. as the recited ‘obtaining, identifying, and segmenting’ are not precluded from being performed mentally/visually.
Even if the recited steps are conducted using ‘a computer’, limitations involving intermediate computerized processing steps do not preclude the method itself from being performed mentally, and the recited ‘current map creation’, is not necessitated to be performed at a level of complexity (as recited) that would render the process impossible/impractical for mental evaluation. See MPEP 2106.04(a)(2) subsection C. A Claim that Requires a Computer May Still Recite a Mental Process. Dependent claims are similarly analyzed at Prong One (e.g. claims 2-5, 9-15, 17-18, and 20) as they further comprise one or more limitations that may similarly be drawn under the mental processes Abstract Idea grouping (Step 2A, Prong One: Yes).
Step 2A Prong Two: This part of the eligibility analysis evaluates whether the claim integrates the recited judicial exception into a practical application (derived from Alice/Mayo step two and not to be conflated with an assessment of utility– MPEP 2103) of the exception. This evaluation is performed by (1) identifying whether there are any ‘additional elements’ recited in the claim beyond the judicial exception, and (2) evaluating those additional elements individually and in combination to determine whether the claim integrates the exception into a practical application. See MPEP 2106.04(d). Examiner notes for consideration at Prong Two of 2A that MPEP 2106.05(a), (b), (c), and (e) generally concern limitations that are indicative of integration, whereas 2106.05(f), (g), and (h) generally concern limitations that are not indicative of integration. As an additional note, ‘additional elements’ are generally limitations excluded from interpretation under the Abstract Idea groupings and may comprise portions of limitations otherwise identified as falling under those Abstract Idea groupings of the 2019 PEG (e.g. any ‘determination’ that may be made mentally accompanied by the use of generic computer hardware considered under the ‘apply it’ considerations of 2106.05(f)). Any ‘providing’/outputting broadly, and ‘collection’ of data (i.e. image acquisition(s)), be they images visually observable/ evaluated by a user/operator, also fail(s) to integrate at least in view of MPEP 2106.05(g) (extra-solution data gathering/output). The same determination holds for dependent claims that serve to limit the collection of data/images (by means of what is collected based on recited conditions) and/or introduce limitations generally linking to a field of use. None of the instant claims appear to explicitly/clearly capture/recite any disclosed improvement in technology (see MPEP 2106.05(a), with note that ‘functioning of a computer’ concerns functions integral to the way a computer operates and not ‘functions’ that a generic computer can be programmed/adapted to perform (see also 2106.05(f))) and any ‘additional elements’, even when considered in combination, fail to integrate at Prong Two of Step 2A accordingly. Integration in view of subsection (a) of 2106.05 requires an identification of the manner in which the improvement is achieved, to be explicitly and specifically (not at a high level of generality i.e. the ‘use of’ complementary modalities/data broadly for any conceivable satellite image analysis not inconsistent with diverse example embodiments) recited in the claims, as ‘additional elements’ precluded from interpretation under any of the Abstract Idea groupings (since the improvement cannot be to the exception itself).
Regarding claims 2, 9- 12, and 14, the recited ‘by the computer’, adds only generic computer components to the proposed method and thus, does not integrate the recited exception into a practical application. Per Applicant’s specification at [0033], “A "server computer" may include a powerful computer or cluster of computers. For example, the server computer can be a large mainframe, a minicomputer cluster, or a group of servers functioning as a unit. In one example, the server computer may be a database server coupled to a Web server. The server computer may comprise one or more computational apparatuses and may use any of a variety of computing”. This instance is exemplary of how Applicant’s specification fails to establish how they specifically improve the present technology, as they provide examples of existing devices on which their method, directed to a judicial exception, can be performed. Furthermore, see MPEP 2106.04(a)(2) subsection C. A Claim that Requires a Computer May Still Recite a Mental Process. See MPEP 2106.05(a) below for further clarification.
Regarding claims 2 and 15, these claims fail to integrate the use of the recited ‘machine learning model (MLM)’ into a practical application. Although the use of MLMs are recited throughout the incident disclosure, the claims as recited fail to provide information about how the MLM operates to perform the recited method and/or system steps. As such, interpretation using the plain meaning of ‘identifying’ does not preclude a human user from conducting the same steps as the MLM, and therefore, does not sufficiently integrate the MLM into a practical application. See below for an excerpt from Example 47 of the 2024 PEG, as the provided interpretation of ‘ANN’ is similar to the incident use of ‘MLM’. Note that “selecting a particular data source or type of data to be manipulated” is an explicit example of an ‘insignificant extra-solution activity’ per MPEP 2106.05(g). Additionally, see below for MPEP 2106.05(a) regarding improvements to computer functioning, noting that the claims as recited can all be performed mentally without unreasonable time or effort.
2024 PEG, Example 47:
The limitations “(a) receiving, at a computer, continuous training data” and “(f) outputting the anomaly data from the trained ANN” are mere data gathering and output recited at a high level of generality, and thus are insignificant extra-solution activity
MPEP 2106.05(a):
It is important to note that in order for a method claim to improve computer functionality, the broadest reasonable interpretation of the claim must be limited to computer implementation. That is, a claim whose entire scope can be performed mentally, cannot be said to improve computer technology.
Examples that the courts have indicated may not be sufficient to show an improvement in computer-functionality:
iii. Mere automation of manual processes, such as using a generic computer to process an application for financing a purchase, Credit Acceptance Corp. v. Westlake Services, 859 F.3d 1044, 1055, 123 USPQ2d 1100, 1108-09 (Fed. Cir. 2017) or speeding up a loan-application process by enabling borrowers to avoid physically going to or calling each lender and filling out a loan application, LendingTree, LLC v. Zillow, Inc., 656 Fed. App'x 991, 996-97 (Fed. Cir. 2016) (non-precedential)
It is important to note, the judicial exception alone cannot provide the improvement. The improvement can be provided by one or more additional elements. See the discussion of Diamond v. Diehr, 450 U.S. 175, 187 and 191-92, 209 USPQ 1, 10 (1981))
To show that the involvement of a computer assists in improving the technology, the claims must recite the details regarding how a computer aids the method, the extent to which the computer aids the method, or the significance of a computer to the performance of the method. Merely adding generic computer components to perform the method is not sufficient. Thus, the claim must include more than mere instructions to perform the method on a generic component or machinery to qualify as an improvement to an existing technology
Even when viewed in combination, the ‘additional elements’ present (namely the use of generic computer/technology components, insignificant extra-solution activity, and necessary data gathering and outputting) do not integrate the recited judicial exception into a practical application, the claims are directed to the judicial exception (Step 2A Prong Two: No).
Step 2B: This part of the eligibility analysis evaluates whether the claim amounts to ‘significantly more’ than the recited exception, i.e., whether any ‘additional element’, or combination of additional elements, adds an inventive concept to the claim. The considerations of Step 2A Prong 2 and Step 2B overlap but differ in that 2B also requires considering whether the claims/limitations feature any “specific limitation(s) other than what is well-understood, routine, conventional activity in the field” (WURC) (MPEP 2106.05(d)). Such a limitation if specifically recited however, must still be excluded from interpretation under any of the Abstract Idea groupings (else it is not an ‘additional element’ and extra-solution activity need not then be implicated). Limitations precluded from serving as indications of an inventive concept/ ‘significantly more’ include those that are not specifically recited (instead recited at a high level of generality), those that are established as WURC, those that are not ‘additional elements’ by nature of their analysis at Prong One (i.e. directed to the exception), and/or those that otherwise fail to serve for integration at Prong Two of 2A in view of 2106.05(f), (g) and/or (h). The claim(s) in question recite little beyond those limitations recited at a high level of generality and falling under the exception. (Step 2B: No).
Examiner notes that claims 6-8 were intentionally omitted from the analysis above because they are directed to eligible subject matter in view of analysis at Step 2A Prong One, as the recited graph generation requiring nodes, edges, and vectors precludes the processing from being performed mentally and thus, is not directed to a judicial exception. Examiner notes that while generically, graphs/maps can be created mentally and do not necessitate the use of computer intervention, the complexity of the graph as disclosed in claims 6-8 limits the ability for the method to be performed mentally in a reasonable amount of time and without excessive effort. Claims 7 and 8 both depend on claim 6 and provide further limiting steps of graph generation that further preclude the recited method from being performed mentally.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 12-17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Saqlain (Hybrid Approach for Shelf Monitoring and Planogram Compliance (Hyb-SMPC) in Retails Using Deep Learning and Computer Vision) in view of Yair (WO2018002709A2)
Regarding claim 1, Saqlain teaches a method comprising: receiving, by a computer, image data of an image of a shelf unit with specific items (see Figures 7A or 5A-D below, which depicts image data of an image of a shelf with specific items, and note the computer-generated e.g. bounding boxes and/or planogram generation); identifying, by the computer, the specific items in the image (The first module is product detection module. It per-forms detection and localization of racks as well as SKUs in parallel manner” [page 5], and further that “After detection, the detector provides the processed images as indicated in Figures 5(a) and 5(b) which contain the list of detected SKUs, along with their counters and IDs to illustrate the number of instances of the SKUs present on the shelves [page 6]); creating, by the computer, a current map based on the specific items in the image (“the postprocessed retail images received from the first module are converted into planogram images by using Python libraries called OpenCV and PIL image” [page 7]. Examiner notes that per Applicant’s specification at [0023], “a map can be a planogram”); determining, by the computer, differences between the current map and one or more historical maps of historical items from past images (see Figure 7, provided below, which shows the generated planogram and the company-provided (or previous) planogram side by side. Note further that “As we have all the coordinates of racks and SKUs provided by detectors, we pick the cropped image part of each SKU from both planogram images (company-provided and module-generated) to verify its color. If the RGB values of both cropped images get matched, then we store TRUE as a string in an object. Similarly, if they cannot be matched, then we store FALSE” [page 7]. Examiner notes that a SKU is a ‘stock-keeping unit’, which can be understood as a single item, evidence by the bounding boxes in Figure 5 (images A and B) below); and determining, by the computer, that one or more of the items from the one or more historical maps are not present in the current map (“Compliance accuracy of retail image with respect to company-provided planogram layout is calculated by the following equation, where Pmatched is the number of matched SKUs and Ptotal indicates the total number of SKUs” [page 12], and see the relevant equation below. Examiner notes that this equation covers cases where items are missing from the company-provided planogram, or added, but that it covers the missing case as necessitated by the recited limitation). Saqlain does not teach item tags comprising machine readable codes adjacent to the specific items.
PNG
media_image1.png
693
1168
media_image1.png
Greyscale
PNG
media_image2.png
705
870
media_image2.png
Greyscale
Equation 6
PNG
media_image3.png
60
348
media_image3.png
Greyscale
However, Yair teaches the method receiving item tags comprising machine readable codes adjacent to the specific items (“Fig. 3 illustrates exemplary images depicting a plurality of products on a plurality of store shelves, and a plurality of labels coupled to the store shelves and associated with the plurality of products. A capturing device (e.g., capturing device 105) may acquire the images illustrated in Fig. 3. An image processing system (e.g., system 100) may process the images and provide an indication about the shelf label accuracy” [046] and see Fig. 3 below).
PNG
media_image4.png
514
761
media_image4.png
Greyscale
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the planogram generation and shelf monitoring method and system as disclosed by Saqlain to include the barcode identification software as taught by Yair, the proposed motivation being that additionally identifying changes in the designated location for a specific object according to their associated shelf label allows for further verification of correct shelf placement, as objects may be misplaced according to their corresponding tag or barcode regardless of historical object location.
Regarding claim 2, Saqlain in view of Yair teaches the method of claim 1. Saqlain further teaches wherein identifying the specific items in the image comprises: identifying, by the computer, the specific items in the image using an object detection machine learning model (“For this purpose, the study used three one-stage deep learning detectors, that is, YOLO V4 [47], YOLO V5 [48], and YOLOR [49], to detect and classify the products.” [page 5]. Although not explicitly called a ‘machine learning object detection model’ in this instance, Examiner notes that one of ordinary skill in the art understands that YOLO models are object detectors using machine learning).
Regarding claim 3, Saqlain in view of Yair teaches the method of claim 2. Saqlain further teaches wherein the current map is a planogram (see Figure 7(b) above under claim 1 rejections).
Regarding claim 4, Saqlain in view of Yair teaches the method of claim 1. Yair further teaches wherein the specific items are determined based on the machine readable codes adjacent to the specific items (“Server 115 may also process the received images to determine from labels associated with each of the identified products, a specific product identifier and a specific displayed price” [029]).
Regarding claim 5, Saqlain in view of Yair teaches the method of claim 1. Yair further teaches wherein a user device captured the image data and provided the image data to a server computer (see Fig. 2 below, specifically noting capturing device 105 with touch screen capabilities 212/218, and this module’s connection to server 115), wherein the server computer stored the image data in an image database (see Fig. 2 below, specifically noting server 115 and database 120).
PNG
media_image5.png
806
593
media_image5.png
Greyscale
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified the planogram generation and shelf monitoring method and system as disclosed by Saqlain to include the image capture, data distribution, and storage capabilities as taught by Yair, the proposed motivation being that a streamlined method of data collection, transfer to necessary parts of the system, and subsequent storage of relevant information automates the disclosed process, making the method more user friendly and leaving less room for use-related data handling errors.
Regarding claim 12, Saqlain in view of Yair teaches the method of claim 1. Saqlain further teaches generating, by the computer, an item information data entry that indicates that the one or more of the items from the one or more historical maps are not present in the current map (“by counting the number of total TRUEs and FALSEs, we will generate the report of planogram compliance” [page 7]. Examiner further provides the same passage from Saqlain used in the claim 1 rejection here for necessary context to reject the whole limitation as recited: “Compliance accuracy of retail image with respect to company-provided planogram layout is calculated by the following equation, where Pmatched is the number of matched SKUs and Ptotal indicates the total number of SKUs” [page 12], and see equation 6 above); and storing, by the computer, the item information data entry into an item information database (“If the RGB values of both cropped images get matched, then we store TRUE as a string in an object. Similarly, if they cannot be matched, then we store FALSE” [page 7]. Examiner notes that Applicant’s specification provides no further specificity regarding ‘item information data’, and so under broadest reasonable interpretation, the presence vs. absence of items between planograms fulfills this limitation. Examiner further notes that should it be necessary, Yair discloses other item information data which can be applied).
Regarding claim 13, Saqlain in view of Yair teaches the method of claim 12. Yair further teaches wherein a central server computer obtains the item information data entry from the item information database (“Server 115 can, based on the image in which the product-label mismatch, the price mismatch, or the product-promotion mismatch are identified, provide electronic notification of any of the one or more mismatches to output unit 125” [031]) and updates an item listing in an application based on the item information data entry (“The processing device can further detect a plurality of mobile devices in proximity to or within the retail establishment. The detection can include mobile devices of known customers of the retail establishment. The known customers may include customers having an application of the retail establishment on their mobile devices” [071], see Fig. 7, provided below, noting further that per Applicant’s specification at [0045], “Item listings in a delivery application or an inventory application can be updated to reflect the changes indicated in the item information data entry. Based on this disclosure, updates to an application that prompt the user to provide updated item listing information based on acquired data fall within the broadest reasonable interpretation of the limitation as recited).
PNG
media_image6.png
561
400
media_image6.png
Greyscale
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified the planogram generation and shelf monitoring method and system as disclosed by Saqlain to include the automated application information updating capabilities as taught by Yair, the proposed motivation being that automating the process of application updating based on received information eliminates the necessity for a user to be constantly monitoring the disclosed system and provides for the fastest updates possible by immediately sending the most up-to-date information post processing.
Regarding claim 14, Saqlain in view of Yair teaches the method of claim 1. Yair further teaches wherein the image data is received in an image database query response message (see Fig. 1 below, specifically focusing on output unit 125, and noting that “Consistent with the present disclosure, capturing device 105 and/or server 115 may communicate with output unit 125 to present information derived from processing image data acquired by capturing device 105…output unit 125 may be incorporated with capturing device 105 such that the information derived from processing image data is presented on a display of capturing device 105” [031]. Examiner notes that, as Applicant’s specification provides no explicit definition of a ‘query response message’, the information being displayed on a capturing device for use/view by operators sufficiently fulfills the broadest reasonable interpretation of the limitation), wherein the image database query response message includes the image data and additional data, wherein the additional data includes a shelf unit identifier (“The area of interest can be any portion of the retail establishment… the area of interest may comprise a specific aisle in the store or a portion of the store shelves in a specific aisle of the store. The area of interest may be identified based on information stored in the database. For example, products of a specific supplier are contracted to be displayed in a specific aisle of a store and the area of interest would be the store shelves within the specific aisle” [070]), a service provider location identifier, and a service provider identifier (“The location of the user may also indicate the type of the store (e.g., Target or Walmart). In some embodiments, each type of store may be associated with a different product identifier (e.g., bar code, QR code). In some embodiments, the contextual information may include section of the store where the image was captured” [060]. Examiner notes that using location information as disclosed in Yair results in store types and locations being detected simultaneously), wherein the method further comprises: generating, by the computer, an image database query request message that requests image data and additional data (“The processing device can be configured to provide to each of the detected plurality of mobile devices a request for an updated image of the area of interest. In some embodiments, the processing device may transmit requests based on specific location information” [072]).
PNG
media_image7.png
584
466
media_image7.png
Greyscale
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified the planogram generation and shelf monitoring method disclosed by Saqlain in view of Yair to include the context identification capabilities and notification system as taught by Yair, the proposed motivation being to provide the fastest and most accurate inventory updates possible, since updating shelf information can happen the most quickly and effectively when the context, such as store location, is taken into account.
Regarding claim 15, Saqlain in view of Yair teaches the method of claim 14. Saqlain further teaches wherein the specific items in the image are identified using a machine learning model that is trained to identify items in image data (“For this purpose, the study used three one-stage deep learning detectors, that is, YOLO V4 [47], YOLO V5 [48], and YOLOR [49], to detect and classify the products” [page 5] noting further that “The first module is product detection module. It performs detection and localization of racks as well as SKUs in parallel manner…Three one-stage detectors are trained by providing labeled data (YOLO V4, YOLO V5, and YOLOR). The SKUs of the dataset are categorized under twelve categories” [pages 6-7]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the image database management method as proposed by Saqlain in view of Yair to include the machine learning model and training method for image identification as proposed by Yair, the proposed motivation being to further automate the object detection and identification process by leveraging well-developed AI object detection models.
Claim 16 is the computer claim corresponding to the method of claim 1 and is rejected accordingly. See Fig. 2 of Yair above under claim 5 rejections regarding hardware required by the incident computer claim.
Regarding claim 17, Saqlain in view of Yair teaches the method of claim 16. Yair further teaches storing the current map in a map database (Examiner notes that the maps are disclosed by Saqlain, see claim 1 rejections) in association with a shelf unit identifier, a service provider location identifier, and/or a service provider identifier (Examiner notes that limitations concerning identifier data stored in the database are rejected under claim 14).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the image and additional data storage method as proposed by Saqlain in view of Yair to include the generated planograms as proposed by Saqlain, the proposed motivation being to provide a visual demonstration to a viewer of the database output module such that product inventory issues can be quickly and easily identified.
Claim 19 is the system claim corresponding to the method of claim 1 and is rejected accordingly. See Fig. 2 of Yair above under claim 5 rejections regarding hardware required by the incident system claim.
Claims 6-11, 18 and 20 are rejected over Saqlain in view of Yair and further in view of Tonioni (Product recognition in store shelves as a sub-graph isomorphism problem).
Regarding claim 6, Saqlain in view of Yair teaches the method of claim 1. Saqlain further teaches the current map and the one or more historical maps (see claim 1 rejections). Both Saqlain and Yair fall silent regarding wherein the current map includes a graph, wherein the one or more historical maps include graphs, and wherein creating the current map comprises: creating, by the computer, the graph comprising a plurality of nodes and edges, wherein each node indicates a specific item or an item tag in the image data, and wherein each edge indicates a vector that points between neighboring items or item tags in the image data.
However, Tonioni teaches wherein the current map includes a graph, wherein the one or more historical maps include graphs (see Figure 1, provided below, noting that step 3 shows both an observed and a reference planogram with overlaid topological information (nodes and edges)), and wherein creating the current map comprises: creating, by the computer, the graph comprising a plurality of nodes and edges (see the observed planogram at step 3 of Figure 1 below, noting the nodes on each identified shelf item, and the edges connecting each node), wherein each node indicates a specific item or an item tag in the image data (“To build the Observed Planogram we instantiate a node for each item detected in the previous step” [page 4]), and wherein each edge indicates a vector that points between neighboring items or item tags in the image data (“For each node, the search is performed along 8 cardinal directions (N, S, E, W, NW, NE, SW, SE) and, if another bounding box is found at a distance less than a dynamically determined threshold, an edge is created between the two nodes. In the given node the edge is labeled according to the search direction (e.g. N)” [page 4]. Examiner notes that vectors have both magnitude and direction, and as edges are being detected in predetermined directions using predetermined distance thresholds, the edges are ‘indicating a vector’).
PNG
media_image8.png
500
871
media_image8.png
Greyscale
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the map generation method as proposed by Saqlain in view of Yair to include the graph generation method as proposed by Tonioni, the proposed motivation being to verify that products are placed correctly on shelves relative to other objects on the same shelf, which would allow for quicker misplacement detection and further ensure that a shelf unit is laid out as intended.
Regarding claim 7, Saqlain in view of Yair and Tonioni teach the method of claim 6. Tonioni further teaches wherein determining differences between the current map and the one or more historical maps comprises: comparing, by the computer, nodes and edges of the current map to nodes and edges of the one or more historical maps (“After the second step the Observed Planogram should contain true detections only. Hence, those nodes that are missing compared to the Reference Planogram highlight items that appear to be missing” [page 4]).
Regarding claim 8, Saqlain in view of Yair and Tonioni teaches the method of claim 6. Tonioni further teaches wherein determining differences between the current map and the one or more historical maps comprises: identifying, by the computer, a first target node in the current map (see step 3 of Figure 1 above under claim 6 rejections, noting that the Observed Planogram contains several target nodes which have been identified. Examiner notes that Applicant’s specification does not clarify what qualifies as a ‘target’ node, so Examiner is interpreting ‘target’ nodes as ones which have been considered important enough to be labeled and assigned corresponding edges); identifying, by the computer, a second target node in a historical map of the one or more historical maps, wherein the second target node and the first target node are identified as having a same item identifier (“we find the sub-graph isomorphism between the Observed and Reference planograms, so as to identify local clusters of self-consistent detected products, e.g. sets of products placed in the same relative position in both the Observed and Reference planograms” [page 3], further noting that “We formulate our problem as follows: given two graphs I (Reference Planogram) and O (Observed Planogram), find an isomorphism between a subset of nodes in I and a subset of nodes in O such that the former subset has the maximum feasible cardinality given product placements constraints. Each node in I can be associated with a node in O only if they both refer to the same product instance and exhibit coherent neighbors” [page 4]); determining, by the computer, one or more first edges that connect the first target node to first neighboring nodes in the current map (see step 3 of Figure 1 above under claim 6 rejections, noting that the Observed Planogram contains several target nodes with edges that connect the nodes to each other); determining, by the computer, one or more second edges that connect the second target node to second neighboring nodes in the historical map (see step 3 of Figure 1 above under claim 6 rejections, noting that the Reference Planogram contains edges connecting the nodes/objects of interest. Examiner notes that although the nodes are in the background of the planogram instead of the foreground as in the Observed Planogram, it is still evident that there are edges connecting the nodes in the center of the objects of interest); comparing, by the computer, the one or more first edges to the one or more second edges to obtain a similarity score that indicates a similarity of a neighborhood for the first target node and the second target node (“we find the sub-graph isomorphism between the Ob served and Reference planograms, so as to identify local clusters of self-consistent detected products, e.g. sets of products placed in the same relative position in both the Observed and Reference planograms” [page 3]. Examiner notes that ‘sub-graph isomorphism’ is found by Algorithm 1, and further that, as Applicant’s specification provides no further context as to what a ‘similarity score’ should entail, the resultant value(s) from the algorithm are sufficient under broadest reasonable interpretation. Examiner further notes that the alogirthm is provided below); and based on the similarity score, determining, by the computer, whether or not the first target node and the second target node indicate an item that is at the same location or indicate an item that has moved on the shelf unit (“the second step ablates away inconsistent nodes from the Observed Planogram, which typically correspond to false detections yielded by the first step… After the second step the Observed Planogram should contain true detections only. Hence, those nodes that are missing compared to the Reference Planogram highlight items that appear to be missing wrt the planned product layout. The task of the third step, referred to as Product Verification, is to verify whether these product items are really missing in the scene or not. More precisely, we start considering the missing node showing the highest number of already assigned neighbors, for which we can most reliably determine a good approximation of the expected position in the image” [pages 3-4]).
PNG
media_image9.png
542
442
media_image9.png
Greyscale
PNG
media_image10.png
287
391
media_image10.png
Greyscale
Regarding claim 9, Saqlain in view of Yair teaches the method of claim 1. Tonioni further teaches wherein creating the current map comprises: iteratively determining, by the computer, a plurality of edges for each specific item in the image data (see Figure 1 under claim 1 rejections, noting that each item is associated with a plurality of edges, both as bounding boxes or edges connecting nodes as visible in e.g. the Observed Planogram of step 3).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the map generation method and shelf monitoring system as proposed by Saqlain in view of Yair to include the edge creation method for relevant objects in frame as proposed by Tonioni, the proposed motivation being that creating bounding edges around/in relation to each object of interest allows a user to quickly verify that no objects were incorrectly detected/missed/labeled, etc., ensuring an accurate system by providing an easy and quick validation method.
Regarding claim 10, Saqlain in view of Yair and Tonioni teaches the method of claim 9. Tonioni further teaches wherein iteratively determining the plurality of edges for each specific item comprises: determining, by the computer, a plurality of sectors for the specific item in the image (see Figure 1 under claim 6 rejections, noting especially the bounding boxes present in steps e.g. 2, 3, and 5. Examiner notes that as Applicant’s specification provides no further definition for ‘sector’, Examiner is interpreting it to be a bounding box equivalent under broadest reasonable interpretation ); for each sector of the plurality of sectors, identifying, by the computer, zero, one, or two neighboring items in the sector (“The graph is kept self-coherent, e.g. if node B is the North node of A, then A must be the South node of B. In case of ambiguity, e.g. both A and C found to be the South node of B, we retain the edge between the two closest bounding boxes only” [page 4], noting that in the provided example, two neighboring items were identified (both A and C)); and for each neighboring item in the sector, generating, by the computer, an edge that indicates a distance and a direction between the specific item and the neighboring item (“For each node, the search is performed along 8 cardinal directions (N, S, E, W, NW, NE, SW, SE) and, if another bounding box is found at a distance less than a dynamically determined threshold, an edge is created between the two nodes. In the given node the edge is labeled according to the search direction (e.g. N)” [page 4]. Examiner notes that, as each node only connects to other nodes within a specific distance and in specific directions, those values would be indicative of an edge existing between two nodes. As the distance value is dynamically created, the user would know that value upon performing the method themselves).
Regarding claim 11, Saqlain in view of Yair and Tonioni teaches the method of claim 10. Tonioni further teaches wherein identifying zero, one, or two neighboring items in the sector comprises: determining, by the computer, whether or not there is a closest item to the specific item in the sector (“The graph is kept self-coherent, e.g. if node B is the North node of A, then A must be the South node of B. In case of ambiguity, e.g. both A and C found to be the South node of B, we retain the edge between the two closest bounding boxes only” [page 4], noting that in the provided example, two neighboring items were identified (both A and C)); and determining, by the computer, whether or not there is a closest item that is associated with a resolvable machine readable code in the image data to the specific item in the sector (“we retain the edge between the two closest bounding boxes only” [page 4], noting that ‘machine readable codes’, which, according to Applicant’s specification, are e.g. barcodes or QR codes [0021], are present in the image data (see Fig. 4 provided below, noting the barcode on the tag in the center of the image on the shelf, which is in the image data, and is ‘associated with’ the item on the shelf above it). Examiner further notes that ‘machine readable codes’ are additionally enumerated in Saqlain and can be used to reject a claim with a more specific use for the recited codes if necessary).
PNG
media_image11.png
283
382
media_image11.png
Greyscale
Regarding claim 18, Saqlain in view of Yair teach the computer of claim 16, wherein the computer is an image analysis computer, wherein the current map is a first current map, the image data is first image data, the image is a first image, the specific items are first specific items, the item tags are first specific item tags, the shelf unit is a first shelf unit, and wherein the method further comprises: receiving second image data of a second image of a second shelf unit with second specific items and second item tags comprising machine readable codes adjacent to the second specific items; identifying the second specific items in the second image; creating a second current map based on the second specific items in the second image; determining differences between the second current map and the first current map; and determining whether or not any of the first specific items are the same as the second specific items (Examiner notes that the above limitations are ones which have already been rejected regarding Saqlain in view of Yair, simply stating that the recited steps should be performed again and compared to the original current map, which is exactly the method performed above using the current map in comparison with the company-provided one. For this reason, the limitations are rejected accordingly. For clarity, Examiner notes that the recited ‘image analysis computer’ per Applicant’s specification, must be one that can “process image data” [0042], and as the recited method is processing image data, both Saqlain and Yair contain an image analysis computer). Both Saqlain and Yair do not perform deduplicating first specific items that match second specific items in an item information database.
However, Tonioni teaches deduplicating first specific items that match second specific items in an item information database (Examiner notes that in order for a node to be counted and saved into the database as disclosed by Tonioni, there must be a case of the same node in both images, or “the second step ablates away inconsistent nodes from the Observed Planogram, which typically correspond to false detections” [pages 3-4]. In light of this disclosure, Examiner notes that duplicate node cases are ‘deduplicated’ as a singular instance of a node in a specific location is not a strong case for its existence, so requiring both instances to be present for the node to be counted once prevents node duplication from being saved into the database).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the image analysis computer as proposed by Saqlain in view of Yair to include the deduplicating as taught by Tonioni, the proposed motivation being to ensure that items are counted and produced correctly to preserve the accuracy of the proposed system.
Regarding claim 20, Saqlain in view of Yair teaches the system of claim 19 , and further wherein the current map includes a graph, wherein the one or more historical maps include graphs, wherein the central server computer receives images from user devices that include end user devices and transporter user devices (see claim 5 rejection) and stores the images in the image database (see claim 5 rejection), wherein the central server computer updates item listings in a delivery application based on one or more item information data entries in an item information database (see claim 5 rejection. Examiner notes that although a ‘delivery’ specific application was not disclosed in claim 5, the exemplary retailers provided in Yair (used to reject claim 5) are Target and Walmart. Both of these retailers have applications which contain a delivery mode or function, and as such are considered ‘delivery applications’ as they both perform a delivery function through their respective applications), and wherein the method further comprises: generating, by the image analysis computer, an item information data entry that indicates that the one or more of the items from the one or more historical maps are not present in the current map (see claim 12 rejection); and storing, by the image analysis computer, the item information data entry into the item information database, wherein the system further comprises: the item information database (see claim 12 rejection). Tonioni further teaches the recited ‘graphs’ included in the current map and the one or more historical maps (see claim 6 rejection).
Additional References Cited
Prior art made of record and not relied upon that is considered pertinent to applicant’s disclosure:
Additionally cited references (see attached PTO-892) otherwise not relied upon
above have been made of record in view of the manner in which they evidence the general state of the art. Examiner notes that the provided product webpage for ShelfView by Scandit provides a similar product to the one present in the incident disclosure and is attached for Applicant’s review. All other cited references may similarly/alternatively serve to anticipate at least the independent claims as recited, and/or provide examples of current art to Applicant as discovered and noted by Examiner during search.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JULIA T RAMETTA whose telephone number is (571)272-0451. The examiner can normally be reached Monday- Friday, 8 a.m. 5 p.m. ET..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at (571) 272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JULIA T RAMETTA/ Examiner, Art Unit 2669
/CHAN S PARK/ Supervisory Patent Examiner, Art Unit 2669
