DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on April 16, 2026 has been entered.
Claims 42, 46, and 61 have been amended.
Claims 1-41 and 62-221 are cancelled.
Claims 222-223 are added.
Claims 42-61 are pending.
The effective filing date of the claimed invention is May 3, 2024, and claims priority dating back to March 6, 2019.
Response to Amendment
Amendments to Claims 42, 46, and 61 are acknowledged. Amendments to Claims 42, 46, and 61 are sufficient to overcome the 35 USC 103 rejection of Claims 42-61.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 42-61 and 222-223 are rejected under 35 U.S.C. 101 because the claimed invention is directed a judicial exception (i.e., an abstract idea) without significantly more.
Step 1 – Statutory Categories
As indicated in the preamble of the claim, the examiner finds the claim is directed to a process, machine, manufacture, or composition of matter.(Claim 61 are processes and Claims 42-60 are machines). Accordingly, step 1 is satisfied.
Step 2A – Prong 1: was there a Judicial Exception Recited
Claim 42 (and similarly Claims 46 and 61) recites the following abstract concepts that are found to include “abstract idea.” Any additional elements will be analyzed under Step 2A-Prong 2 and Step 2B:
at least one imaging sensor;
a data interface;
an image processing unit comprising at least one processor in electronic communication with the at least one imaging sensor via the data interface and configured to:
access a plurality of planograms stored in a database, wherein each planogram describes a desired placement of products on shelves of a retail store during a time period, wherein the plurality of planograms includes at least a first planogram and a second planogram (See MPEP 2106.04(a)(2)(III) mental processes, a claim to “collecting information, analyzing it, and displaying certain results of the collection and analysis,” where the data analysis steps are recited at a high level of generality such that they could practically be performed in the human mind, Electric Power Group v. Alstom, S.A., 830 F.3d 1350, 1353-54, 119 USPQ2d 1739, 1741-42 (Fed. Cir. 2016));
from the at least one imaging sensor, via the data interface, electronically receive a first set of images captured at a first time and depicting a first plurality of products displayed on at least one of the shelves of the retail store (See MPEP 2106.04(a)(2)(III) mental processes, a claim to “collecting information, analyzing it, and displaying certain results of the collection and analysis,” where the data analysis steps are recited at a high level of generality such that they could practically be performed in the human mind, Electric Power Group v. Alstom, S.A., 830 F.3d 1350, 1353-54, 119 USPQ2d 1739, 1741-42 (Fed. Cir. 2016));
with a machine learning classifier model, automatically analyze the first set of images by performing at least one of object recognition, object detection, image segmentation, or feature extraction to determine at least one visual characteristic of each of the first plurality of products, the at least one visual characteristic including at least one of a size, a shape, a color, a logo, or text associated with the product, and determining a confidence level associated with a determined type of each product, wherein the confidence level is compared to a threshold, to determine an actual placement of the first plurality of products displayed on the at least one of the shelves of the retail store at the first time (See MPEP 2106.04(a)(2)(III) mental processes, a claim to “collecting information, analyzing it, and displaying certain results of the collection and analysis,” where the data analysis steps are recited at a high level of generality such that they could practically be performed in the human mind, Electric Power Group v. Alstom, S.A., 830 F.3d 1350, 1353-54, 119 USPQ2d 1739, 1741-42 (Fed. Cir. 2016) and PEG Example 42, Claim 2, wherein under the broadest reasonable interpretation of the training and application of an artificial neural network encompasses mental processes practically performed in the human mind by observation, evaluation, judgement, and opinion.);
with a relationship determination software component, automatically identify a deviation of the actual placement of at least some of the first plurality of products from the desired placement of products associated with the first planogram (See MPEP 2106.04(a)(2)(III) mental processes, a claim to “collecting information, analyzing it, and displaying certain results of the collection and analysis,” where the data analysis steps are recited at a high level of generality such that they could practically be performed in the human mind, Electric Power Group v. Alstom, S.A., 830 F.3d 1350, 1353-54, 119 USPQ2d 1739, 1741-42 (Fed. Cir. 2016));
with a display screen, issue a first user-notification associated with the deviation of the actual placement of the at least some of the first plurality of products from the desired placement of products associated with the first planogram (See MPEP 2106.04(a)(2)(III) mental processes, a claim to “collecting information, analyzing it, and displaying certain results of the collection and analysis,” where the data analysis steps are recited at a high level of generality such that they could practically be performed in the human mind, Electric Power Group v. Alstom, S.A., 830 F.3d 1350, 1353-54, 119 USPQ2d 1739, 1741-42 (Fed. Cir. 2016));
after issuing the first user-notification, receive from the at least one imaging sensor a second set of images captured at a second time and depicting a second plurality of products displayed on the at least one of the shelves of the retail store (See MPEP 2106.04(a)(2)(III) mental processes, a claim to collecting and comparing known information (claim 1), which are steps that can be practically performed in the human mind, Classen Immunotherapies, Inc. v. Biogen IDEC, 659 F.3d 1057, 1067, 100 USPQ2d 1492, 1500 (Fed. Cir. 2011));
with the machine learning classifier model, automatically analyze the second set of images to determine an actual placement of the second plurality of products displayed on the shelves of the retail store at the second time (See MPEP 2106.04(a)(2)(III) mental processes, a claim to collecting and comparing known information (claim 1), which are steps that can be practically performed in the human mind, Classen Immunotherapies, Inc. v. Biogen IDEC, 659 F.3d 1057, 1067, 100 USPQ2d 1492, 1500 (Fed. Cir. 2011) and PEG Example 42, Claim 2, wherein under the broadest reasonable interpretation of the training and application of an artificial neural network encompasses mental processes practically performed in the human mind by observation, evaluation, judgement, and opinion.);
with the relationship determination software component, automatically:
identify a deviation of the actual placement of at least some of the second plurality of products from the desired placement of products associated with the first planogram (See MPEP 2106.04(a)(2)(III) mental processes, a claim to collecting and comparing known information (claim 1), which are steps that can be practically performed in the human mind, Classen Immunotherapies, Inc. v. Biogen IDEC, 659 F.3d 1057, 1067, 100 USPQ2d 1492, 1500 (Fed. Cir. 2011));
use the second planogram to determine whether an arrangement associated with the second plurality of products conforms to the second planogram rather than to the first planogram, by determining whether the arrangement conforms to the second planogram comprises using a trained machine learning model that was trained using training examples including a plurality of alternative planograms and images of actual arrangements of products, together with labels indicating which planogram of the plurality of planograms the depicted actual placement of products is most conforming with, to analyze the second set of images and the plurality of planograms and select a most conforming planogram (See MPEP 2106.04(a)(2)(III) mental processes, a claim to collecting and comparing known information (claim 1), which are steps that can be practically performed in the human mind, Classen Immunotherapies, Inc. v. Biogen IDEC, 659 F.3d 1057, 1067, 100 USPQ2d 1492, 1500 (Fed. Cir. 2011)); and
when the arrangement associated with the second plurality of products conforms to the second planogram, avoid issuance of a second user- notification indicating a deviation relative to the first planogram (See MPEP 2106.04(a)(2)(III) mental processes, a claim to collecting and comparing known information (claim 1), which are steps that can be practically performed in the human mind, Classen Immunotherapies, Inc. v. Biogen IDEC, 659 F.3d 1057, 1067, 100 USPQ2d 1492, 1500 (Fed. Cir. 2011)).
Claim 42 (and similarly Claims 46 and 61) is directed to a series of steps for managing planogram data, which are mental processes. The mere nominal recitation of at least one imaging sensor, a data interface, and imaging processing unit, a processor, a relationship classifier software component, a display screen, a database, and a non-transitory computer-readable medium (Claim 46) does not take the claim out of the method of organizing human interactions. Thus, Claim 42 (and similarly Claims 46 and 61) recites an abstract idea.
Step 2A – Prong 2: Can the Judicial Exception Recited be integrated into a practical application
Limitations that are indicative of integration into a practical application:
Improvements to the functioning of a computer, or to any other technology or technical field - see MPEP 2106.05(a)
Applying or using a judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition – see Vanda Memo
Applying the judicial exception with, or by use of, a particular machine - see MPEP 2106.05(b)
Effecting a transformation or reduction of a particular article to a different state or thing - see MPEP 2106.05(c)
Applying or using the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception - see MPEP 2106.05(e) and Vanda Memo
Limitations that are not indicative of integration into a practical application:
Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f)
Adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g)
Generally linking the use of the judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h)
The identified abstract idea of exemplary Claim 42 (and similarly Claims 46 and 61) is not integrated into a practical application. The additional elements are: at least one imaging sensor, a data interface, and imaging processing unit, a processor, a relationship classifier software component, a display screen, a database, and a non-transitory computer-readable medium (Claim 46) that implements the underlying abstract idea. These additional elements are broadly recited computer elements that do not add a meaningful limitation to the abstract idea because they amount to merely using a computer as a tool to perform an abstract idea - see MPEP 2106.05(f).
Accordingly, alone and in combination, these additional elements do not integrate the abstract idea into a practical application. Claim 42 (and similarly Claims 46 and 61) is directed to an abstract idea.
Step 2B – Significantly More Analysis
Claim 42 (and similarly Claims 46 and 61) does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered separately and in combination, steps a) access a plurality of planograms, b) receive a first set of images, c) analyze the first set of images, d) identify a deviation of the actual placement of at least some of the first plurality of products, e)issue a first user-notification, f) receive a second set of images, g) analyze the second set of images, h) use the second planogram to determine whether an arrangement associated with the second plurality of products conforms to the second planogram, and h) when the arrangement associated with the second plurality of products conforms to the second planogram, avoid issuance of a second user-notification, do not add significantly more to the exception because they amount to merely using a computer as a tool to perform an abstract idea - see MPEP 2106.05(f). Claim 42 (and similarly Claims 46 and 61) is ineligible.
Claim 43 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Claim 44 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Claim 45 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Claim 47 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Claim 48 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Claim 49 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Claim 50 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Claim 51 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Claim 52 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Claim 53 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Claim 54 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Claim 55 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Claim 56 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Claim 57 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Claim 58 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Claim 59 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Claim 60 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Claim 222 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Claim 223 recites the abstract idea of mental processes. See MPEP 2106.04(a)(2)(III).
Prior Art
The prior arts of record fail to teach the overall combination as claimed for Claims 42-61 and 222-223. Therefore, it would not have been obvious to one of ordinary skill in the art to modify the prior art to meet the combination above without unequivocal hindsight and one of ordinary skill would have no reason to do so. Exemplary claim 42 recites the following:
A system for processing images captured in a retail store and automatically identifying changes in planograms, the system comprising:
at least one imaging sensor;
a data interface;
an image processing unit comprising at least one processor in electronic communication with the at least one imaging sensor via the data interface and configured to:
access a plurality of planograms stored in a database, wherein each planogram describes a desired placement of products on shelves of a retail store during a time period, wherein the plurality of planograms includes at least a first planogram and a second planogram;
from the at least one imaging sensor, via the data interface, electronically receive a first set of images captured at a first time and depicting a first plurality of products displayed on at least one of the shelves of the retail store;
with a machine learning classifier model, automatically analyze the first set of images by performing at least one of object recognition, object detection, image segmentation, or feature extraction to determine at least one visual characteristic of each of the first plurality of products, the at least one visual characteristic including at least one of a size, a shape, a color, a logo, or text associated with the product, and determining a confidence level associated with a determined type of each product, wherein the confidence level is compared to a threshold, to determine an actual placement of the first plurality of products displayed on the at least one of the shelves of the retail store at the first time;
with a relationship determination software component, automatically identify a deviation of the actual placement of at least some of the first plurality of products from the desired placement of products associated with the first planogram;
with a display screen, issue a first user-notification associated with the deviation of the actual placement of the at least some of the first plurality of products from the desired placement of products associated with the first planogram;
after issuing the first user-notification, receive from the at least one imaging sensor a second set of images captured at a second time and depicting a second plurality of products displayed on the at least one of the shelves of the retail store;
with the machine learning classifier model, automatically analyze the second set of images to determine an actual placement of the second plurality of products displayed on the shelves of the retail store at the second time;
with the relationship determination software component, automatically:
identify a deviation of the actual placement of at least some of the second plurality of products from the desired placement of products associated with the first planogram;
use the second planogram to determine whether an arrangement associated with the second plurality of products conforms to the second planogram rather than to the first planogram, by determining whether the arrangement conforms to the second planogram comprises using a trained machine learning model that was trained using training examples including a plurality of alternative planograms and images of actual arrangements of products, together with labels indicating which planogram of the plurality of alternative planograms the depicted actual placement of products is most conforming with, to analyze the second set of images and the plurality of planograms and select a most conforming planogram; and
when the arrangement associated with the second plurality of products conforms to the second planogram, avoid issuance of a second user-notification indicating a deviation relative to the first planogram. (Emphasis added to highlight features that distinguish over the prior art).
As further explained below, the prior art of record, alone or in combination, neither anticipates, reasonably teaches, nor renders obvious the Applicant’s claimed invention.
US Pat Pub 2018/0150788 “Vepakomma” discloses an inventory control in an establishment, wherein a system receives sensor data, planogram data, and image data. Based on the sensor data, the system determines current position of products placed at product support devices (PSD). Further, the current position is compared with predefined arrangement defined in the planogram data. Further, the system determines planogram compliance metric, based on the comparison, indicating deviation of placement of the products. The system further identifies the products in the PSDs based on the image data. Vepakomma fails to disclose using a second planogram to determine whether an arrangement associated with a second plurality of products conforms to the second planogram rather than to a first planogram, by using a trained machine learning model that was trained using training examples including a plurality of alternative planograms and images of actual arrangements of products, together with labels indicating which planogram of the plurality of alternative planograms the depicted actual placement of products is most conforming with, to analyze the second set of images and the plurality of planograms and select a most conforming planogram.
US Pat Pub 2017/0178227 “Graham” teaches aligning a realogram and a planogram. An alignment module of an image recognition application receives a realogram, the realogram including information about product recognitions, and a planogram corresponding to the realogram. The alignment module also generates a planogram brand chunk in the planogram, the planogram brand chunk grouping a plurality of planogram product facings belonging to a same brand, and a realogram brand chunk in the realogram based on the planogram brand chunk. The alignment module additionally identifies a planogram product facing in the planogram brand chunk and a realogram product facing in the realogram brand chunk and aligns the planogram product facing with the realogram product facing. Graham fails to teach using a second planogram to determine whether an arrangement associated with a second plurality of products conforms to the second planogram rather than to a first planogram, by using a trained machine learning model that was trained using training examples including a plurality of alternative planograms and images of actual arrangements of products, together with labels indicating which planogram of the plurality of alternative planograms the depicted actual placement of products is most conforming with, to analyze the second set of images and the plurality of planograms and select a most conforming planogram.
US Pat Pub 2019/0236526 “Sosna” teaches managing visual product placement. A user computing device may have trained machine learning models to detect a shelf and a target product. A shelf may be detected by a computing device with a machine learning model, and the scope of the shelf may be divided into a number of small boxes, each corresponding to a product on the shelf. A first target product and its actual placement information may be detected with a machine learning model. The actual placement information may be compared with a set of requirements for visual placement of the first target product on the shelf. Deviation and adjustment to correct the deviation may be determined, and the adjustment may be displayed in an AR environment. Sosna fails to teach using a second planogram to determine whether an arrangement associated with a second plurality of products conforms to the second planogram rather than to a first planogram, by using a trained machine learning model that was trained using training examples including a plurality of alternative planograms and images of actual arrangements of products, together with labels indicating which planogram of the plurality of alternative planograms the depicted actual placement of products is most conforming with, to analyze the second set of images and the plurality of planograms and select a most conforming planogram.
Response to Arguments
35 USC 101
Applicant's arguments filed April 16, 2026 have been fully considered but they are not persuasive.
Applicant argues that the judicial exception is integrated into a practical application because the technical details transform the ML analysis from an abstract “analyze” step into a concrete, multi-step image-processing pipeline. The Examiner finds that the “multi-step image-processing pipeline” amounts to using ML to analyze images for similarities to other images or planograms, and fails to see an improvement to the actual ML itself. Instead, the use of ML in the claims is most similar to PEG Example 47, Claim 2, and thus, classified as mental processes.
35 USC 103
Applicant’s arguments, see Applicant Arguments/Remarks Made in an Amendment, filed April 16, 2026, with respect to 35 USC 103 have been fully considered and are persuasive. The 35 USC 103 rejection of Claims 42-61 has been withdrawn.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to REVA R MOORE whose telephone number is (571)270-7942. The examiner can normally be reached M-Th: 9:00-6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Fahd Obeid can be reached at 571-270-3324. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/REVA R MOORE/Examiner, Art Unit 3627
/FAHD A OBEID/Supervisory Patent Examiner, Art Unit 3627
Prosecution Insights