DETAILED ACTION
The following is a non-final, first office action in response to the application filed February 6, 2025. Claims 1-20 are currently pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (abstract idea) without significantly more.
Step 1: Statutory Category
(MPEP § 2106)
Claims 1-20 are directed towards a method, a system, and a computer-readable medium. The claims are directed to a statutory category: a process, a machine, and article of manufacture as defined under 35 U.S.C. § 101.
Regarding Claim 1:
Step 2A, Prong One: Judicial Exception – Abstract Idea
(MPEP § 2106.04)
Claim 1 recites processing images depicting interactions between persons and products in a store, identifying a product taken by a person and adding the product to a virtual shopping cart associated with the person, reidentifying the person in a checkout area, obtaining a scan list comprising products scanned for the person, comparing the virtual shopping cart to the scan list, and generating a notification upon determining a discrepancy between the virtual shopping cart and the scan list.
These limitations, when considered together, recite the concept of monitoring a person's interaction with products during a commercial transaction, maintaining information regarding products associated with the person, comparing that information with products recorded at checkout, and identifying a discrepancy between the records. This concept constitutes a certain method of organizing human activity, specifically commercial interactions, and therefore recites an abstract idea.
Although the claim further recites extracting features from images, applying an embedding model to features of persons, applying trained classifiers to features of interactions and products, and classifying an interaction as taking or returning a product, these limitations provide computerized techniques by which information regarding the person, interaction, and product is identified for use in the recited commercial monitoring process. Thus, the claim as a whole recites an abstract idea.
Step 2A, Prong Two: Integration into a Practical Application
(MPEP § 2106.04(d))
The claim further recites a first set of cameras, a second set of cameras, an embedding model, first and second trained classifiers, a computer connected to a checkout terminal, and the processing of images obtained from the cameras. These additional elements do not integrate the judicial exception into a practical application.
In particular, the cameras are used to perform their ordinary function of capturing images; the computer and checkout terminal are used to obtain information regarding products scanned for a person; and the embedding model and trained classifiers are used as tools to analyze image information to identify persons, products, and interactions for use in the recited commercial monitoring process. The claim does not recite a particular improvement to the operation of the cameras, computer, checkout terminal, embedding model, classifiers, or image-processing technology itself. Rather, these elements are used to automate and implement the abstract idea of monitoring products associated with a person and comparing such products with products recorded at checkout.
Likewise, reidentifying the first person based on images captured by the second set of cameras does not, as claimed, recite a particular technological improvement to person-reidentification technology. Instead, the limitation recites the result of identifying the same person in another area of the store so that information associated with that person may be compared with checkout information.
Generating a notification upon determining a discrepancy merely communicates the result of the comparison and does not impose a meaningful limit on the judicial exception.
Accordingly, the additional elements, individually and in combination, do not reflect an improvement to the functioning of a computer or other technology or technical field, do not apply the exception with a particular machine in a manner that imposes a meaningful limit on the exception, and do not otherwise integrate the judicial exception into a practical application. Therefore, claim 1 is directed to the abstract idea.
Step 2B: Inventive Concept
(MPEP § 2106.05)
The additional elements also do not amount to significantly more than the judicial exception. The recited cameras, computer, checkout terminal, image processing, embedding model, and trained classifiers are used according to their ordinary functions to capture, process, classify, obtain, and compare information in implementing the recited commercial monitoring process. The claim does not recite additional limitations that amount to an inventive concept sufficient to transform the abstract idea into patent-eligible subject matter.
Considering the limitations both individually and as an ordered combination, the claim merely uses computer and computer-vision components to automate the underlying commercial practice of determining products associated with a customer, comparing those products with products recorded at checkout, and providing a notification when a discrepancy exists. The ordered combination does not add significantly more than the abstract idea itself.
Therefore, the claim is not directed to patent-eligible subject matter under 35 U.S.C. § 101.
Regarding Claims 9 and 17
Independent claims 9 and 17 are parallel in scope to claim 1 and ineligible for similar reasons.
Regarding Claims 2-8, 10-16, and 18-20
Dependent claims 2-8, 10-16, and 18-20 merely set forth further embellishments to the abstract idea, and therefore do not confer eligibility on the claimed invention and are ineligible for similar reasons to claim 1.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Malgarini et al (US 2024/0249342 A1).
Regarding claims 1, 9, and 17, Malgarini discloses a method comprising:
processing a first set of images captured by a first set of cameras and depicting interactions between first persons and products in a store, said processing to (a) extract features of the first set of images, (Malgarini: paragraph [0188] - An action detection module 940, which comprises a CNN 944 and a Graph Convolutional Neural Network (GCN) 944, is configured to perform designated action detection of 907 for each customer in the frame by processing the content of customer bounding boxes and body key point coordinates for each customer for a sequence of video frames);
(b) apply an embedding model to features of first persons (Malgarini: paragraph [0181] - The CNN 932, which encodes the appearance model, uses the content of each customer bounding box from the current frame and one or more previous frames. The CNN 932 outputs a feature vector for each detected customer.);
(c) apply a first trained classifier to features of interactions, and (Malgarini: paragraph [0196] - An item determination module 960 is configured to perform item determination 911 in which an item associated with a designated action and the involving customer are determined);
(d) apply a second trained classifier to features of products, wherein an interaction is classified as one of taking a product or returning a product (Malgarini: paragraph [0161] - The designated actions are a subset of detectable/recognizable actions predesignated from a set of detectable/recognizable actions which the computer vision system 230 is trained to recognize.);
identifying, based on said processing, a first product taken by a first person and adding the first product to a virtual shopping cart associated with the first person (Malgarini: paragraph [0032] - detecting, via a computer vision system, actions performed by the user from video captured by one or more cameras located in the retail environment, wherein the camera is one of a plurality of cameras in the retail environment, each camera being uniquely located such that each camera in the plurality of cameras has a unique field of view (FOV) of a portion of the retail environment, wherein the actions are one of placing an item in the physical shopping container or removing an item from the physical shopping container; determining an item associated with each detected action based on an inventory storage location at which the detected action occurred from the video captured by the camera and a planogram of the retail environment; and updating the virtual inventory of the physical shopping container based on each detected action and the associated item);
reidentifying the first person in a checkout area of the store based on a second set of images captured by a second set of cameras; (Malgarini: paragraph [0010] - the trigger is one of receiving input from a wireless communication device of the user to perform an electronic checkout or a determination that the user and/or physical shopping container is in a detection zone of the retail environment);
obtaining, from a computer connected to a checkout terminal, a scan list for the first person, the scan list comprising products scanned for the first person; (Malgarini: paragraph [0093] - A user 101 of the shopping application 560 may add item 103 to the inventory of the electronic shopping cart in a number of ways, depending on the item 103. One method of adding items to the inventory of the electronic shopping cart that is suitable for packaged items 103 is scanning a barcode (e.g., UPC or QR code) associated with the item 103 via a camera of the wireless communication device 220);
comparing the virtual shopping cart to the scan list; and generating a notification upon determining a discrepancy between the virtual shopping cart and the scan list (Malgarini: paragraph [0104] - The performing of the security check comprises determining whether the virtual inventory of the physical shopping container matches the inventory of the electronic shopping cart. In response to a determination that the virtual inventory of the physical shopping container does not match the inventory of the electronic shopping cart, the status of the cart is electronic shopping cart is changed to alarmed or the like, and an alert is generated.).
Regarding claims 2 and 10, Malgarini discloses all of the limitations as noted above in claims 1 and 9. Malgarini further discloses sending the notification to a device operated by a store supervisor (Malgarini: paragraph [0104] - Depending on how and when the security check is triggered, the alert may comprise one or a combination of an electronic alert in the form of an electronic message or notification, an audible generated by a speaker 132 in the retail environment 100, or a visual alert generated by a light 234 in the retail environment 100. An electronic alert may be generated, sent to, and displayed upon any one or more of the wireless communication device 220 of the user and/or a computing device of the merchant).
Regarding claims 4 and 12, Malgarini discloses all of the limitations as noted above in claims 1 and 9. Malgarini further discloses wherein the notification contains information regarding the discrepancy, the information indicating at least one of:(a) a product in the virtual shopping cart is not in the scan list,(b) a product in the scan list is not in the virtualshopping cart,(c) a quantity of a product in the virtual shopping is greater than a quantity of the product in the scan list, and (d) a quantity of a product in the virtual shopping is less than a quantity of the product in the scan list (Malgarini: paragraph [0108] - In one example, an electronic alert may be used in response to a trigger. The trigger may be generated in response to a determination that the virtual inventory of the physical shopping container does not match the inventory of the electronic shopping cart (i.e., the status of the cart is electronic shopping cart is changed to alarmed or the like)).
Regarding claims 5 and 13, Malgarini discloses all of the limitations as noted above in claims 1 and 9. Malgarini further discloses wherein the checkout area of the store includes one or both of (a) in a vicinity of a checkout terminal of the store, and (b) in a vicinity of an exit of the store (Malgarini: paragraph [0012] - In some or all examples of the first aspect, the detection zone is an entrance/exit zone of the retail environment).
Regarding claims 6, 14, and 18, Malgarini discloses all of the limitations as noted above in claims 1, 9, and 17. Malgarini further discloses wherein reidentifying the first person in the second set of images comprises: extracting features of a second person from the second set of images; applying the embedding model to the features extracted from the second set of images to generate a descriptor of the second person from the second set of images; comparing the descriptor generated for the second person from the second set of images to one or more descriptors that were generated for the first person from the first set of images; and based on results of the comparing, determining that the second person in the second set of images is the first person in the first set of images (Malgarini: paragraph [0180] - The Hungarian optimization algorithm submodule 936 applies the Hungarian algorithm, described more fully below, that is a combinatorial optimization algorithm which is used to solve the assignment problem between all the detected (or propagated) customer bounding boxes between the current and previous frames. By solving the assignment problem, the Hungarian algorithm determines which customer bounding boxes from previous and current frames should have the same ID. In other words, the Hungarian algorithm matches the detected customer bounding boxes from the previous frames with the detected customer bounding boxes of the new frames and the tracking algorithm assigns the matching detected customer bounding boxes the same ID).
Regarding claims 7, 15, and 19, Malgarini discloses all of the limitations as noted above in claims 1, 9, and 17. Malgarini further discloses tracking the first person across multiple images captured by a third set of cameras covering differentfields of view (FOV) of the store, wherein the tracking is performed at least in part using FOV mapping data that describes spatial and temporal relationships between the different FOVs (Malgarini: paragraph [0183] - The computer vision system 900 may need to track customers in multiple cameras 104 and so uses a mechanism to re-identify customers if they leave the FOV of one camera 104 and enter the FOV of another camera 104. In addition, if the same customer is image by multiple cameras 104 at the same time, the computer vision system 900 may need to determine that the customer in the video frame of the multiple cameras 104 is the same person).
Regarding claims 8, 16, and 20, Malgarini discloses all of the limitations as noted above in claims 7, 15, and 19. Malgarini further discloses generating the FOV mapping data, by, during a time period prior to the tracking: processing a plurality of images captured during the time period by the first set of cameras to generate descriptors for persons depicted in the plurality of images; identifying recurring patterns of movement of similar descriptors within or between FOVs of different cameras of the first set of cameras to thereby learn the spatial and temporal relationships between the different FOVs (Malgarini: paragraph [0183] - The computer vision system 900 may need to track customers in multiple cameras 104 and so uses a mechanism to re-identify customers if they leave the FOV of one camera 104 and enter the FOV of another camera 104. In addition, if the same customer is image by multiple cameras 104 at the same time, the computer vision system 900 may need to determine that the customer in the video frame of the multiple cameras 104 is the same person).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
PTO-892 Reference U discloses IoT Applications on Secure Smart Shopping System.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATHLEEN G PALAVECINO whose telephone number is (571)270-1355. The examiner can normally be reached M-F 9-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Marissa Thein can be reached at (571) 272-6764. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
KATHLEEN GAGE PALAVECINO
Primary Examiner
Art Unit 3688
/KATHLEEN PALAVECINO/Primary Examiner, Art Unit 3688
Prosecution Insights