DETAILED ACTION
Remarks
This 2nd non-final office action is in response to the RCE amendments filled on 05/26/2026. Claims 1, 2, 9, 10, 17 and 18 are amended. Claims 1-24 are pending and examined below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 9, 10, 17 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2022/0383384 (“Bronicki”), and further in view of US 2020/0118401 (“Zalewski”).
Regarding claim 1 (and similarly claim 9 and 17), Bronicki discloses a method comprising: tracking, by at least one fixed overhead imaging assembly, a location of an individual associated with a container within a zone of a venue (see at least [0154], where “As shown in FIG. 6A, the at least one processor contained in a second housing 504 may control a plurality of image capture devices 506… Controlling image capturing device 506 may include instructing image capturing device 506 to capture an image and/or transmit captured images to a remote server”; see also fig 5B, where 506 is camera. See also [0143], where the imaging system, 500 is mounted with the retail shelving unit. So, the imaging assembly is fixed overhead. See also [0139]; shopping cart is interpreted as container), the fixed overhead imaging assembly being mounted above the zone and capturing first image data including the individual and the container while the individual moves through the zone (see at least fig 6A, where location of a person/individual, 608 with cart inside a retail store is identified by using camera.);
detecting at least one of the container or at least one object within the container present in first image data captured by the at least one fixed overhead imaging assembly (see at least [0127], where “server 135 may execute an image processing algorithm to identify in received images one or more products and/or obstacles, such as shopping carts, people, and more.”; see also [0138], where “Differing numbers of capturing devices 125 may be used to cover shelving unit 402”);
identifying at least one of the at least one object or a region of interest associated with the container present in the first image data captured by the at least one fixed overhead imaging assembly (see at least [0148], where “FIG. 5C illustrates an exploded view of second housing 504. In some embodiments, the network interface located in second housing 504 (e.g., network interface 306) may be configured to transmit to server 135 information associated with a plurality of images captured by image capture device 506. For example, the transmitted information may be used to determine if a disparity exists between at least one contractual obligation (e.g., planogram) and product placement.”; the overhead fixed imaging unit is capturing images of a person with container and server is analyzing images for region of interest e.g., product placement. So, a region of interest associated with the container is identified by images captured by overhead camera);
determining, based on the identification, at least one of a value of at least one attribute of the at least one object or a first sub-area and a second sub-area of the region of interest (see at least [0152], where “system 500B may receive output signals from a sensing system located on second retail shelving unit 604. The output signals may be indicative of a sensed lifting of a product from second retail shelving unit 604 or a sensed positioning of a product on second retail shelving unit 604.”; see also [0162], where “the at least one attribute associated with retail shelving unit 640 may include a lighting condition, the dimensions of opposing retail shelving unit 640, the size of products displayed on opposing retail shelving unit 640, the type of labels used on opposing retail shelving unit 640, and more. In some embodiments, the attribute may be determined, based on analysis of one or more acquired images, by at least one processor contained in second housing 504. Alternatively, the attribute may be automatically sensed and conveyed to the at least one processor contained in second housing 504.”; see also [0153] and [0354]);
determining whether at least one of the value of the at least one attribute is greater than a first threshold or a ratio of the first sub-area and the second sub-area is less than a second threshold (see at least [0162], where images of a particular product is analyzed. See also [0161], where “issue a warning when a change is detected, when a change larger than a selected threshold is detected, when a change is detected for a duration longer than a selected threshold”; change larger than a threshold include attribute related to the area/shelve. see also [0105] and [0106]); and
generating and transmitting a notification to a device when at least one of the value of the at least one attribute is greater than the first threshold or the ratio of the first sub-area and the second sub-area is less than the second threshold (see at least [0223], where “FIG. 11D, GUI 1130 may include a first display area 1132 for showing a list of notifications or text messages indicating selected in-store execution events that require attention. The notifications or text messages may include a link to an image (or the image itself) of the specific aisle with the in-store execution event.”).
Bronicki does not disclose the following limitation:
the notification being indicative of the location of the individual and instructions to the device to navigate to the individual based on the location.
However, Zalewski discloses a method wherein the notification being indicative of the location of the individual (see at least [0033], where “monitoring a changing location of the user inside the store and providing navigation guidance to an asset in the store or to the portable device of the user, said guidance aiding the user in traversing through the store along a navigation path through isles that contain one or more items on the shopping list of the user account”; see also [0215]) and instructions to the device to navigate to the individual based on the location (see at least [0220], where “One guidance overlay is an arrow showing navigational direction to an item in the store.”).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Bronicki to incorporate the teachings of Zalewski by including the above feature for reducing human involvement in the retail store and providing faster shopping experience for user.
Regarding claim 2 (and similarly claim 10 and 18), Bronicki further discloses a method wherein the container is at least one of a cart, a basket, a bin, a platform truck, a hand truck, or a dolly (see at least fig 6A, where a shopping cart is shown).
Bronicki does not disclose the following limitation:
the device is at least one of an autonomous mobile robot (AMR) transporting a container or an AMR integrated with a container.
However, Zalewski further discloses a method wherein the device is at least one of an autonomous mobile robot (AMR) transporting a container or an AMR integrated with a container (see at least [0854], where “A delivery system consisting of a human or autonomous or self-driving cart that meets the shopper in the store is provided, to deliver the pre-fetched items.”).
Claim(s) 3, 11 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2022/0383384 (“Bronicki”), in view of 2020/0118401 (“Zalewski”), as applied to claim 1, 9 and 17 above, and further in view of US 2015/0187080 (“Kundu”).
Regarding claim 3 (and similarly claim 11 and 19), Bronicki further discloses a method wherein the at least one attribute of the at least one object is a weight of the at least one object (see at least [0181]).
Bronicki in view of Zalewski does not disclose the following limitation:
the first sub-area is indicative of non-occupied space within the container and the second sub-area is indicative of occupied space within the container.
However, Kundu discloses a method wherein the first sub-area is indicative of non-occupied space within the container and the second sub-area is indicative of occupied space within the container (see at least [0059], where “During video analysis 150-1, the Cart Inspector 150-1 obtains a target image. The target image can be a video image 170-1 that shows the shopping cart 210 was empty at 2 o'clock when it was in the transaction area 200. In addition, the Cart Inspector 150-1 obtains a reference representation, which can be a predefined image of an empty cart 150-3.”; see also fig 3).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Bronicki in view of Zalewski to incorporate the teachings of Kundu by including the above feature for reducing number of transaction and faster task completion by evaluation amount of space remain on the cart.
Claim(s) 4, 12 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2022/0383384 (“Bronicki”), and in view of US 2020/0118401 (“Zalewski”), as applied to claim 1, 9 and 17 above, and further in view of US 2025/0029406 (“Kim”).
Regarding claim 4 (and similarly claim 12 and 20), Bronicki further discloses a method comprising localizing at least one of the detected container or the detected at least one object (see at least [0108], where “a trained machine learning algorithm may include an object detector, the input may include an image, and the inferred output may include one or more detected objects in the image and/or one or more locations of objects within the image.”; see also fig 6A, where location of cart is shown).
Bronicki in view of Zalewski does not disclose the following limitation:
localizing at least one of the detected container or the detected at least one object by removing background noise from the first image data.
However, Kim discloses a method wherein localizing at least one of the detected container or the detected at least one object by removing background noise from the first image data (see at least [0150], where “the noise removal part 325 may identify a preset object in the image through segmentation based on artificial intelligence (AI) that has been previously machine-trained.”).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Bronicki in view of Zalewski to incorporate the teachings of Kim by including the above feature for increasing accuracy of object detection by removing noise.
Claim(s) 5-8, 13-16 and 21-24 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2022/0383384 (“Bronicki”), and in view of US 2020/0118401 (“Zalewski”), as applied to claim 1, 9 and 17 above, and further in view of US 2022/0292815 (“Donnelly”).
Regarding claim 5 (and similarly claim 13 and 21), Bronicki further discloses a method wherein identifying the at least one object comprises:
generating, by applying a feature extractor model to the first image data, at least one object descriptor indicative of one or more features of the detected at least one object (see at least [0117], [0231] and [0354]);
executing, by a visual search engine, a or more known objects to determine an object (see at least [0231] and [0117]); and
selecting, by the visual search engine, a known object corresponding to the detected at least one object from a ranked list of known objects, (see at least [0183] and [0513]).
Bronicki in view of Zalewski does not disclose the following limitations:
a nearest neighbor search within a database storing one or more known object descriptors corresponding to respective image data of one or more known objects to determine a respective metric distance between the at least one object descriptor and the one or more known object descriptors; and
a known object corresponding to the detected at least one object from a ranked list of known objects, the ranked list of known objects being prioritized based on a respective metric distance between the at least one object descriptor and the one or more known object descriptors.
However, Donnelly discloses a method wherein executing a nearest neighbor search within a database storing one or more known object descriptors corresponding to respective image data of one or more known objects to determine a respective metric distance between the at least one object descriptor and the one or more known object descriptors (see at least [0088], where “a CNN.sub.1 classifies the target item by using the descriptor F of the target item to retrieve a most similar shape in a data set, rather than by supplying the descriptor F to a second stage CNN.sub.2. For example, all of the objects in the training set may be supplied to the first stage CNN.sub.1 to generate a set of known descriptors {F.sub.ds(m)}, where the index m indicates a particular labeled shape in the training data. A similarity metric is defined to measure the distance between any two given descriptors (vectors) F and F.sub.ds(m). Some simple examples of similarity metrics are a Euclidean vector distance and a Mahalanobis vector distance.”); and
selecting a known object corresponding to the detected at least one object from a ranked list of known objects, the ranked list of known objects being prioritized based on a respective metric distance between the at least one object descriptor and the one or more known object descriptors (see at least [0084], where “In max-pooling, the n feature vectors are combined to generate a single combined feature vector or descriptor F, where the j-th entry of the descriptor F is equal to the maximum among the j-th entries among the n feature vectors f The resulting descriptor F has the same length (or rank) as the n feature vectors f and therefore descriptor F can also be supplied as input to the second stage CNN.sub.2 to compute a classification of the object.”; see also [0072]).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Bronicki in view of Zalewski to incorporate the teachings of Donnelly by including the above feature for reducing shopping error by identifying the object using database models and similarity search.
Regarding claim 6 (and similarly claim 14 and 22), Bronicki further discloses a method wherein the feature extractor model is a machine learning model comprising a convolutional neural network classifier or visual transformer classifier trained on one or more of supervised learning tasks or unsupervised learning tasks (see at least [0110], where “analyzing image data (for example by the methods, steps and modules described herein) may comprise analyzing the image data and/or the preprocessed image data using one or more rules, functions, procedures, artificial neural networks, object detection algorithms”).
Regarding claim 7 (and similarly claim 15 and 23), Bronicki further discloses a method wherein for each known object represented in the database, the database stores at least one attribute of the known object including one or more of (i) a known object location, (ii) a known object weight, or (iii) a known object volume (see at least [0117], where “Server 135 may access database 140 to detect and/or identify products. The detection may occur through analysis of features in the image using an algorithm and stored data. The identification may occur through analysis of product features in the image according to stored product models…the product model may include a description of visual and contextual properties of the particular product (e.g., the shape, the size, the colors, the texture, the brand name, the price, the logo, text appearing on the particular product, the shelf associated with the particular product, adjacent products in a planogram, the location within the retail store, etc.)”).
Regarding claim 8 (and similarly claim 16 and 24), Bronicki further discloses a method wherein the at least one object descriptor and the one or more known object descriptors are indicative of one or more features comprising one or more of a shape, a color, a height, a width, or a length (see at least [0117]).
Bronicki in view of Zalewski does not disclose the following limitation:
at least one object descriptor and the one or more known object descriptors correspond to vectors and the respective metric distance between the at least one object descriptor and the one or more known object descriptors corresponds to differences between respective vectors of the at least one object descriptor and the one or more known object descriptors.
However, Donnelly further discloses a method wherein the at least one object descriptor and the one or more known object descriptors correspond to vectors and the respective metric distance between the at least one object descriptor and the one or more known object descriptors corresponds to differences between respective vectors of the at least one object descriptor and the one or more known object descriptors (see at least [0015], where “The identification model includes feature vectors attributable to visual patterns of a tray or surgical tool, implant, fastener, or other object identified in each training dataset. The feature vectors can be combined into matrices to provide a 2-dimensional array of feature vectors.”; see also [0036], [0040], [0082-84]). Same motivation of claim 5 applies.
Response to Arguments
Applicant’s arguments with respect to claim 1-24 have been considered but are moot because the arguments do not apply to the new combination used in the current rejection.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOHANA TANJU KHAYER whose telephone number is (408)918-7597. The examiner can normally be reached on Monday - Thursday, 7 am-5.30 pm, PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached on 5712703976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SOHANA TANJU KHAYER/Primary Examiner, Art Unit 3657
Prosecution Insights