DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 17 is objected to because of the following informalities:
Claim 17, line 2 – “deferent devices” should be “different devices”.
Appropriate correction is required.
Allowable Subject Matter
Claim 8 is objected to as being dependent upon a rejected base claim, but would be allowable if the claim is incorporated into the corresponding independent claim including all of the limitations of the base claim and any corresponding, intervening claims. The following is a statement of reasons for the indication of allowable subject matter:
In regards to dependent claim 8, none of the cited prior art alone or in combination provides motivation to teach “wherein the processor applies a persistent marker to objects in the 3D scene to maintain their identification as the sensor's viewpoint changes, such that 2D images of an object-captured over time and from different perspectives-are automatically annotated with the object's identification even when the neural network fails to identify the object in a specific view, thereby enabling self-supervised annotation and subsequent updating of the deep neural network training” as the references teach using deep neural networks for product identification for items on shelves as different camera perspectives are captured, however the references fail to explicitly teach the functions of for automatically annotating the object's identification even when the neural network fails to identify the object in a specific view to facilitate automated processing without needing manual interference, in conjunction with the features of claim 1 from which it depends where final identification for an object is based in a consensus analysis derived from a view of the scene.
In addition, there is no teaching, suggestion, or motivation found in the current references and none that can be inferred from the examiner’s own knowledge with respect to the current limitation.
As allowable subject matter has been indicated, applicant's reply must either comply with all formal requirements or specifically traverse each requirement not complied with. See 37 CFR 1.111(b) and MPEP § 707.07(a).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-7 and 9-20 are rejected under 35 U.S.C. 103 as being unpatentable over Skaff (US 2022/0303445 A1, hereinafter referenced “Skaff”) in view of Kim (2016 “Hierarchical committee of deep convolutional neural networks for robust facial expression recognition”, hereinafter referenced “Kim”).
In regards to claim 1. Skaff discloses a system for identifying objects in a three-dimensional (3D) scene (Skaff, para [0009]), comprising:
-a sensor configured to capture visual data of the environment (Skaff, para [0058]; Reference discloses the autonomous robot has multiple cameras 140 attached to movable base 110 by a vertically extending camera support 141….The object sensing suite includes forward 133, side 134 and 135, top 132 and rear (not shown) image and depth sensors to aid in object detection, localization, and navigation (i.e. the cameras and sensors provide visual data of environment));
-at least one processor (Skaff, para [0085]; Reference discloses as shown in FIG. 7, the camera system 700 includes processor 701, memory 702, image sensor 703, communication interfaces 704, camera optical and actuator system 705, and storage 706); and
-a non-transitory computer-readable medium storing instructions that, when executed by the processor (Skaff, para [0147]; Reference discloses some embodiments described herein relate to a computer storage product with a non-transitory computer-readable medium (also can be referred to as a non-transitory processor-readable medium) having instructions or computer code thereon for performing various computer-implemented operations), cause the processor to:
-i. generate a spatial map of the 3D scene using simultaneous localization and mapping (SLAM) techniques (Skaff, para [0063]; Reference discloses using absolute location sensors, relative distance measurements to the shelves, triangulation to a known landmark, conventional simultaneous localization and mapping (SLAM) methodologies, or relying on beacons positioned at known locations in a blueprint or a previously built map, the robots 230 and 232 can move along a path generally parallel to shelves 202. As the robots 230 and 232 move, vertically positioned cameras (e.g., a stack of cameras associated with each of the robots) are synchronized to simultaneously capture images of the shelves 202 (i.e. generated spatial map of the scene using techniques such as SLAM));
-ii. detect objects within the spatial map and delineate bounding regions for each detected object (Skaff, para [0072] and [0074]; Reference at [0072] discloses a robot moves to an identified position and proceeds along an aisle path at a predetermined distance (step 410) (i.e. navigation within spatial map). Para [0074] discloses typically, stitched or original images are segmented, and the segmented images are used to help define a product bounding box that putatively identifies a product facing…A segmented image can include multiple product bounding boxes (i.e. detected objects delineated by bounding regions), typically ranging from dozens to hundreds of outlined or distinct image areas);
-iii. assign an identification to each object based on an analysis of captured visual data using a deep neural network (Skaff, para [0074] and [0076]; Reference at [0074] discloses typically, stitched or original images are segmented, and the segmented images are used to help define a product bounding box that putatively identifies a product facing. This information is often necessary to develop a product library. Para [0076] discloses an alternative to extracting product descriptors is to use the bounding boxes as labeled categories and train classifiers on the images contained in the bounding boxes…A deep learning based classifier can automatically learn image descriptors based on an annotated training data (i.e. identification of products in the 2D images using deep neural network); and
-iv. determine a final identification for each object based Skaff, Fig. 2 and para [0074] and [0075]; Reference at para [0074] discloses typically, stitched or original images are segmented, and the segmented images are used to help define a product bounding box that putatively identifies a product facing…this information is often necessary to develop a product library. A segmented image can include multiple product bounding boxes, typically ranging from dozens to hundreds of outlined or distinct image areas. Para [0075] discloses in one embodiment, the product bounding box, with suitable identifiers, can be registered to a simple or panoramic stitched image of the shelf, and image descriptors extracted for the portion of the image contained in the bounding box (i.e. final identification of object based on the images or view of the 3D scene and identifiers)).
Skaff does not explicitly disclose but Kim teaches
-(identification based) on a consensus analysis (Kim, “5.2 Hierarchical Committee of Deep CNN” continued page 181; Reference discloses hierarchical committee building in which deep CNN’s are organized in different structural levels as it details Therefore, we decide to use the EWA rule only for the first level. For decision fusions in higher levels, the majority voting and simple average rules are used for a better generalization (interpreted as consensus analysis functions for neural network classification aggregation))
Skaff and Kim are combinable because they are in the same field of endeavor regarding deep learning detection. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for image capture and shelf content system of Skaff to include the hierarchical deep convolutional neural network features of Kim in order to provide a system that allows for use of a mobile system which tracks target objects for inventory monitoring as taught by Skaff while incorporating the hierarchical deep convolutional neural network features of Kim to allow for implementation of a hierarchical architecture of the committee with exponentially-weighted decision fusion for identifying various features in an image with improved performance above general CNN architectures, thus applicable to shelf content and status tracking systems implemented with classification features such as those taught by Skaff.
In regards to claim 2. Skaff in view of Kim teach the system of claim 1.
Skaff further discloses
-wherein the processor continuously updates Skaff, para [0063]; Reference discloses FIG. 2 is a graphical illustration 200 depicting two autonomous robots 230 and 232, inspecting opposite shelves 202 in an aisle. As shown, each robot follows path 205 along the length of an aisle, with multiple cameras capturing images of the shelves 202 (i.e. additional visual data regarding images of shelf captured from different perspectives over time also see further para [0099] disclosing various camera orientations).
Skaff does not explicitly disclose but Kim teaches
-(updates) the consensus analysis (Kim, “3.2.2 Hierarchical Committee Architecture” and “5.2 Hierarchical Committee of Deep CNN” continued page 181; Reference discloses hierarchical committee building in which deep CNN’s are organized in different structural levels as it details Therefore, we decide to use the EWA rule only for the first level. For decision fusions in higher levels, the majority voting and simple average rules are used for a better generalization (interpreted as consensus analysis functions for neural network classification aggregation)).
Skaff and Kim are combinable because they are in the same field of endeavor regarding deep learning detection. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for image capture and shelf content system of Skaff to include the hierarchical deep convolutional neural network features of Kim in order to provide a system that allows for use of a mobile system which tracks target objects for inventory monitoring as taught by Skaff while incorporating the hierarchical deep convolutional neural network features of Kim to allow for implementation of a hierarchical architecture of the committee with exponentially-weighted decision fusion for identifying various features in an image with improved performance above general CNN architectures, thus applicable to shelf content and status tracking systems implemented with classification features such as those taught by Skaff.
In regards to claim 3. Skaff in view of Kim teach the system of claim 1.
Skaff further discloses
-wherein Skaff, para [0063] and [0076]; Reference discloses FIG. 2 is a graphical illustration 200 depicting two autonomous robots 230 and 232, inspecting opposite shelves 202 in an aisle. As shown, each robot follows path 205 along the length of an aisle, with multiple cameras capturing images of the shelves 202 (i.e. images of shelf captured from different perspectives over time). Para [0076] discloses an alternative to extracting product descriptors is to use the bounding boxes as labeled categories and train classifiers on the images contained in the bounding boxes. Classifiers may include those based on deep structured learning, hierarchical learning, deep machine learning, or other suitable deep learning algorithms associated with convolutional, feedforward, recurrent, or other suitable neural network. A deep learning based classifier can automatically learn image descriptors based on an annotated training data. For example, deep learning based image descriptors can be hierarchical, corresponding to multiple layers in deep convolutional neural networks. (i.e. final identification of multiple views of an object).
Skaff does not explicitly disclose but Kim teaches
-the consensus (analysis) (Kim, “3.2.2 Hierarchical Committee Architecture” and “5.2 Hierarchical Committee of Deep CNN” continued page 181; Reference discloses hierarchical committee building in which deep CNN’s are organized in different structural levels as it details Therefore, we decide to use the EWA rule only for the first level. For decision fusions in higher levels, the majority voting and simple average rules are used for a better generalization (interpreted as consensus analysis functions for neural network classification aggregation)).
Skaff and Kim are combinable because they are in the same field of endeavor regarding deep learning detection. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for image capture and shelf content system of Skaff to include the hierarchical deep convolutional neural network features of Kim in order to provide a system that allows for use of a mobile system which tracks target objects for inventory monitoring as taught by Skaff while incorporating the hierarchical deep convolutional neural network features of Kim to allow for implementation of a hierarchical architecture of the committee with exponentially-weighted decision fusion for identifying various features in an image with improved performance above general CNN architectures, thus applicable to shelf content and status tracking systems implemented with classification features such as those taught by Skaff.
In regards to claim 4. Skaff in view of Kim teach the system of claim 1.
Skaff further discloses
-wherein the sensor comprises an image sensor configured to capture two-dimensional images of the 3D scene (Skaff, para [0058]; Reference discloses the autonomous robot has multiple cameras 140 attached to movable base 110 by a vertically extending camera support 141….The object sensing suite includes forward 133, side 134 and 135, top 132 and rear (not shown) image and depth sensors to aid in object detection, localization, and navigation).
In regards to claim 5. Skaff in view of Kim teach the system of claim 1.
Skaff further discloses
-further comprising a network interface configured to receive object identification data from one or more devices, potentially obtained at different times, wherein the processor aggregates the received data to enhance identification accuracy (Skaff, para [0063]-[0065]; Reference at [0063] discloses each robot follows path 205 along the length of an aisle, with multiple cameras capturing images of the shelves 202. In some embodiments, the robots 230 and 232 support at least one range finding sensor to measure distance between the multiple cameras and the shelves and products on shelves…Using absolute location sensors, relative distance measurements to the shelves, triangulation to a known landmark, conventional simultaneous localization and mapping (SLAM) methodologies, or relying on beacons positioned at known locations in a blueprint or a previously built map, the robots 230 and 232 can move along a path generally parallel to shelves 202…. Para [0064] discloses data can be communicated to a server local to the store, or transmitted by suitable internet or networking devices to remote company servers or cloud accessible data sites (i.e. network interface). Para [0065] discloses the inventory cameras 340…to aid in product identification or image construction...(The robots use of cameras following a path simultaneous capturing images of shelves that also possess cameras and moving down an aisle for product identification interpreted as receiving object identification data from one or more devices potentially at different times as the robot moves and the data aggregated in regards to product identification).
In regards to claim 6. Skaff in view of Kim teach the system of claim 1.
Skaff further discloses
-wherein the system is operable on devices selected from the group consisting of mobile devices, wearable devices, augmented reality (AR) glasses, robots, mixed reality headset, drones or any device that is capable of moving or being moved around a 3D scene (Skaff, para [0010]; Reference discloses the movable base can be a manually pushed or guidable cart. Alternatively, in some embodiments, the movable base can be a tele-operated robot or an autonomous robot capable of guiding itself through a store or warehouse).
In regards to claim 7. Skaff in view of Kim teach the system of claim 1.
Skaff further discloses
-further comprising a non-transitory computer-readable medium storing instructions executable by the processor to update a machine learning model based on crowd-sourced data from multiple devices (Skaff, para [0079]; Reference discloses in one embodiment, products within product bounding boxes can be manually identified, identified using crowd source or paid reviewer image identification systems, identified with or without the aid of an initial planogram or realogram, or automatically identified using various image classifiers discussed herein).
In regards to claim 9. Skaff in view of Kim teach the system of claim 1.
Skaff further discloses
-wherein the processor is configured to assign a unique identifier to an object upon its initial detection and to automatically apply the unique identifier to subsequent detections of the object from different viewpoints, thereby autonomously tagging the object without additional manual input (Skaff, para [0075] and [0083]; Reference at [0075] discloses in one embodiment, the product bounding box, with suitable identifiers, can be registered to a simple or panoramic stitched image of the shelf, and image descriptors extracted for the portion of the image contained in the bounding box (i.e. product identifier interpreted as assigned unique identifier to an object upon initial detection). Para [0083] discloses if a product library is created or made available, the library can be searched for realogram related information. For example, products objects with a large number of similar features can be used to assist in developing the product bounding box. For each potential product object match, the geometric consistency of the feature locations in the library can be compared with the features in a shelf image (i.e. interpreted as allowing for subsequent classification or “tagging” of an object without manual input given the established library)).
In regards to claim 10. Skaff discloses a method for identifying objects in a three-dimensional (3D) scene (Skaff, para [0009]), comprising:
-capturing visual data of the environment with a sensor (Skaff, para [0058]; Reference discloses the autonomous robot has multiple cameras 140 attached to movable base 110 by a vertically extending camera support 141….The object sensing suite includes forward 133, side 134 and 135, top 132 and rear (not shown) image and depth sensors to aid in object detection, localization, and navigation (i.e. the cameras and sensors provide visual data of environment));
-generating a spatial map of the 3D scene using SLAM techniques (Skaff, para [0063]; Reference discloses using absolute location sensors, relative distance measurements to the shelves, triangulation to a known landmark, conventional simultaneous localization and mapping (SLAM) methodologies, or relying on beacons positioned at known locations in a blueprint or a previously built map, the robots 230 and 232 can move along a path generally parallel to shelves 202. As the robots 230 and 232 move, vertically positioned cameras (e.g., a stack of cameras associated with each of the robots) are synchronized to simultaneously capture images of the shelves 202 (i.e. generated spatial map of the scene using techniques such as SLAM));
-detecting objects within the spatial map and delineating bounding regions for each detected object (Skaff, para [0072] and [0074]; Reference at [0072] discloses a robot moves to an identified position and proceeds along an aisle path at a predetermined distance (step 410) (i.e. navigation within spatial map). Para [0074] discloses typically, stitched or original images are segmented, and the segmented images are used to help define a product bounding box that putatively identifies a product facing…A segmented image can include multiple product bounding boxes (i.e. detected objects delineated by bounding regions), typically ranging from dozens to hundreds of outlined or distinct image areas);
-assigning an identification to each object using a deep neural network based on captured visual data (Skaff, para [0074] and [0076]; Reference at [0074] discloses typically, stitched or original images are segmented, and the segmented images are used to help define a product bounding box that putatively identifies a product facing. This information is often necessary to develop a product library. Para [0076] discloses an alternative to extracting product descriptors is to use the bounding boxes as labeled categories and train classifiers on the images contained in the bounding boxes…A deep learning based classifier can automatically learn image descriptors based on an annotated training data (i.e. identification of products in the 2D images using deep neural network); and
-determining a final identification for each object based Skaff, Fig. 2 and para [0074] and [0075]; Reference at para [0074] discloses typically, stitched or original images are segmented, and the segmented images are used to help define a product bounding box that putatively identifies a product facing…this information is often necessary to develop a product library. A segmented image can include multiple product bounding boxes, typically ranging from dozens to hundreds of outlined or distinct image areas. Para [0075] discloses in one embodiment, the product bounding box, with suitable identifiers, can be registered to a simple or panoramic stitched image of the shelf, and image descriptors extracted for the portion of the image contained in the bounding box (i.e. final identification of object based on the images or view of the 3D scene and identifiers)).
Skaff does not explicitly disclose but Kim teaches
-(identification based) on a consensus analysis (Kim, “5.2 Hierarchical Committee of Deep CNN” continued page 181; Reference discloses hierarchical committee building in which deep CNN’s are organized in different structural levels as it details therefore, we decide to use the EWA rule only for the first level. For decision fusions in higher levels, the majority voting and simple average rules are used for a better generalization (interpreted as consensus analysis functions for neural network classification aggregation))
Skaff and Kim are combinable because they are in the same field of endeavor regarding deep learning detection. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for image capture and shelf content system of Skaff to include the hierarchical deep convolutional neural network features of Kim in order to provide a system that allows for use of a mobile system which tracks target objects for inventory monitoring as taught by Skaff while incorporating the hierarchical deep convolutional neural network features of Kim to allow for implementation of a hierarchical architecture of the committee with exponentially-weighted decision fusion for identifying various features in an image with improved performance above general CNN architectures, thus applicable to shelf content and status tracking systems implemented with classification features such as those taught by Skaff.
In regards to claim 11. Skaff in view of Kim teach the method of claim 10.
Skaff further discloses
-further comprising continuously updating Skaff, para [0063]; Reference discloses FIG. 2 is a graphical illustration 200 depicting two autonomous robots 230 and 232, inspecting opposite shelves 202 in an aisle. As shown, each robot follows path 205 along the length of an aisle, with multiple cameras capturing images of the shelves 202 (i.e. updating based on additional visual data regarding images of shelf captured from different perspectives over time also see further para [0099] disclosing various camera orientations).
Skaff does not explicitly disclose but Kim teaches
-(updates) the consensus analysis (Kim, “3.2.2 Hierarchical Committee Architecture” and “5.2 Hierarchical Committee of Deep CNN” continued page 181; Reference discloses hierarchical committee building in which deep CNN’s are organized in different structural levels as it details Therefore, we decide to use the EWA rule only for the first level. For decision fusions in higher levels, the majority voting and simple average rules are used for a better generalization (interpreted as consensus analysis functions for neural network classification aggregation)).
Skaff and Kim are combinable because they are in the same field of endeavor regarding deep learning detection. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for image capture and shelf content system of Skaff to include the hierarchical deep convolutional neural network features of Kim in order to provide a system that allows for use of a mobile system which tracks target objects for inventory monitoring as taught by Skaff while incorporating the hierarchical deep convolutional neural network features of Kim to allow for implementation of a hierarchical architecture of the committee with exponentially-weighted decision fusion for identifying various features in an image with improved performance above general CNN architectures, thus applicable to shelf content and status tracking systems implemented with classification features such as those taught by Skaff.
In regards to claim 12. Skaff in view of Kim teach the method of claim 10.
Skaff further discloses
-wherein the deep neural network is implemented as a hierarchical system of neural networks operating sequentially, the hierarchy including a first level for categorizing objects, a second level for identifying a manufacturer or brand, and a third level for identifying a specific object (Skaff, para [0059], [0076], and [0082]; Reference at [0059] discloses typically, one or more shelf units or fixtures (e.g., target 102) would be imaged by a diverse set of camera types,…zoomable and focusable camera 146 to capture barcodes, product identification numbers, and shelf labels. Alternatively, a high resolution, tilt controllable camera can be used to identify shelf labels (i.e. identify a specific object). These camera 140 derived images can be stitched together, with products in the images identified, and position determined. Para [0076] disclose an alternative to extracting product descriptors is to use the bounding boxes as labeled categories and train classifiers on the images contained in the bounding boxes. Classifiers may include those based on deep structured learning, hierarchical learning, deep machine learning, or other suitable deep learning algorithms associated with convolutional, feedforward, recurrent, or other suitable neural network (i.e. categorizing objects). A deep learning based classifier can automatically learn image descriptors based on an annotated training data. For example, deep learning based image descriptors can be hierarchical, corresponding to multiple layers in deep convolutional neural networks. The final layer of a convolutional layer network outputs the confidence values of the product (identifying specific product) being in one of the designated image categories. Para [0082] discloses classification algorithms such as convolution neural networks or other deep learning methods, template matching or HAAR cascades can be used to aid in detection of each shelf label. Each shelf label is analyzed to obtain one or more product identifiers….Product identifiers may be UPC code, the product name…(i.e. identifying a brand)).
In regards to claim 13. Skaff in view of Kim teach the method of claim 10.
Skaff further discloses
-further comprising storing the hierarchical deep neural network on a long-term memory and uploading selected levels to a short-term memory prior to capturing visual data, so that identification is performed as an edge computing process (Skaff, para [0070]; Reference discloses the communication system 316 can include connections to both a wired or wireless connect subsystem for interaction with devices such as servers, desktop computers, laptops, tablets, or smartphones…. In addition, sources of local data (e.g. a hard drive, solid state drive, flash memory, or any other suitable memory, including dynamic memory, such as SRAM or DRAM) that can allow for local data storage of user-specified preferences or protocols)
In regards to claim 14. Skaff in view of Kim teach the method of claim 10.
-further comprising casting an identification Skaff, para [0075], [0079], and [0083]; Reference at [0075] discloses in one embodiment, the product bounding box, with suitable identifiers, can be registered to a simple or panoramic stitched image of the shelf, and image descriptors extracted for the portion of the image contained in the bounding box (i.e. cast identification via corresponding bounding box and associate each bounding box with an identifier signifying the identified product). Para [0079] discloses gaps between products are useful for identifying shelf spacings, product separation, or missing/absent inventor (i.e. unable to identify product due to absence interpreted as even when the deep neural network fails to assign one regarding identification). Para [0083] discloses if a product library is created or made available, the library can be searched for realogram related information. For example, products objects with a large number of similar features can be used to assist in developing the product bounding box. For each potential product object match, the geometric consistency of the feature locations in the library can be compared with the features in a shelf image).
Skaff does not explicitly disclose but Kim teaches
-(identification) from the consensus analysis (Kim, “3.2.2 Hierarchical Committee Architecture” and “5.2 Hierarchical Committee of Deep CNN” continued page 181; Reference discloses hierarchical committee building in which deep CNN’s are organized in different structural levels as it details Therefore, we decide to use the EWA rule only for the first level. For decision fusions in higher levels, the majority voting and simple average rules are used for a better generalization (interpreted as consensus analysis functions for neural network classification aggregation))
Skaff and Kim are combinable because they are in the same field of endeavor regarding deep learning detection. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for image capture and shelf content system of Skaff to include the hierarchical deep convolutional neural network features of Kim in order to provide a system that allows for use of a mobile system which tracks target objects for inventory monitoring as taught by Skaff while incorporating the hierarchical deep convolutional neural network features of Kim to allow for implementation of a hierarchical architecture of the committee with exponentially-weighted decision fusion for identifying various features in an image with improved performance above general CNN architectures, thus applicable to shelf content and status tracking systems implemented with classification features such as those taught by Skaff.
In regards to claim 15. Skaff in view of Kim teach the method of claim 10.
Skaff further discloses
-further comprising receiving a user-generated identification of an object within a captured image and applying the user- generated identification to additional instances of the object (Skaff, para [0078]-[0079]; Reference at [0078] discloses in still other embodiments, the image descriptors can be classified and labelled with the identifier. Para [0079] discloses products within product bounding boxes can be manually identified (i.e. receive and apply user-generated object ID’s using images), identified using crowd source or paid reviewer image identification systems, identified with or without the aid of an initial planogram or realogram).
In regards to claim 16. Skaff in view of Kim teach the method of claim 10.
Skaff further discloses
-further comprising applying a persistent marker to objects in the environment to maintain their identification as the sensor's viewpoint changes (Skaff, para [0063] and [0065]; Reference at [0063] discloses each robot follows path 205 along the length of an aisle, with multiple cameras capturing images of the shelves 202. In some embodiments, the robots 230 and 232 support at least one range finding sensor to measure distance between the multiple cameras and the shelves and products on shelves ….Using absolute location sensors, relative distance measurements to the shelves, triangulation to a known landmark, conventional simultaneous localization and mapping (SLAM) methodologies, or relying on beacons positioned at known locations in a blueprint or a previously built map, the robots 230 and 232 can move along a path generally parallel to shelves 202….The depth map is registered onto the images captured by the shelf cameras, so the location of each pixel on target can be estimated in 3D. Para [0065] discloses the inventory cameras 340…to aid in product identification or image construction... Embodiments can also include sensors capable of detecting infrared, ultraviolet, or other wavelengths to allow for hyperspectral image processing. This can allow, for example, monitoring and tracking of markers, labels or guides that are not visible to people (The robots use of cameras following a path simultaneous capturing images of shelves while moving down an aisle for product identification and 3D estimation techniques via markers interpreted as apply a persistent marker to objects in the environment to maintain their identification as the sensor's viewpoint changes, as the image sensor changes its vantage point relative to the shelf)).
In regards to claim 17. Skaff in view of Kim teach the method of claim 10.
Skaff further discloses
-further comprising transmitting object identification data from deferent devices via a network interface and aggregating such data to enhance identification accuracy (Skaff, para [0063]-[0065]; Reference at [0063] discloses each robot follows path 205 along the length of an aisle, with multiple cameras capturing images of the shelves 202. In some embodiments, the robots 230 and 232 support at least one range finding sensor to measure distance between the multiple cameras and the shelves and products on shelves…Using absolute location sensors, relative distance measurements to the shelves, triangulation to a known landmark, conventional simultaneous localization and mapping (SLAM) methodologies, or relying on beacons positioned at known locations in a blueprint or a previously built map, the robots 230 and 232 can move along a path generally parallel to shelves 202…. Para [0064] discloses data can be communicated to a server local to the store, or transmitted by suitable internet or networking devices to remote company servers or cloud accessible data sites (i.e. network interface). Para [0065] discloses the inventory cameras 340…to aid in product identification or image construction...(The robots use of cameras following a path simultaneous capturing images of shelves that also possess cameras and moving down an aisle for product identification interpreted as receiving object identification data from one or more devices potentially at different times as the robot moves and the data aggregated in regards to product identification).
In regards to claim 18. Skaff in view of Kim teach the method of claim 10.
Skaff further discloses
-wherein the method is performed on a device selected from the group consisting of mobile devices, wearable devices, augmented reality (AR) glasses, robots, and drones (Skaff, para [0010]; Reference discloses the movable base can be a manually pushed or guidable cart. Alternatively, in some embodiments, the movable base can be a tele-operated robot or an autonomous robot capable of guiding itself through a store or warehouse).
In regards to claim 19. Skaff in view of Kim teach the method of claim 10.
Skaff further discloses
-further comprising updating a machine learning model based on crowd-sourced data from multiple devices, possibly over time, including long periods of time (Skaff, para [0079]; Reference discloses in one embodiment, products within product bounding boxes can be manually identified, identified using crowd source or paid reviewer image identification systems, identified with or without the aid of an initial planogram or realogram, or automatically identified using various image classifiers discussed herein).
In regards to claim 20. Skaff in view of Kim teach the method of claim 10.
Skaff further discloses
-further comprising assigning a unique identifier to an object upon its initial detection and automatically applying the unique identifier to subsequent detections of the object from different viewpoints, thereby autonomously tagging the object without additional manual input (Skaff, para [0075] and [0083]; Reference at [0075] discloses in one embodiment, the product bounding box, with suitable identifiers, can be registered to a simple or panoramic stitched image of the shelf, and image descriptors extracted for the portion of the image contained in the bounding box (i.e. product identifier interpreted as assigned unique identifier to an object upon initial detection). Para [0083] discloses if a product library is created or made available, the library can be searched for realogram related information. For example, products objects with a large number of similar features can be used to assist in developing the product bounding box. For each potential product object match, the geometric consistency of the feature locations in the library can be compared with the features in a shelf image (i.e. interpreted as allowing for subsequent classification or “tagging” of an object without manual input given the established library)).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: See the Notice of References Cited (PTO-892)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TERRELL M ROBINSON whose telephone number is (571)270-3526. The examiner can normally be reached 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, KENT CHANG can be reached at 571-272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TERRELL M ROBINSON/Primary Examiner, Art Unit 2614