DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/30/2026 has been entered.
Claim Status
Claims 1, 3-11, and 13-22 are pending for examination in the application filed 06/30/2026. Claims 1, 6, 11, and 13-16 have been amended, claims 2 and 12 have been cancelled, and claims 21-22 are new.
Priority
Acknowledgement is made of Applicant’s claim to priority of provisional application 63/439,701, filing date 01/18/2023.
Response to Arguments and Amendments
The 35 U.S.C. 112(a) rejections of claims 6 and 16 are withdrawn in view of the amendments.
Applicant's arguments filed 06/30/2026 regarding independent claims 1 and 16 have been fully considered but they are not persuasive. Applicant argues on page 9 of the Remarks that Sangeneni does not disclose filtering out images that have a quality level that enables object identification and that one of ordinary skill would not combine Sangeneni with Meckesheimer as states in the Office Action to make obvious the steps of claim 1. As stated on pages 8-10 of the Final Rejection filed 04/02/2026: Sangeneni teaches selecting a subset of the images ([0048] Frame selection and deblurring as disclosed herein comprises analyzing multiple image frames for blur and using the frames with the least amount of blur for analysis) and Meckesheimer, in the same field of endeavor of cargo image analysis, teaches selecting a subset of the images that includes a limited number of the images based on a predetermined factor of where the cargo is located within the image ([0030] For example, in some embodiments, parameter module 130 may be configured to process the image data using key frame extraction techniques to determine which frames to be used for evaluation. For example, in some embodiments, parameter module 130 may be configured to filter the image data based on one or more of motion detection, object detection, image quality, etc. For example, in some embodiments, parameter module 130 may be configured to use frames without a predetermined level of movement, and/or without a specific object (e.g., forklift image, human image, or other objects in the image). In some embodiments, parameter module 130 may be configured to use monocular depth measurement techniques (e.g., perspective cues) to locate proper guidelines and/or calculate depth. For example, visual gradient may be used to locate guidelines, product edges, trailer head, and/or other information). In order to establish a prima facie case of obviousness, Examiner must set forth (a) the relevant teachings of the prior art relied upon, (b) the differences between the prior art in the claim and the applied references, (c) the proposed modification of the applied references necessary to arrive at the claimed subject matter, and (d) an explanation as to why the claimed invention would have been obvious to one of ordinary skill in the art at the relevant time. See MPEP 2142. Here, Examiner has mapped the Sangeneni reference to the claim, explained the deficiencies of the Sangeneni reference, proposed a modification of the Sangeneni reference with the Meckesheimer reference, and provided a motivation for the combination. Therefore, a prima facie case of obviousness has been made. Similarly, in independent claim 16, as discussed on page 10 of the Remarks, Sangeneni teaches a working set of images comprising a limited number of the images ([0048] Frame selection and deblurring as disclosed herein comprises analyzing multiple image frames for blur and using the frames with the least amount of blur for analysis) and Meckesheimer teaches after identifying whether the cargo is present in the images, determining a working set of the images comprising a limited number of the images in which the cargo is present in the image ([0030] parameter module 130 may be configured to filter the image data based on one or more of motion detection, object detection, image quality, etc. For example, in some embodiments, parameter module 130 may be configured to use frames without a predetermined level of movement, and/or without a specific object (e.g., forklift image, human image, or other objects in the image)) Please see the updated 35 U.S.C 103 rejections based on the newly added amendments.
Applicant’s arguments with respect to independent claim 11 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument, as facilitated by the newly added amendments. Please see the updated 35 U.S.C 103 rejections based on the newly added amendments.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 20 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 20 recites the limitation “determining the one or more aspects of the cargo based on a limited number of the images that are received”. Claim 20 depends from claim 16 where there is no recitation of “the one or more aspects of the cargo”. Please clarify.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 3-4, 7-8, 10, 16, 18-20, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Sangeneni (US20210319582A1) in view of Meckesheimer (US20220366556A1).
Regarding claim 1, Sangeneni teaches a computing device configured to monitor cargo that is being loaded onto a vehicle (Fig. 2 [0006] a system for continuous volume determination of the cargo space of a vehicle…generate an updated spatial model of the cargo space using the images from the plurality of cameras, upon detection of parcels being loaded or unloaded into the cargo space),
the computing device comprising: memory circuitry; processing circuitry configured to operate according to programming instructions stored in the memory circuitry ([0020] Generally, the processor receives (reads) instructions and content from a memory (such as a read-only memory and/or a random-access memory) and writes (stores) instructions and content to the memory) to:
receive images of the cargo from electro-optical sensors affixed to the vehicle ([Abstract] A method of determining volume of a cargo space of a vehicle. [0006] images from a plurality of cameras including at least one camera for capturing depth and color, positioned in and around the cargo space);
identify one or more objects within the images ([0063] First, an initial data collected from the cameras (208) is processed by a CPU and reformatted in a way that is acceptable to the deep learning GPU model (GPU Model 1). The GPU Model 1 takes about 23 milliseconds to recognize and crop out the barcode, label, QR code, or box using DNN);
selecting a subset of the images ([0064] Next, once the label is cropped out, the system (104) then proceeds to find the barcode in a second GPU model…the blur value (GPU Model 3) is also detected. [0047] Decoding of the barcode from images may comprise image frame selection, deblurring and label extraction from such selected and deblurred frames. [0048] Frame selection and deblurring as disclosed herein comprises analyzing multiple image frames for blur and using the frames with the least amount of blur for analysis);
based on just the images of the subset, compute one or more aspects of the cargo including a volume of the cargo ([0006] volume estimation unit (214) configured to estimate a volume of loaded parcels in the updated spatial model);
determine a position on the vehicle where the cargo is loaded based on the subset of images of the cargo ([0042] Furthermore, as illustrated by block 314, the identification unit (216) may be used for identification of locations of newly loaded objects into the cargo space by generating new updated spatial models of the cargo space each time movement is detected);
and based on the loaded position on the vehicle, associate one or more aspects with the cargo including the volume of the cargo ([0040] Next, as illustrated in FIG. 3B block 310, a volume estimation unit (214) of the system (104) may map an area of the bounding box within the total volume of the cargo space, where an estimate of the remaining cargo volume may be determined. [0006] the volume estimation unit (214) is configured to determine a remaining volume of the cargo space based on the said estimated volume of the loaded parcels and a total volume of the cargo space, wherein the total volume is calculated based on the initial spatial model).
Sangeneni does not explicitly teach selecting a subset of the images that includes a limited number of the images based on a predetermined factor of where the cargo is located within the image.
Meckesheimer, in the same field of endeavor of cargo image analysis, teaches selecting a subset of the images that includes a limited number of the images based on a predetermined factor of where the cargo is located within the image ([0030] parameter module 130 may be configured to filter the image data based on one or more of motion detection, object detection, image quality, etc. For example, in some embodiments, parameter module 130 may be configured to use frames without a predetermined level of movement, and/or without a specific object (e.g., forklift image, human image, or other objects in the image). [0026] “Cargo content” and “cargo items” refer to items located inside the container, items to be loaded (placed) inside the container, and/or items unloaded (removed) from the container).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the device of Sangeneni with the teachings of Meckesheimer to select a subset of images based on where the cargo is located within the image because "image data may be combined, filtered, and processed to identify an object (e.g., container, container components, and/or cargo items). Once identified, the object may be measured, and its position and orientation extracted." [Meckesheimer 0029].
Regarding claim 3, Sangeneni and Meckesheimer teach the device of claim 1. Sangeneni further teaches wherein the processing circuitry is configured to: determine a temporary position of the cargo within an alignment area within the vehicle ([0078] At blocks 508 and 510, the method goes on to model a multi-dimensional bounding box that could be 3D corresponding to the clustered planes and identifying the position of the object in the cargo space based on the position of the multi-dimensional bounding box within the DPC or global coordinate system);
determine a lane within the vehicle that the cargo enters after being positioned at the alignment area ([0073] The region of interest may be a location with a high probability of locating an object. For example, in a cargo space where the objects are loaded on shelves, the area of the shelves may be determined as the ROI whereas the full cargo space will be in the field of view. The ROI may be identified using the technique of subtracting space as described in the previous paragraphs with regards to detection of movement and subtraction of pixels from the model);
and determine a location along the lane where the cargo is finally positioned ([0080] Further on repetition of the method FIG. 5, it may be determined that the location of an object has changed from the position identified during the previous iteration of the method. For example, the system (104) will be able to identify what movement occurred between frames by identifying a similar bounding box at a different position from the previous iteration of the method and thereby determine a change in position).
Regarding claim 4, Sangeneni and Meckesheimer teach the device of claim 1. Sangeneni further teaches communication circuitry configured to upload through a communication network the position of the cargo on the vehicle while onboard the vehicle ([0042] transceivers (224) may be used to notify a user (e.g. a delivery person) of the position of a parcel by transmitting the location to e.g. devices (108)).
Regarding claim 7, Sangeneni and Meckesheimer teach the device of claim 1. Sangeneni further teaches wherein the processing circuitry is configured to determine a point on the cargo based on the images and track the position of the cargo within the vehicle based on the point ([0005] In an aspect, images from each camera of the plurality of cameras are stitched together into a Dense Point Cloud (DPC) in real-time. The disclosed method further comprises generating an updated spatial model of the cargo space using the images from the plurality of cameras, upon detection of parcels or objects being loaded or unloaded into the cargo space, wherein the updated spatial model includes changes to the cargo space and estimating a volume of loaded parcels in the updated spatial model).
Regarding claim 8, Sangeneni and Meckesheimer teach the device of claim 1. Sangeneni further teaches wherein the processing circuitry is further configured to: receive the images from a primary camera and one or more subordinate cameras; include in the subset of images the images from the primary camera that capture the cargo; include in the subset of images the images from the one or more subordinate cameras that were taken at the same time as the images that are selected from the primary camera ([0035] Movement may be analyzed by combining data from two types of cameras i.e. a depth camera (such as IR depth cameras) and an RGB camera. [0038] The combination of these three sensors relays to the system exactly where movement is happening. At the same time, depth cameras pointed at the shelves enable the system (104) to monitor movement in the shelf zones, and a computer vision DNN classifies objects within those regions either as parcels or non-parcels (eg, a hand). [0048] Frame selection and deblurring as disclosed herein comprises analyzing multiple image frames for blur and using the frames with the least amount of blur for analysis).
and identifying the cargo based on the subset of images ([0038] The combination of these three sensors relays to the system exactly where movement is happening. At the same time, depth cameras pointed at the shelves enable the system (104) to monitor movement in the shelf zones, and a computer vision DNN classifies objects within those regions either as parcels or non-parcels (eg, a hand)).
Regarding claim 10, Sangeneni and Meckesheimer teach the device of claim 1. Sangeneni further teaches wherein the processing circuitry is further configured to calculate one of a 3D mesh or a 3D point cloud of the cargo based on the images and determine the volume of the cargo using the 3D mesh or the 3D point cloud ([0029] The model generation unit (212) is configured to stitch the data from the camera (208) in the form of a DPC or a mesh in real time. [0067] Referring now to FIG. 4, a flowchart illustrating a method of determining volume of a cargo space of a vehicle in real-time is shown. The method starts at block 402 where an initial spatial model of the cargo space is generated using images from a plurality of cameras (208). [0070] Next, at block 406, volume of the loaded parcels in the updated spatial model are estimated using the volume estimation unit (214)).
Regarding claim 16, Sangeneni teaches a method of monitoring cargo that is being loaded onto a vehicle ([0005] a method of determining the volume of a cargo space of a vehicle in real-time…generating an updated spatial model of the cargo space using the images from the plurality of cameras, upon detection of parcels or objects being loaded or unloaded into the cargo space), the method comprising:
receiving images of the cargo from a plurality of electro-optical sensors ([0006] images from a plurality of cameras including at least one camera for capturing depth and color, positioned in and around the cargo space);
determining a working set of the images comprising a limited number of the images ([0048] Frame selection and deblurring as disclosed herein comprises analyzing multiple image frames for blur and using the frames with the least amount of blur for analysis);
determining a shape and size of the cargo based on just the working set of the images ([0074] Further as illustrated at block 504, a plurality of planes are extracted from the ROI wherein the plurality of planes correspond to the geometry of the object to be identified. [0077] Now at block 506, similar and nearby planes are clustered together using the clustering unit (220). The clustering is based on a weight of the two or more planes of the plurality of planes wherein the weight is assigned based on a property of orthogonality and dimensions of two or more planes of the plurality of planes);
determining a point on the cargo based on just the working set of the images ([0005] In an aspect, images from each camera of the plurality of cameras are stitched together into a Dense Point Cloud (DPC) in real-time);
determining a volume of the cargo based on just the working set of the images ([0006] a volume estimation unit (214) configured to estimate a volume of loaded parcels in the updated spatial model);
and determining a position on the vehicle where the cargo is loaded during transport by the vehicle ([0042] Furthermore, as illustrated by block 314, the identification unit (216) may be used for identification of locations of newly loaded objects into the cargo space by generating new updated spatial models of the cargo space each time movement is detected).
Sangeneni does not teach identifying whether the cargo is present or not present in the images; after identifying whether the cargo is present in the images, determining a working set of the images comprising a limited number of the images in which the cargo is present in the image.
Meckesheimer, in the same field of endeavor of cargo image analysis, teaches identifying whether the cargo is present or not present in the images; after identifying whether the cargo is present in the images, determining a working set of the images comprising a limited number of the images in which the cargo is present in the image ([0030] parameter module 130 may be configured to filter the image data based on one or more of motion detection, object detection, image quality, etc. For example, in some embodiments, parameter module 130 may be configured to use frames without a predetermined level of movement, and/or without a specific object (e.g., forklift image, human image, or other objects in the image). [0026] “Cargo content” and “cargo items” refer to items located inside the container, items to be loaded (placed) inside the container, and/or items unloaded (removed) from the container).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Sangeneni with the teachings of Meckesheimer to select a limited number of images based on whether the cargo is present in the image "to process the image data using key frame extraction techniques to determine which frames to be used for evaluation…to filter the image data based on one or more of motion detection, object detection, image quality, etc." [Meckesheimer 0030].
Regarding claim 18, Sangeneni and Meckesheimer teach the method of claim 16. Sangeneni further teaches selecting the images from a primary camera that capture the cargo; selecting the images from the one or more subordinate cameras that were taken at the same time as the images that are selected from the primary camera ([0035] Movement may be analyzed by combining data from two types of cameras i.e. a depth camera (such as IR depth cameras) and an RGB camera. [0038] The combination of these three sensors relays to the system exactly where movement is happening. At the same time, depth cameras pointed at the shelves enable the system (104) to monitor movement in the shelf zones, and a computer vision DNN classifies objects within those regions either as parcels or non-parcels (eg, a hand));
and creating the working set from the selected images ([0048] Frame selection and deblurring as disclosed herein comprises analyzing multiple image frames for blur and using the frames with the least amount of blur for analysis).
Regarding claim 19, Sangeneni and Meckesheimer teach the method of claim 16. Sangeneni further teaches transmitting to a remote node the position of the cargo on the vehicle with the transmitting occurring while the cargo is loaded on the vehicle ([0042] transceivers (224) may be used to notify a user (e.g. a delivery person) of the position of a parcel by transmitting the location to e.g. devices (108). [0021] Further, the server (102) may be connected to devices such as mobile base stations, satellite communication networks, etc. to receive and forward any information received by the server (102)).
Regarding claim 20, Sangeneni and Meckesheimer teach the method of claim 16. Sangeneni further teaches determining the one or more aspects of the cargo based on a limited number of the images that are received ([0048] Frame selection and deblurring as disclosed herein comprises analyzing multiple image frames for blur and using the frames with the least amount of blur for analysis. [0006] volume estimation unit (214) configured to estimate a volume of loaded parcels in the updated spatial model).
Regarding claim 22, Sangeneni and Meckesheimer teach the device of claim 1. Sangeneni further teaches wherein the electro-optical sensors comprise cameras ([0006] images from a plurality of cameras including at least one camera for capturing depth and color, positioned in and around the cargo space).
Claims 5 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Sangeneni in view of Meckesheimer and Zhao (US20150189239A1).
Regarding claim 5, Sangeneni and Meckesheimer teach the device of claim 1. Sangeneni does not explicitly teach wherein the processing circuitry is configured to: use semantic segmentation to identify a first set of pixels in the images as the cargo and identify a different second set of pixels as another object through semantic segmentation; and determine a confidence value of an identification of the cargo based on an amount of the cargo that is visible in the images.
Zhao, in the same field of endeavor of cargo image analysis, teaches wherein the processing circuitry is configured to: use semantic segmentation to identify a first set of pixels in the images as the cargo and identify a different second set of pixels as another object through semantic segmentation ([0002] an automatic analysis and intelligent inspection method for bulk cargoes in containers, an automatic classification and recognition method for bulk cargoes in containers, and semantic segmentation and categorization of scanned images of bulk cargoes. [0013] the scanned image is pre-segmented to generate several small regions each being relatively consistent in terms of gray scale and texture; subsequently, features of the small regions are extracted, and the small regions are recognized by using a classifier generated by means of training according to the extracted features to obtain probabilities that the small regions pertain to various categories of cargoes);
and determine a confidence value of an identification of the cargo based on an amount of the cargo that is visible in the images ([0099] The regions obtained through segmentation are all super pixels. [0105] Each value is a confidence of a super pixel pertaining to a category).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the device of Sangeneni with the teachings of Zhao to use semantic segmentation to identify the cargo for "intelligent inspection methods such as analysis of illegally smuggled cargoes in containers, estimation of quantities of cargoes, tax amount computation" [Zhao 0002].
Regarding claim 6, Sangeneni and Meckesheimer teach the device of claim 1. Sangeneni further teaches wherein the processing circuitry is further configured to: identify the cargo within the images with a bounding box that encapsulates each instance of the cargo ([0041] As illustrated by block 312 shown in FIG. 3B, the area of the bounding box is mapped with a location on a shelf of the cargo space. It may be noted that the identification unit (216) may be used to identify the location of the bounding box and area thereof on a shelf of the cargo space of the vehicle).
Sangeneni does not explicitly teach determine a confidence value of an identification of cargo.
Zhao, in the same field of endeavor of cargo image analysis, teaches determine a confidence value of an identification of cargo ([0013] the small regions are recognized by using a classifier generated by means of training according to the extracted features to obtain probabilities that the small regions pertain to various categories of cargoe. [0105] Each value is a confidence of a super pixel pertaining to a category).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the device of Sangeneni with the teachings of Zhao to determine a confidence value of the identification of cargo “to obtain a probability of each small region pertaining to a certain category of cargoes, and merge small regions to obtain large regions each representing a category" [Abstract].
Claims 9 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Sangeneni in view of Meckesheimer and Sreeram (US20210201471A1).
Regarding claim 9, Sangeneni and Meckesheimer teach the device of claim 8. Sangeneni further teaches the primary camera and the one or more subordinate cameras ([0035] Movement may be analyzed by combining data from two types of cameras i.e. a depth camera (such as IR depth cameras) and an RGB camera. [0038] The combination of these three sensors relays to the system exactly where movement is happening. At the same time, depth cameras pointed at the shelves enable the system (104) to monitor movement in the shelf zones, and a computer vision DNN classifies objects within those regions either as parcels or non-parcels (eg, a hand));
Sangeneni does not explicitly teach discard the images that were not selected from the camera.
Sreeram, in the same field of endeavor of image object detection, teaches discard the images that were not selected from the camera ([0064] If, at block 816, a determination is made that the presence of a vacuum seal package is not detected, then the method 800 proceeds to block 818 where the image data is discarded (e.g., deleted)).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the device of Sangeneni with the teachings of Sreeram to discard images that were not selected because "processing the image data at block 814 to obtain the processed image data, as shown at block 820, prior to classifying a state of the vacuum seal package represented in the data increases the accuracy of the later-performed classification by the classification decision-making process 806" [Sreeram 0064].
Regarding claim 17, Sangeneni and Meckesheimer teach the method of claim 16. Sangeneni does not explicitly teach discarding the images that do not capture the cargo.
Sreeram, in the same field of endeavor of image object detection, teaches discarding the images that do not capture the cargo ([0064] If, at block 816, a determination is made that the presence of a vacuum seal package is not detected, then the method 800 proceeds to block 818 where the image data is discarded (e.g., deleted)).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Sangeneni with the teachings of Sreeram to discard images that do not capture the cargo because "processing the image data at block 814 to obtain the processed image data, as shown at block 820, prior to classifying a state of the vacuum seal package represented in the data increases the accuracy of the later-performed classification by the classification decision-making process 806" [Sreeram 0064].
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Sangeneni in view of Meckesheimer and Li (US20250071231A1).
Regarding claim 21, Sangeneni and Meckesheimer teach the device of claim 1. Sangeneni does not explicitly teach discarding the images based on a hierarchy of the electro-optical sensors.
Li, in the same field of endeavor of electro-optical sensor systems, teaches discarding the images based on a hierarchy of the electro-optical sensors ([0030] when one camera (for example, the master camera) performs capturing, the other camera (for example, the slave camera) does not perform capturing or the captured image is discarded).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the device of Sangeneni with the teachings of Li to discard images based on a hierarchy of the electro-optical sensors "in order to perform clock synchronization on the two cameras" [0030].
Claims 11 and 13-15 are rejected under 35 U.S.C. 103 as being unpatentable over Sangeneni in view of Afrouzi (US20220066456A1).
Regarding claim 11, Sangeneni teaches a computing device configured to monitor cargo that is being loaded onto a vehicle (Fig. 2 [0006] a system for continuous volume determination of the cargo space of a vehicle…generate an updated spatial model of the cargo space using the images from the plurality of cameras, upon detection of parcels being loaded or unloaded into the cargo space),
the computing device comprises: memory circuitry; processing circuitry configured to operate according to programming instructions stored in the memory circuitry ([0020] Generally, the processor receives (reads) instructions and content from a memory (such as a read-only memory and/or a random-access memory) and writes (stores) instructions and content to the memory) to:
receive images of the cargo from a primary electro-optical sensor and one or more subordinate electro-optical sensors ([0035] Movement may be analyzed by combining data from two types of cameras i.e. a depth camera (such as IR depth cameras) and an RGB camera);
analyze the images from the primary electro-optical sensor and from the one or more subordinate electro-optical sensors ([0038] The combination of these three sensors relays to the system exactly where movement is happening. At the same time, depth cameras pointed at the shelves enable the system (104) to monitor movement in the shelf zones, and a computer vision DNN classifies objects within those regions either as parcels or non-parcels (eg, a hand));
using the images from the primary electro-optical sensor and from the one or more subordinate electro-optical sensors, determine a point on the cargo ([0005] images from each camera of the plurality of cameras are stitched together into a Dense Point Cloud (DPC) in real-time);
select a first subset of images ([0048] Frame selection and deblurring as disclosed herein comprises analyzing multiple image frames for blur and using the frames with the least amount of blur for analysis) that comprises the images from the primary electro-optical sensor that capture the cargo and the images from the one or more subordinate electro-optical sensors that were captured at the same time and which are used to determine the point on the cargo ([0038] The combination of these three sensors relays to the system exactly where movement is happening. At the same time, depth cameras pointed at the shelves enable the system (104) to monitor movement in the shelf zones, and a computer vision DNN classifies objects within those regions either as parcels or non-parcels (eg, a hand). [0005] images from each camera of the plurality of cameras are stitched together into a Dense Point Cloud (DPC) in real-time);
based on just first subset of images, determine a shape of the cargo and dimensions of the cargo ([0073] Referring now to FIG. 5, a flowchart illustrating a method for identifying a position of an object in a cargo space is shown. [0074] Further as illustrated at block 504, a plurality of planes are extracted from the ROI wherein the plurality of planes correspond to the geometry of the object to be identified. [0077] Now at block 506, similar and nearby planes are clustered together using the clustering unit (220). The clustering is based on a weight of the two or more planes of the plurality of planes wherein the weight is assigned based on a property of orthogonality and dimensions of two or more planes of the plurality of planes);
based on just the first subset of images, determine a position on the vehicle where the cargo is loaded during transport by the vehicle ([0042] Furthermore, as illustrated by block 314, the identification unit (216) may be used for identification of locations of newly loaded objects into the cargo space by generating new updated spatial models of the cargo space each time movement is detected).
Sangeneni does not explicitly teach select a second subset of images that are not included in the first subset of images and that were used to identify the point on the cargo.
Afrouzi, in the same field of endeavor of cargo image analysis, teaches select a second subset of images that are not included in the first subset of images and that were used to identify the point on the cargo ([0558] Similarly, when a feature detector detects more than one usable point, it may prune the less desirable points and only use 1, 2, 3 or a subset of what the points tracked that are more distinguished or useful…some images from a set of images in an image stream may be pruned depending on factors such as quality, redundancy, and/or combination. For example, when the robot is standing still or moving slowly and all incoming images are substantially similar, the redundant images may be thrown away by the processor. [0802] a delivery robot including a smart pivoting belt system for moving packages);
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the device of Sangeneni with the teachings of Afrouzi to select a second subset of images that were used to identify the point on the cargo because “the redundant images may be thrown away by the processor" [0558].
Regarding claim 13, Sangeneni and Afrouzi teach the device of claim 11. Sangeneni does not explicitly teach wherein the processing circuitry is further configured to discard the images from the second subset of images.
Afrouzi, in the same field of endeavor of cargo image analysis, teaches wherein the processing circuitry is further configured to discard the images from the second subset of images ([0558] For example, when the robot is standing still or moving slowly and all incoming images are substantially similar, the redundant images may be thrown away by the processor).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the device of Sangeneni with the teachings of Afrouzi to discard the images from the second subset "depending on factors such as quality, redundancy, and/or combination" [0558].
Regarding claim 14, Sangeneni and Afrouzi teach the device of claim 11. Sangeneni further teaches wherein the processing circuitry is further configured to: receive the images from a primary camera and one or more subordinate cameras; include within the first subset of images the images from the primary camera that capture the cargo; include within the first subset of images the images from the one or more subordinate cameras that were taken at the same time as the images that are selected from the primary camera ([0035] Movement may be analyzed by combining data from two types of cameras i.e. a depth camera (such as IR depth cameras) and an RGB camera. [0038] The combination of these three sensors relays to the system exactly where movement is happening. At the same time, depth cameras pointed at the shelves enable the system (104) to monitor movement in the shelf zones, and a computer vision DNN classifies objects within those regions either as parcels or non-parcels (eg, a hand). [0048] Frame selection and deblurring as disclosed herein comprises analyzing multiple image frames for blur and using the frames with the least amount of blur for analysis;
and analyze and identify the cargo based on just the first subset of images ([0038] The combination of these three sensors relays to the system exactly where movement is happening. At the same time, depth cameras pointed at the shelves enable the system (104) to monitor movement in the shelf zones, and a computer vision DNN classifies objects within those regions either as parcels or non-parcels (eg, a hand)).
Regarding claim 15, Sangeneni teaches the device of claim 14. Sangeneni further teaches the primary camera and the one or more subordinate cameras ([0035] Movement may be analyzed by combining data from two types of cameras i.e. a depth camera (such as IR depth cameras) and an RGB camera).
Sangeneni does not explicitly teach wherein the processing circuitry is further configured to discard the second subset of images that comprise the images that were not selected from the cameras.
Afrouzi, in the same field of endeavor of cargo image analysis, teaches wherein the processing circuitry is further configured to discard the second subset of images that comprise the images that were not selected from the cameras ([0558] For example, when the robot is standing still or moving slowly and all incoming images are substantially similar, the redundant images may be thrown away by the processor).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the device of Sangeneni with the teachings of Afrouzi to discard the images from the second subset "depending on factors such as quality, redundancy, and/or combination" [0558].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jacqueline R Zak whose telephone number is (571)272-4077. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACQUELINE R ZAK/Examiner, Art Unit 2666
/SJ Park/Primary Examiner, Art Unit 2675
Prosecution Insights