DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-20 are pending for examination in the application filed 02/23/2026. Claims 1, 5-6, 8, 11, 13, and 16 have been amended.
Priority
Acknowledgement is made of Applicant’s claim to priority of provisional application 63/439,701, filing date 01/18/2023.
Response to Arguments and Amendments
The objection of claim 11 is withdrawn in view of the amendments. Applicant’s arguments with respect to claim 1 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument, as facilitated by the newly added amendments.
Applicant argues that the arguments stated regarding claim 1 apply to amended independent claims 11 and 16. The amendments to claims 11 and 16 are not identical to the amendments to claim 1 and therefore the arguments are not applicable. For further clarification, amended claims 11 and 16 do not specify selecting a subset of the images that includes a limited number of the images based on a predetermined factor of where the cargo is located within the image as in amended claim 1. Additionally, applicant argues that Sangeneni does not disclose filtering the images based on primary and subordinate images, however the claim language of claim 11 states: select a subset of images that comprises images from the primary electro-optical sensor that capture the cargo and the images from the one or more subordinate electro-optical sensors that were captured at the same time. The subset of images which Sangeneni performs processing on comprises images from the primary electro-optical sensor that capture the cargo and the images from the one or more subordinate electro-optical sensors that were captured at the same time ([0048] Frame selection and deblurring as disclosed herein comprises analyzing multiple image frames for blur and using the frames with the least amount of blur for analysis. [0033] The system (104), using the cameras (208), continually samples the cargo space to detect any changes in pixel values between the initial spatial model and the new frames. When changes are detected, an updated spatial model of the cargo space is created as discussed in block 306 FIG. 3A. In one embodiment the system (104) compares the initial spatial model to this updated spatial model between both color and depth image pairs. [0037] In an aspect, the location of this movement is measured for absolute depth values by the depth camera so that the depth camera does not need to calculate depth values within its entire field of view. The direction of movement is then determined by comparing the sequential change in pixels on a linear scale extending away from the camera in both directions. If movement is detected outside the van in one location, followed in sequence by movement detected closer to the camera/origin of the scale, the system knows the object in question is moving toward the camera/into the cargo space. [0038] The combination of these three sensors relays to the system exactly where movement is happening. At the same time, depth cameras pointed at the shelves enable the system (104) to monitor movement in the shelf zones, and a computer vision DNN classifies objects within those regions either as parcels or non-parcels (eg, a hand)). Please see the updated 35 USC 102 and 35 USC 103 rejections including these amendments.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 6 and 16 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 6 and 16 as amended describe “determine a confidence value of an identification of the cargo based on a percentage of the cargo that is visible in the images” (claim 6) and “determining a confidence value of the cargo based on an amount of the cargo that is visible in the working images relative to the shape and size of the cargo” (claim 16).
The specification describes in [0060] “The cargo detection process analyzes the images and identifies cargo 200 in the image. Based on the results of the analysis, the algorithm assigns a confidence value to the identified cargo in the image. The confidence value is how likely the identified aspect of the image is actually cargo. In one example, the confidence value ranges from 1.00 (highest confidence value) and reduces downward. Figure 6 illustrates an image 71 that has been analyzed by the computing device 20. The analysis identifies the various separate cargo 200 that are visible in the image 71. A bounding box 73 is formed that encapsulates the identified cargo 200. A confidence value 72 is determined for each cargo 200. In this example, the confidence value 72 is higher for the cargo 200 that is more fully visible in the image 71. Cargo 200 that is more hidden in the image 71 have lower confidence values 72. In one example, the confidence values are determined using standard object detection using a neural network”.
Paragraph [0060] describes that the confidence value is higher for cargo that is more fully visible compared to cargo that is more hidden, but it does not describe a percentage of cargo, as claimed in amended claim 6. Paragraph [0060] also does not describe that the confidence value is based on an amount of the cargo that is visible in the working images relative to the shape and size of the cargo.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 11 and 14 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Sangeneni (US20210319582A1).
Regarding claim 11, Sangeneni teaches a computing device configured to monitor cargo that is being loaded onto a vehicle, the computing device comprises: memory circuitry; processing circuitry configured to operate according to programing instructions stored in the memory circuitry ([0006] In another embodiment, the present disclosure describes a system for continuous volume determination of the cargo space of a vehicle…Further, the model generation unit (212) is also configured to generate an updated spatial model of the cargo space using the images from the plurality of cameras, upon detection of parcels being loaded or unloaded into the cargo space. [0020] Generally, the processor receives (reads) instructions and content from a memory (such as a read-only memory and/or a random-access memory) and writes (stores) instructions and content to the memory. Storage devices suitable for tangibly embodying computer program instructions and content include, for example, all forms of non-volatile memory) to:
receive images of the cargo from a primary electro-optical sensor and one or more subordinate electro-optical sensors; analyze the images from the primary electro-optical sensor and from the one or more subordinate electro-optical sensors; ([0033] The system (104), using the cameras (208), continually samples the cargo space to detect any changes in pixel values between the initial spatial model and the new frames. When changes are detected, an updated spatial model of the cargo space is created as discussed in block 306 FIG. 3A. In one embodiment the system (104) compares the initial spatial model to this updated spatial model between both color and depth image pairs. [0037] In an aspect, the location of this movement is measured for absolute depth values by the depth camera so that the depth camera does not need to calculate depth values within its entire field of view. The direction of movement is then determined by comparing the sequential change in pixels on a linear scale extending away from the camera in both directions. If movement is detected outside the van in one location, followed in sequence by movement detected closer to the camera/origin of the scale, the system knows the object in question is moving toward the camera/into the cargo space. [0038] The combination of these three sensors relays to the system exactly where movement is happening. At the same time, depth cameras pointed at the shelves enable the system (104) to monitor movement in the shelf zones, and a computer vision DNN classifies objects within those regions either as parcels or non-parcels (eg, a hand));
select a subset of images that comprises the images from the primary electro-optical sensor that capture the cargo and the images from the one or more subordinate electro-optical sensors that were captured at the same time ([0048] Frame selection and deblurring as disclosed herein comprises analyzing multiple image frames for blur and using the frames with the least amount of blur for analysis. [0037] If movement is detected outside the van in one location, followed in sequence by movement detected closer to the camera/origin of the scale, the system knows the object in question is moving toward the camera/into the cargo space. [0038] The combination of these three sensors relays to the system exactly where movement is happening. At the same time, depth cameras pointed at the shelves enable the system (104) to monitor movement in the shelf zones, and a computer vision DNN classifies objects within those regions either as parcels or non-parcels (eg, a hand));
determine a point on the cargo based on the subset of images ([0005] In an aspect, images from each camera of the plurality of cameras are stitched together into a Dense Point Cloud (DPC) in real-time. The disclosed method further comprises generating an updated spatial model of the cargo space using the images from the plurality of cameras, upon detection of parcels or objects being loaded or unloaded into the cargo space, wherein the updated spatial model includes changes to the cargo space and estimating a volume of loaded parcels in the updated spatial model);
and based on the subset of images determine a position on the vehicle where the cargo is loaded during transport by the vehicle ([0042] Furthermore, as illustrated by block 314, the identification unit (216) may be used for identification of locations of newly loaded objects into the cargo space by generating new updated spatial models of the cargo space each time movement is detected).
Regarding claim 14, Sangeneni teaches the device of claim 11. Sangeneni further teaches wherein the processing circuitry is further configured to: receive the images from a primary camera and one or more subordinate cameras; select the images from the primary camera that capture the cargo; select the images from the one or more subordinate cameras that were taken at the same time as the images that are selected from the primary camera ([0033] The system (104), using the cameras (208), continually samples the cargo space to detect any changes in pixel values between the initial spatial model and the new frames. When changes are detected, an updated spatial model of the cargo space is created as discussed in block 306 FIG. 3A. In one embodiment the system (104) compares the initial spatial model to this updated spatial model between both color and depth image pairs. [0037] In an aspect, the location of this movement is measured for absolute depth values by the depth camera so that the depth camera does not need to calculate depth values within its entire field of view. The direction of movement is then determined by comparing the sequential change in pixels on a linear scale extending away from the camera in both directions. If movement is detected outside the van in one location, followed in sequence by movement detected closer to the camera/origin of the scale, the system knows the object in question is moving toward the camera/into the cargo space. [0038] The combination of these three sensors relays to the system exactly where movement is happening. At the same time, depth cameras pointed at the shelves enable the system (104) to monitor movement in the shelf zones, and a computer vision DNN classifies objects within those regions either as parcels or non-parcels (eg, a hand)).
and analyze and identify the cargo based on the selected images ([0038] The combination of these three sensors relays to the system exactly where movement is happening. At the same time, depth cameras pointed at the shelves enable the system (104) to monitor movement in the shelf zones, and a computer vision DNN classifies objects within those regions either as parcels or non-parcels (eg, a hand)).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-4, 7-8, 10, and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Sangeneni in view of Meckesheimer (US20220366556A1).
Regarding claim 1, Sangeneni teaches a computing device configured to monitor cargo that is being loaded onto a vehicle ([0006] In another embodiment, the present disclosure describes a system for continuous volume determination of the cargo space of a vehicle…Further, the model generation unit (212) is also configured to generate an updated spatial model of the cargo space using the images from the plurality of cameras, upon detection of parcels being loaded or unloaded into the cargo space),
the computing device comprising: memory circuitry; processing circuitry configured to operate according to programming instructions stored in the memory circuitry ([0020] Generally, the processor receives (reads) instructions and content from a memory (such as a read-only memory and/or a random-access memory) and writes (stores) instructions and content to the memory. Storage devices suitable for tangibly embodying computer program instructions and content include, for example, all forms of non-volatile memory) to:
receive images of the cargo from electro-optical sensors affixed to the vehicle ([0006] The disclosed system comprises a model generation unit configured to generate an initial spatial model of the cargo space using images from a plurality of cameras including at least one camera for capturing depth and color, positioned in and around the cargo space);
selecting a subset of the images([0048] Frame selection and deblurring as disclosed herein comprises analyzing multiple image frames for blur and using the frames with the least amount of blur for analysis);
identify the cargo based on the subset of the images without using a remainder of the images that were not included in the subset ([0055] The process of label extraction begins after an image frame captured by cameras (208) is preprocessed (i.e. the frame is deblurred or goes through a similar procedure). In the label extraction process the edges of all objects, labels, and other features may be detected by using an edge detection algorithm);
based on just the images of the subset, compute one or more aspects of the cargo including a volume of the cargo ([0006] The system as disclosed also comprises a volume estimation unit (214) configured to estimate a volume of loaded parcels in the updated spatial model);
determine a position on the vehicle where the cargo is loaded based on the subset of images of the cargo ([0042] Furthermore, as illustrated by block 314, the identification unit (216) may be used for identification of locations of newly loaded objects into the cargo space by generating new updated spatial models of the cargo space each time movement is detected);
and based on the loaded position on the vehicle, associate one or more aspects with the cargo including the volume of the cargo ([0040] Next, as illustrated in FIG. 3B block 310, a volume estimation unit (214) of the system (104) may map an area of the bounding box within the total volume of the cargo space, where an estimate of the remaining cargo volume may be determined. [0006] Further, the volume estimation unit (214) is configured to determine a remaining volume of the cargo space based on the said estimated volume of the loaded parcels and a total volume of the cargo space, wherein the total volume is calculated based on the initial spatial model).
Sangeneni does not teach selecting a subset of the images that includes a limited number of the images based on a predetermined factor of where the cargo is located within the image.
Meckesheimer, in the same field of endeavor of cargo image analysis, teaches selecting a subset of the images that includes a limited number of the images based on a predetermined factor of where the cargo is located within the image ([0030] For example, in some embodiments, parameter module 130 may be configured to process the image data using key frame extraction techniques to determine which frames to be used for evaluation. For example, in some embodiments, parameter module 130 may be configured to filter the image data based on one or more of motion detection, object detection, image quality, etc. For example, in some embodiments, parameter module 130 may be configured to use frames without a predetermined level of movement, and/or without a specific object (e.g., forklift image, human image, or other objects in the image). In some embodiments, parameter module 130 may be configured to use monocular depth measurement techniques (e.g., perspective cues) to locate proper guidelines and/or calculate depth. For example, visual gradient may be used to locate guidelines, product edges, trailer head, and/or other information).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the device of Sangeneni with the teachings of Meckesheimer to select a subset of images based on where the cargo is located within the image because "image data may be combined, filtered, and processed to identify an object (e.g., container, container components, and/or cargo items). Once identified, the object may be measured, and its position and orientation extracted." [Meckesheimer 0029].
Regarding claim 2, Sangeneni and Meckesheimer teach the device of claim 1. Sangeneni further teaches wherein the processing circuitry is further configured to identify the cargo based on at least one of the images ([0039] As illustrated at block 308 of FIG. 3A an identification unit (216) illustrated in FIG. 2 may be configured to determine the location of any object loaded into the cargo space based on modeling a bounding box around the area in which pixel difference is not equal to zero).
Regarding claim 3, Sangeneni and Meckesheimer teach the device of claim 1. Sangeneni further teaches wherein the processing circuitry is configured to: determine a temporary position of the cargo within an alignment area within the vehicle ([0078] At blocks 508 and 510, the method goes on to model a multi-dimensional bounding box that could be 3D corresponding to the clustered planes and identifying the position of the object in the cargo space based on the position of the multi-dimensional bounding box within the DPC or global coordinate system);
determine a lane within the vehicle that the cargo enters after being positioned at the alignment area ([0073] The region of interest may be a location with a high probability of locating an object. For example, in a cargo space where the objects are loaded on shelves, the area of the shelves may be determined as the ROI whereas the full cargo space will be in the field of view. The ROI may be identified using the technique of subtracting space as described in the previous paragraphs with regards to detection of movement and subtraction of pixels from the model);
and determine a location along the lane where the cargo is finally positioned ([0080] Further on repetition of the method FIG. 5, it may be determined that the location of an object has changed from the position identified during the previous iteration of the method. For example, the system (104) will be able to identify what movement occurred between frames by identifying a similar bounding box at a different position from the previous iteration of the method and thereby determine a change in position).
Regarding claim 4, Sangeneni and Meckesheimer teach the device of claim 1. Sangeneni further teaches communication circuitry configured to upload through a communication network the position of the cargo on the vehicle while onboard the vehicle ([0042] transceivers (224) may be used to notify a user (e.g. a delivery person) of the position of a parcel by transmitting the location to e.g. devices (108)).
Regarding claim 7, Sangeneni and Meckesheimer teach the device of claim 1. Sangeneni further teaches wherein the processing circuitry is configured to determine a point on the cargo based on the images and track the position of the cargo within the vehicle based on the point ([0005] In an aspect, images from each camera of the plurality of cameras are stitched together into a Dense Point Cloud (DPC) in real-time. The disclosed method further comprises generating an updated spatial model of the cargo space using the images from the plurality of cameras, upon detection of parcels or objects being loaded or unloaded into the cargo space, wherein the updated spatial model includes changes to the cargo space and estimating a volume of loaded parcels in the updated spatial model).
Regarding claim 8, Sangeneni and Meckesheimer teach the device of claim 1. Sangeneni further teaches wherein the processing circuitry is further configured to: receive the images from a primary camera and one or more subordinate cameras; include in the subset of images the images from the primary camera that capture the cargo; include in the subset of images the images from the one or more subordinate cameras that were taken at the same time as the images that are selected from the primary camera ([0033] The system (104), using the cameras (208), continually samples the cargo space to detect any changes in pixel values between the initial spatial model and the new frames. When changes are detected, an updated spatial model of the cargo space is created as discussed in block 306 FIG. 3A. In one embodiment the system (104) compares the initial spatial model to this updated spatial model between both color and depth image pairs. [0037] In an aspect, the location of this movement is measured for absolute depth values by the depth camera so that the depth camera does not need to calculate depth values within its entire field of view. The direction of movement is then determined by comparing the sequential change in pixels on a linear scale extending away from the camera in both directions. If movement is detected outside the van in one location, followed in sequence by movement detected closer to the camera/origin of the scale, the system knows the object in question is moving toward the camera/into the cargo space. [0038] The combination of these three sensors relays to the system exactly where movement is happening. At the same time, depth cameras pointed at the shelves enable the system (104) to monitor movement in the shelf zones, and a computer vision DNN classifies objects within those regions either as parcels or non-parcels (eg, a hand)).
and identifying the cargo based on the subset of images ([0038] The combination of these three sensors relays to the system exactly where movement is happening. At the same time, depth cameras pointed at the shelves enable the system (104) to monitor movement in the shelf zones, and a computer vision DNN classifies objects within those regions either as parcels or non-parcels (eg, a hand)).
Regarding claim 10, Sangeneni and Meckesheimer teach the device of claim 1. Sangeneni further teaches wherein the processing circuitry is further configured to calculate one of a 3D mesh or a 3D point cloud of the cargo based on the images and determine the volume of the cargo using the 3D mesh or the 3D point cloud ([0029] The model generation unit (212) is configured to stitch the data from the camera (208) in the form of a DPC or a mesh in real time. [0067] Referring now to FIG. 4, a flowchart illustrating a method of determining volume of a cargo space of a vehicle in real-time is shown. The method starts at block 402 where an initial spatial model of the cargo space is generated using images from a plurality of cameras (208). [0070] Next, at block 406, volume of the loaded parcels in the updated spatial model are estimated using the volume estimation unit (214)).
Regarding claim 12, Sangeneni teaches the device of claim 11. Meckesheimer, in the same field of endeavor of cargo image analysis, teaches determine a shape of the cargo, and dimensions of the cargo ([0041] Loading pattern determination module 140 is configured to determine one or more loading characteristics (or loading pattern) of the container based on information from parameter module 130 (e.g., container parameters and/or cargo items parameters). For example, in some embodiments, the loading characteristics may be determined based on the container information (e.g., size, dimensions, shape, etc.)).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the device of Sangeneni with the teachings of Meckesheimer to determine shape and dimensions of the cargo so that "loading pattern determination module 140 may be configured to determine if pallets are touching doors/walls; if there is space between pallets at center; if pallet top exceeds a predetermined height; if pallet top touches (compresses) air chute; if load locks are present; if a height mark is present; if environmental sensors present inside the container; and/or other loading patterns" [Meckesheimer 0041].
Claims 16 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Sangeneni in view of Meckesheimer and Wozniak (US11922368B1).
Regarding claim 16, Sangeneni teaches a method of monitoring cargo that is being loaded onto a vehicle ([0005] a method of determining the volume of a cargo space of a vehicle in real-time…The disclosed method further comprises generating an updated spatial model of the cargo space using the images from the plurality of cameras, upon detection of parcels or objects being loaded or unloaded into the cargo space), the method comprising:
receiving images of the cargo from a plurality of electro-optical sensors ([0006] The disclosed system comprises a model generation unit configured to generate an initial spatial model of the cargo space using images from a plurality of cameras including at least one camera for capturing depth and color, positioned in and around the cargo space);
determining a working set of the images ([0033] The system (104), using the cameras (208), continually samples the cargo space to detect any changes in pixel values between the initial spatial model and the new frames. When changes are detected, an updated spatial model of the cargo space is created as discussed in block 306 FIG. 3A. In one embodiment the system (104) compares the initial spatial model to this updated spatial model between both color and depth image pairs. This comparison is done by subtracting the pixel values of the initial spatial model from the updated spatial model and eliminating any pixels that are the same between both models (i.e. returned a difference of 0));
identifying the cargo within the working set of the images ([0039] As illustrated at block 308 of FIG. 3A an identification unit (216) illustrated in FIG. 2 may be configured to determine the location of any object loaded into the cargo space based on modeling a bounding box around the area in which pixel difference is not equal to zero);
determining a point on the cargo based on the working set of the images ([0005] In an aspect, images from each camera of the plurality of cameras are stitched together into a Dense Point Cloud (DPC) in real-time. The disclosed method further comprises generating an updated spatial model of the cargo space using the images from the plurality of cameras, upon detection of parcels or objects being loaded or unloaded into the cargo space, wherein the updated spatial model includes changes to the cargo space and estimating a volume of loaded parcels in the updated spatial model);
determining a volume of the cargo based on the working set of the images ([0006] The system as disclosed also comprises a volume estimation unit (214) configured to estimate a volume of loaded parcels in the updated spatial model);
and determining a position on the vehicle where the cargo is loaded during transport by the vehicle ([0042] Furthermore, as illustrated by block 314, the identification unit (216) may be used for identification of locations of newly loaded objects into the cargo space by generating new updated spatial models of the cargo space each time movement is detected).
Sangeneni does not teach a working set of the images comprising a limited number of the images; determining a shape and size of the cargo.
Meckesheimer, in the same field of endeavor of cargo image analysis, teaches a working set of the images comprising a limited number of the images ([0029] For example, image data may be combined, filtered, and processed to identify an object (e.g., container, container components, and/or cargo items). Once identified, the object may be measured, and its position and orientation extracted);
determining a shape and size of the cargo ([0041] Loading pattern determination module 140 is configured to determine one or more loading characteristics (or loading pattern) of the container based on information from parameter module 130 (e.g., container parameters and/or cargo items parameters). For example, in some embodiments, the loading characteristics may be determined based on the container information (e.g., size, dimensions, shape, etc.)).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the device of Sangeneni with the teachings of Meckesheimer to select a limited number of images "to process the image data using key frame extraction techniques to determine which frames to be used for evaluation…to filter the image data based on one or more of motion detection, object detection, image quality, etc." [Meckesheimer 0030] and to determine shape and size of the cargo so that "loading pattern determination module 140 may be configured to determine if pallets are touching doors/walls; if there is space between pallets at center; if pallet top exceeds a predetermined height; if pallet top touches (compresses) air chute; if load locks are present; if a height mark is present; if environmental sensors present inside the container; and/or other loading patterns" [Meckesheimer 0041].
Sangeneni does not teach determining a confidence value of the cargo based on an amount of the cargo that is visible in the working images relative to the shape and size of the cargo.
Wozniak, in the same field of endeavor of cargo classification, teaches determining a confidence value of the cargo based on an amount of the cargo that is visible in the working images relative to the shape and size of the cargo ([col. 5 ln. 23-59] For example, the central station may receive data associated with a physical object from one or more sources (e.g., robotic systems, distribution systems, handheld computing devices, an inventory database, etc.). The central station may receive similar data for a number of physical objects (e.g., from one or more robotic systems), in which those physical objects have been incapable to be classified with high confidence in some aspect. The central station may then utilize an unsupervised learning methodology to determine a class (e.g., a cluster) of physical objects that are known to be problematic, and one or more common attributes of that class...For example, as throughput levels are increased for a given robotic system (e.g., since it may successfully classify and process a higher percentage of items it detects), less machine computing resources may be required to process a given set of physical objects over time. [col. 12 ln. 59-63] The central computer system 130 may have received data associated with each of these physical objects (e.g., including attribute data, images of the objects, an indication that the each physical object was incapable of being classified and/or operated upon in a particular way, etc. [col. 2 ln. 65-67] Based on the images captured, the first robotic system may determine one or more attributes (e.g., features) of the object (e.g., dimensions of the physical object within the image, color, reflectivity, shape, etc.).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Sangeneni with the teachings of Wozniak to determine a confidence of cargo identification relative to the shape and size of the cargo because "techniques described herein enable a system (e.g., including a central station, distribution system, and/or robotic system) to more efficiently adapt to different (e.g., new) types of physical objects and/or otherwise new scenarios in which an existing classifier may not be able to classify physical objects with acceptable confidence" [Wozniak col. 5 ln. 13-23].
Regarding claim 18, Sangeneni, Meckesheimer, and Wozniak teach the method of claim 16. Sangeneni further teaches selecting the images from a primary camera that capture the cargo; selecting the images from the one or more subordinate cameras that were taken at the same time as the images that are selected from the primary camera; and creating the working set from the selected images ([0033] The system (104), using the cameras (208), continually samples the cargo space to detect any changes in pixel values between the initial spatial model and the new frames. When changes are detected, an updated spatial model of the cargo space is created as discussed in block 306 FIG. 3A. In one embodiment the system (104) compares the initial spatial model to this updated spatial model between both color and depth image pairs. [0037] In an aspect, the location of this movement is measured for absolute depth values by the depth camera so that the depth camera does not need to calculate depth values within its entire field of view. The direction of movement is then determined by comparing the sequential change in pixels on a linear scale extending away from the camera in both directions. If movement is detected outside the van in one location, followed in sequence by movement detected closer to the camera/origin of the scale, the system knows the object in question is moving toward the camera/into the cargo space. [0038] The combination of these three sensors relays to the system exactly where movement is happening. At the same time, depth cameras pointed at the shelves enable the system (104) to monitor movement in the shelf zones, and a computer vision DNN classifies objects within those regions either as parcels or non-parcels (eg, a hand)).
Regarding claim 19, Sangeneni, Meckesheimer, and Wozniak teach the method of claim 16. Sangeneni further teaches transmitting to a remote node the position of the cargo on the vehicle with the transmitting occurring while the cargo is loaded on the vehicle ([0042] transceivers (224) may be used to notify a user (e.g. a delivery person) of the position of a parcel by transmitting the location to e.g. devices (108). [0021] Further, the server (102) may be connected to devices such as mobile base stations, satellite communication networks, etc. to receive and forward any information received by the server (102)).
Regarding claim 20, Sangeneni, Meckesheimer, and Wozniak teach the method of claim 16. Meckesheimer teaches determining the one or more aspects of the cargo based on a limited number of the images that are received ([0029] For example, image data may be combined, filtered, and processed to identify an object (e.g., container, container components, and/or cargo items). Once identified, the object may be measured, and its position and orientation extracted).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Sangeneni with the teachings of Meckesheimer to select a limited number of images received and determine aspects of the cargo "to process the image data using key frame extraction techniques to determine which frames to be used for evaluation…to filter the image data based on one or more of motion detection, object detection, image quality, etc." [Meckesheimer 0030] and "to determine a condition of one or more components of the container based on the image data of the container" [Meckesheimer 0006].
Claims 6 is rejected under 35 U.S.C. 103 as being unpatentable over Sangeneni in view of Meckesheimer and Wozniak.
Regarding claim 6, Sangeneni and Meckesheimer teach the device of claim 1. Sangeneni further teaches wherein the processing circuitry is further configured to: identify the cargo within the images with a bounding box that encapsulates each instance of the cargo ([0041] As illustrated by block 312 shown in FIG. 3B, the area of the bounding box is mapped with a location on a shelf of the cargo space. It may be noted that the identification unit (216) may be used to identify the location of the bounding box and area thereof on a shelf of the cargo space of the vehicle).
Sangeneni does not teach determine a confidence value of an identification of the cargo based on a percentage of the cargo that is visible in the images.
Wozniak, in the same field of endeavor of cargo classification, teaches determine a confidence value of an identification of the cargo based on a percentage of the cargo that is visible in the images ([col. 5 ln. 23-59] For example, the central station may receive data associated with a physical object from one or more sources (e.g., robotic systems, distribution systems, handheld computing devices, an inventory database, etc.). The central station may receive similar data for a number of physical objects (e.g., from one or more robotic systems), in which those physical objects have been incapable to be classified with high confidence in some aspect. The central station may then utilize an unsupervised learning methodology to determine a class (e.g., a cluster) of physical objects that are known to be problematic, and one or more common attributes of that class...For example, as throughput levels are increased for a given robotic system (e.g., since it may successfully classify and process a higher percentage of items it detects), less machine computing resources may be required to process a given set of physical objects over time. [col. 12 ln. 59-63] The central computer system 130 may have received data associated with each of these physical objects (e.g., including attribute data, images of the objects, an indication that the each physical object was incapable of being classified and/or operated upon in a particular way, etc.).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the device of Sangeneni with the teachings of Wozniak to determine a confidence of cargo identification based on the percentage of cargo visible because "techniques described herein enable a system (e.g., including a central station, distribution system, and/or robotic system) to more efficiently adapt to different (e.g., new) types of physical objects and/or otherwise new scenarios in which an existing classifier may not be able to classify physical objects with acceptable confidence" [Wozniak col. 5 ln. 13-23].
Claims 5 is rejected under 35 U.S.C. 103 as being unpatentable over Sangeneni in view of Meckesheimer and Zhao (US20150189239A1).
Regarding claim 5, Sangeneni and Meckesheimer teach the device of claim 1. Zhao, in the same field of endeavor of cargo image analysis, teaches wherein the processing circuitry is configured to: use semantic segmentation to identify a first set of pixels in the images as the cargo and identify a different second set of pixels as another object through semantic segmentation ([0002] an automatic analysis and intelligent inspection method for bulk cargoes in containers, an automatic classification and recognition method for bulk cargoes in containers, and semantic segmentation and categorization of scanned images of bulk cargoes. [0013] the scanned image is pre-segmented to generate several small regions each being relatively consistent in terms of gray scale and texture; subsequently, features of the small regions are extracted, and the small regions are recognized by using a classifier generated by means of training according to the extracted features to obtain probabilities that the small regions pertain to various categories of cargoes);
and determine a confidence value of an identification of the cargo based on an amount of the cargo that is visible in the images ([0099] The regions obtained through segmentation are all super pixels. [0105] Each value is a confidence of a super pixel pertaining to a category).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the device of Sangeneni with the teachings of Zhao to use semantic segmentation to identify the cargo for "intelligent inspection methods such as analysis of illegally smuggled cargoes in containers, estimation of quantities of cargoes, tax amount computation" [Zhao 0002].
Claims 13 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Sangeneni in view of Sreeram (US20210201471A1).
Regarding claim 13, Sangeneni teaches the device of claim 11. Sreeram, in the same field of endeavor of image object detection, teaches discard the images that are not included in the subset of images ([0064] At block 816, a determination is made whether the presence of a vacuum seal package is detected in the image data. In some embodiments, the determination made at block 816 is a part of the processing of the image data at block 816. In some embodiments, the determination of whether vacuum seal package is detected at block 816 is a separate process from the processing of the image data at block 816. If, at block 816, a determination is made that the presence of a vacuum seal package is not detected, then the method 800 proceeds to block 818 where the image data is discarded (e.g., deleted)).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the device of Sangeneni with the teachings of Sreeram to discard images that do not capture the cargo because "processing the image data at block 814 to obtain the processed image data, as shown at block 820, prior to classifying a state of the vacuum seal package represented in the data increases the accuracy of the later-performed classification by the classification decision-making process 806" [Sreeram 0064].
Regarding claim 15, Sangeneni teaches the device of claim 14. Sangeneni further teaches the primary camera and the one or more subordinate cameras ([0033] The system (104), using the cameras (208), continually samples the cargo space to detect any changes in pixel values between the initial spatial model and the new frames. When changes are detected, an updated spatial model of the cargo space is created as discussed in block 306 FIG. 3A. In one embodiment the system (104) compares the initial spatial model to this updated spatial model between both color and depth image pairs. [0037] In an aspect, the location of this movement is measured for absolute depth values by the depth camera so that the depth camera does not need to calculate depth values within its entire field of view. The direction of movement is then determined by comparing the sequential change in pixels on a linear scale extending away from the camera in both directions. If movement is detected outside the van in one location, followed in sequence by movement detected closer to the camera/origin of the scale, the system knows the object in question is moving toward the camera/into the cargo space. [0038] The combination of these three sensors relays to the system exactly where movement is happening. At the same time, depth cameras pointed at the shelves enable the system (104) to monitor movement in the shelf zones, and a computer vision DNN classifies objects within those regions either as parcels or non-parcels (eg, a hand)).
Sangeneni does not teach discard the images that were not selected from the camera.
Sreeram, in the same field of endeavor of image object detection, teaches discard the images that were not selected from the camera ([0064] At block 816, a determination is made whether the presence of a vacuum seal package is detected in the image data. In some embodiments, the determination made at block 816 is a part of the processing of the image data at block 816. In some embodiments, the determination of whether vacuum seal package is detected at block 816 is a separate process from the processing of the image data at block 816. If, at block 816, a determination is made that the presence of a vacuum seal package is not detected, then the method 800 proceeds to block 818 where the image data is discarded (e.g., deleted)).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the device of Sangeneni with the teachings of Sreeram to discard images that were not selected because "processing the image data at block 814 to obtain the processed image data, as shown at block 820, prior to classifying a state of the vacuum seal package represented in the data increases the accuracy of the later-performed classification by the classification decision-making process 806" [Sreeram 0064].
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Sangeneni in view of Meckesheimer and Sreeram.
Regarding claim 9, Sangeneni and Meckesheimer teach the device of claim 8. Sangeneni further teaches the primary camera and the one or more subordinate cameras ([0033] The system (104), using the cameras (208), continually samples the cargo space to detect any changes in pixel values between the initial spatial model and the new frames. When changes are detected, an updated spatial model of the cargo space is created as discussed in block 306 FIG. 3A. In one embodiment the system (104) compares the initial spatial model to this updated spatial model between both color and depth image pairs. [0037] In an aspect, the location of this movement is measured for absolute depth values by the depth camera so that the depth camera does not need to calculate depth values within its entire field of view. The direction of movement is then determined by comparing the sequential change in pixels on a linear scale extending away from the camera in both directions. If movement is detected outside the van in one location, followed in sequence by movement detected closer to the camera/origin of the scale, the system knows the object in question is moving toward the camera/into the cargo space. [0038] The combination of these three sensors relays to the system exactly where movement is happening. At the same time, depth cameras pointed at the shelves enable the system (104) to monitor movement in the shelf zones, and a computer vision DNN classifies objects within those regions either as parcels or non-parcels (eg, a hand)).
Sangeneni does not teach discard the images that were not selected from the camera.
Sreeram, in the same field of endeavor of image object detection, teaches discard the images that were not selected from the camera ([0064] At block 816, a determination is made whether the presence of a vacuum seal package is detected in the image data. In some embodiments, the determination made at block 816 is a part of the processing of the image data at block 816. In some embodiments, the determination of whether vacuum seal package is detected at block 816 is a separate process from the processing of the image data at block 816. If, at block 816, a determination is made that the presence of a vacuum seal package is not detected, then the method 800 proceeds to block 818 where the image data is discarded (e.g., deleted)).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the device of Sangeneni with the teachings of Sreeram to discard images that were not selected because "processing the image data at block 814 to obtain the processed image data, as shown at block 820, prior to classifying a state of the vacuum seal package represented in the data increases the accuracy of the later-performed classification by the classification decision-making process 806" [Sreeram 0064].
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Sangeneni in view of Meckesheimer, Wozniak and Sreeram.
Regarding claim 17, Sangeneni, Meckesheimer, and Wozniak teach the method of claim 16. Sreeram, in the same field of endeavor of image object detection, teaches discarding the images that do not capture the cargo ([0064] At block 816, a determination is made whether the presence of a vacuum seal package is detected in the image data. In some embodiments, the determination made at block 816 is a part of the processing of the image data at block 816. In some embodiments, the determination of whether vacuum seal package is detected at block 816 is a separate process from the processing of the image data at block 816. If, at block 816, a determination is made that the presence of a vacuum seal package is not detected, then the method 800 proceeds to block 818 where the image data is discarded (e.g., deleted)).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Sangeneni with the teachings of Sreeram to discard images that do not capture the cargo because "processing the image data at block 814 to obtain the processed image data, as shown at block 820, prior to classifying a state of the vacuum seal package represented in the data increases the accuracy of the later-performed classification by the classification decision-making process 806" [Sreeram 0064].
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jacqueline R Zak whose telephone number is (571)272-4077. The examiner can normally be reached M-F 9-5. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACQUELINE R ZAK/Examiner, Art Unit 2666
/EMILY C TERRELL/Supervisory Patent Examiner, Art Unit 2666