DETAILED ACTION
Response to Amendment
This is a Final Office Action on the merits in response to communications on 2026/02/03. Claims 1, 7, 9, 15, 17, and 20 are amended. Claims 1 – 20 are pending and are addressed below.
Response to Arguments
Applicant has changed the scope of the claim language. Amendments made to independent claims 1, 9 and 17 are still read on by Boyraz, however new prior art Kumar, et. al. (US 20210148709 A1) read on the amended claims 7, 15, and 20. The amendments are further addressed in the body of the Non-Final Rejection.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 2, 4 – 6, 8 – 14, and 16 are rejected under 35 U.S.C. 102(a)(1) as being unpatentable over Hakan Boyraz, et. al. (US 20210405638 A1), hereinafter referred to as Boyraz.
Regarding Claim 1:
A feature fusion system for fusing sensor data from a plurality of sensors of an autonomous vehicle, the feature fusion system comprising: circuitry configured to:
identify first sensor data and second sensor data from first and second sensors of the plurality of sensors, respectively, the first sensor data and the second sensor data being of an environment in which the autonomous vehicle is operating, the autonomous vehicle being a roadway vehicle;
Boyraz discloses “the sensor and data analysis system described herein provides a sensor and data analysis framework that dynamically accepts and employs inputs as available to continuously generate a best possible interpretation of the robotic system's environment. With reference to an illustrative example, the sensor and data analysis system may have access to information from multiple sensors that provide information regarding a current environment (for example, RGB sensors, LIDAR sensors, and depth sensors) and historical information that includes map data defining object placement and potential traversal paths. Accordingly, the sensor and data analysis system may utilize the sensor data from these three sensor systems to generate at least a partial map information from the combination of available sensor data.” (Boyraz, [0016]).
Examiner interprets “roadway vehicle” as a vehicle that is capable of driving on the road. The robot system taught by Boyraz is capable of this, as taught by the following: “Thus, if the robotic system 101 travels from the transportation vehicle on the roadway 104 (i.e., the first location) to the recipient across one of the first yard or the second yard (i.e., the second location). Thus, the robotic system 101 transporting an item from the first location to the second location travels across at least two different terrains (e.g., at least the asphalt of the roadway 104 and the concrete of the sidewalk).” (Boyraz, [0024]).
extract first features and second features, by one or more neural networks, based on the first sensor data and the second sensor data, respectively, the first features and the second features including the same area in the environment;
Boyraz discloses “As part of object tracking, the robotic system 101 may track objects that the robotic system 101 encounters frequently. For example, autonomous systems 101 may come across same types of objects regardless of the particular deliveries the robotic system 101 is serving. For example, the robotic system 101 may often see people, cars, bicycles, wheelchairs, strollers, mailboxes, and the like. As the autonomous systems 101 contact different objects, details of the object (for example, size, speed, type of motion, and so forth) may be stored. Object tracking by the robotic system 101 may utilize multiple sources of data. For example, the sources of data include data from sensors or similar devices. Such a device may comprise a convolutional neural network based object detector that enables the robotic system 101 (for example, via the perception system 200 or similar computing or processing system) to generate or estimate two-dimensional (2D) or three-dimensional (3D boundary or bounding boxes around detected objects.” (Boyraz, [0082]).
convert the first features and second features to first and second bird's eye view (BEV) projections of the first and second features, respectively;
Boyraz discloses “The CNN based object detector may further enable the perception system 200 to predict a class or confidence score (for example, for each of class and position confidences) for a detected object. The object tracking device may generate various outputs of information, including one or more: [0083] a 2D or 3D position (x,y,z) in a sensor frame, [0084] a 2D or 3D BEV bounding box, [0085] an object class, [0086] one or more confidence probabilities (for example, class or position).” (Boyraz, [0082]).
align, by a spatial transform network (STN) the second BEV projection with the first BEV projection into an aligned second BEV projection;
Boyraz discloses “the perception system 200 may implement a deep continuous integration pointwise integration approach or a spatial transform network for the integration of the panoramic viewpoint and the BEV.” (Boyraz, [0067]).
generate a fused feature map comprising the aligned second BEV projection and the first BEV projection; and
Boyraz discloses “The RGB depth network module 208 generates three (3) outputs based on the data received from the sensors 202 and the localization module 204. These outputs include: (1) 3D object proposal data 270 that is passed to an object tracking and future state prediction module 214; (2) labeled point cloud data 272 (for example, including terrain or obstacle data or semantic classes) to pass to the probabilistic 3D map module 216; and (3) curb traversability prediction data 274 passed to the BEV map module 218. In some embodiments, RGB depth network module 208 may combine data from one or more of the RGB sensors, LIDAR sensors, TOF sensors, or the stereo depth sensors to perform 3D object detection, semantic segmentation, terrain prediction, and depth prediction for partitions of the environment around the robotic system 101. Without limitation, the 3D object can include, but is not limited to one or more of a car, an emergency vehicle, a bicyclist, a pedestrian, a stroller, a wheelchair, a trashcan, a dog, a cat, and various other obstacles.” (Boyraz, [0034]).
control operation of the autonomous vehicle based on the fused feature map.
Boyraz discloses “The BEV map module 218 may project the 3D map data received from the probabilistic 3D map module 216 onto a 2D BEV map for use by the robotic system 101 to navigate through the environment of the robotic system 101.” (Boyraz, [0047]).
Regarding Claim 2:
The feature fusion system of claim 1, wherein: the STN comprises:
an artificial neural network configured to estimate a geometric transformation to align the second BEV projection with the first BEV projection; and
Boyraz discloses “Those features are transformed to a common camera frame via extrinsic parameters (e.g., via a transformation matrix between sensor frames to common coordinate frame). After the transformation, the features are aligned (spatially or temporally) to be merged or combined. The perception system 200, for example via the framework of FIG. 5, supports different combination methods, for example averaging, max-pooling, or concatenation, among others.” (Boyraz, [0072]).
a differentiable warping function configured to transform the second BEV projection based on the geometric transformation.
Boyraz discloses “For example, features are extracted from every RGB sensor viewpoint and every other sensor viewpoint and the information is combined using geometric warping or merging into or with data from other sensor types. For example, LIDAR features are obtained and geometrically warped into the data from the RGB sensors for the semantic segmentation (for example, by the segmentation module 312), the obstacle detection, and the depth completion (for example, by the depth completion module 314) while the features from the RGB sensor data are merged or applied to the LIDAR sensor data for 3D object detection” (Boyraz, [0065]).
Regarding Claim 4:
The feature fusion system of claim 1, wherein:
the fused feature map represents a spatial region proximate to the feature fusion system, and
Boyraz discloses “In some embodiments, the perception system 200 generates the terrain map to enable the robotic system 101 to determine where to not travel, regardless of capabilities of the robotic system 101. For example, when delivering objects to a residential delivery address, the robotic system 101 may be configured to avoid traveling on lawns, through gardens, or underneath vehicles, among other scenarios. The robotic system 101 may use the terrain map to identify partitions in the terrain map that are not traversable. For example, the probabilistic 3D map module 216 or the BED map module 218 may identify partitions of generated maps that are traversable or not traversable using state identifiers for each partition. The state identifiers (or labels) may indicate which partitions are traversable and what the terrain is for each partition (for example, concrete—sidewalk, concrete—driveway, grass, mulch, dirt, sand, asphalt, and so forth).” (Boyraz, [0112]).
the circuitry is further configured to determine, utilizing the fused feature map, a path through the spatial region.
Boyraz discloses “Using the various modules and algorithms described herein, the robotic system 101 may identify or generate a terrain map for the environment of the robotic system 101. The robotic system 101 may dynamically update or regenerate the terrain map as the robotic system 101 is traveling through the environment. The robotic system 101 may use the terrain map and the modules described herein to identify a path through the environment from the first location to the second location.” (Boyraz, [0111]).
Regarding Claim 5:
The feature fusion system of claim 1, wherein:
the circuitry is further configured to extract the first features and second features from the first sensor data and second sensor data, respectively, utilizing one or more convolutional neural networks.
Boyraz discloses “As part of object tracking, the robotic system 101 may track objects that the robotic system 101 encounters frequently. For example, autonomous systems 101 may come across same types of objects regardless of the particular deliveries the robotic system 101 is serving. For example, the robotic system 101 may often see people, cars, bicycles, wheelchairs, strollers, mailboxes, and the like. As the autonomous systems 101 contact different objects, details of the object (for example, size, speed, type of motion, and so forth) may be stored. Object tracking by the robotic system 101 may utilize multiple sources of data. For example, the sources of data include data from sensors or similar devices. Such a device may comprise a convolutional neural network based object detector that enables the robotic system 101 (for example, via the perception system 200 or similar computing or processing system) to generate or estimate two-dimensional (2D) or three-dimensional (3D boundary or bounding boxes around detected objects.” (Boyraz, [0082]).
Regarding Claim 6:
The feature fusion system of claim 1, wherein:
the first and second BEV projections are represented in two-dimensional space.
Boyraz discloses “The BEV map module 218 may project the 3D map data received from the probabilistic 3D map module 216 onto a 2D BEV map for use by the robotic system 101 to navigate through the environment of the robotic system 101.” (Boyraz, [0047]).
Regarding Claim 8:
The feature fusion system of claim 1, wherein:
the first and second sensors comprise cameras and/or light detection and ranging (LIDAR) sensors.
Boyraz discloses “The sensors 202 may include a plurality of sensors, including a plurality of RGB sensors disposed on a front or front surface or a back or back surface of the robotic system 101, a plurality of ultrasonic sensors, a time-of-flight (TOF) sensor, stereo sensors, a LIDAR sensor, and so forth.” (Boyraz, [0032]).
Regarding Claim 9:
A method of fusing sensor data from a plurality of sensors of an autonomous vehicle, the method comprising:
identifying first sensor data and second sensor data from first and second sensors of the plurality of sensors, respectively, the first sensor data and the second sensor data being of an environment in which the autonomous vehicle is operating, the autonomous vehicle being a roadway vehicle;
Boyraz discloses “the sensor and data analysis system described herein provides a sensor and data analysis framework that dynamically accepts and employs inputs as available to continuously generate a best possible interpretation of the robotic system's environment. With reference to an illustrative example, the sensor and data analysis system may have access to information from multiple sensors that provide information regarding a current environment (for example, RGB sensors, LIDAR sensors, and depth sensors) and historical information that includes map data defining object placement and potential traversal paths. Accordingly, the sensor and data analysis system may utilize the sensor data from these three sensor systems to generate at least a partial map information from the combination of available sensor data.” (Boyraz, [0016]).
extracting first features and second features, from one or more neural networks, based on the first sensor data and the second sensor data, respectively, the first features and the second features including the same area in the environment;
Boyraz discloses “As part of object tracking, the robotic system 101 may track objects that the robotic system 101 encounters frequently. For example, autonomous systems 101 may come across same types of objects regardless of the particular deliveries the robotic system 101 is serving. For example, the robotic system 101 may often see people, cars, bicycles, wheelchairs, strollers, mailboxes, and the like. As the autonomous systems 101 contact different objects, details of the object (for example, size, speed, type of motion, and so forth) may be stored. Object tracking by the robotic system 101 may utilize multiple sources of data. For example, the sources of data include data from sensors or similar devices. Such a device may comprise a convolutional neural network based object detector that enables the robotic system 101 (for example, via the perception system 200 or similar computing or processing system) to generate or estimate two-dimensional (2D) or three-dimensional (3D boundary or bounding boxes around detected objects.” (Boyraz, [0082]).
converting the first features and second features to first and second bird's eye view (BEV) projections of the first and second features, respectively;
Boyraz discloses “The CNN based object detector may further enable the perception system 200 to predict a class or confidence score (for example, for each of class and position confidences) for a detected object. The object tracking device may generate various outputs of information, including one or more: [0083] a 2D or 3D position (x,y,z) in a sensor frame, [0084] a 2D or 3D BEV bounding box, [0085] an object class, [0086] one or more confidence probabilities (for example, class or position).” (Boyraz, [0082]).
aligning, by a spatial transform network (STN) the second BEV projection with the first BEV projection into an aligned second BEV projection;
Boyraz discloses “the perception system 200 may implement a deep continuous integration pointwise integration approach or a spatial transform network for the integration of the panoramic viewpoint and the BEV.” (Boyraz, [0067]).
generating a fused feature map comprising the aligned second BEV projection and the first BEV projection; and
Boyraz discloses “The RGB depth network module 208 generates three (3) outputs based on the data received from the sensors 202 and the localization module 204. These outputs include: (1) 3D object proposal data 270 that is passed to an object tracking and future state prediction module 214; (2) labeled point cloud data 272 (for example, including terrain or obstacle data or semantic classes) to pass to the probabilistic 3D map module 216; and (3) curb traversability prediction data 274 passed to the BEV map module 218. In some embodiments, RGB depth network module 208 may combine data from one or more of the RGB sensors, LIDAR sensors, TOF sensors, or the stereo depth sensors to perform 3D object detection, semantic segmentation, terrain prediction, and depth prediction for partitions of the environment around the robotic system 101. Without limitation, the 3D object can include, but is not limited to one or more of a car, an emergency vehicle, a bicyclist, a pedestrian, a stroller, a wheelchair, a trashcan, a dog, a cat, and various other obstacles.” (Boyraz, [0034]).
controlling operation of the autonomous vehicle based on the fused feature map.
Boyraz discloses “The BEV map module 218 may project the 3D map data received from the probabilistic 3D map module 216 onto a 2D BEV map for use by the robotic system 101 to navigate through the environment of the robotic system 101.” (Boyraz, [0047]).
Regarding Claim 10:
The method of claim 9, wherein:
the STN comprises an artificial neural network and a differentiable warping function, and the method further comprises:
estimating, by the artificial neural network, a geometric transformation to align the second BEV projection with the first BEV projection; and
Boyraz discloses “Those features are transformed to a common camera frame via extrinsic parameters (e.g., via a transformation matrix between sensor frames to common coordinate frame). After the transformation, the features are aligned (spatially or temporally) to be merged or combined. The perception system 200, for example via the framework of FIG. 5, supports different combination methods, for example averaging, max-pooling, or concatenation, among others.” (Boyraz, [0072]).
transforming, by the differentiable warping function, the second BEV projection based on the geometric transformation.
Boyraz discloses “For example, features are extracted from every RGB sensor viewpoint and every other sensor viewpoint and the information is combined using geometric warping or merging into or with data from other sensor types. For example, LIDAR features are obtained and geometrically warped into the data from the RGB sensors for the semantic segmentation (for example, by the segmentation module 312), the obstacle detection, and the depth completion (for example, by the depth completion module 314) while the features from the RGB sensor data are merged or applied to the LIDAR sensor data for 3D object detection” (Boyraz, [0065]).
Regarding Claim 11:
The method of claim 10, further comprising:
comparing the first BEV projection with the aligned second BEV projection to identify a difference; and
Marcotte discloses “As to generating training data to train the system for determining implicit lane boundaries, the system and method may collect sensor data that observes the movement of vehicles from one lane to another. In one example, the sensor data may be collected from vehicles as they travel from lane to lane. In another example, the sensor data may be collected from one or more fixed sensors that observe a particular set of lanes and how vehicles travel from lane to lane, especially though intersections. As such, the collected sensor data may be utilized to generate a ground truth bird's eye view of an observed road environment, overlay a ground truth grid having cells onto the ground truth bird's eye view, and label the cells of the ground truth grid based on the movement of vehicles as observed. This information may then be utilized to train a neural network of a system that determines implicit lane boundaries.” (Marcotte, [0028]).
re-training the artificial neural network of the STN based on the difference.
Marcotte discloses “As to generating training data to train the system for determining implicit lane boundaries, the system and method may collect sensor data that observes the movement of vehicles from one lane to another. In one example, the sensor data may be collected from vehicles as they travel from lane to lane. In another example, the sensor data may be collected from one or more fixed sensors that observe a particular set of lanes and how vehicles travel from lane to lane, especially though intersections. As such, the collected sensor data may be utilized to generate a ground truth bird's eye view of an observed road environment, overlay a ground truth grid having cells onto the ground truth bird's eye view, and label the cells of the ground truth grid based on the movement of vehicles as observed. This information may then be utilized to train a neural network of a system that determines implicit lane boundaries.” (Marcotte, [0028]).
Regarding Claim 12:
The method of claim 9, wherein:
the fused feature map represents a spatial region proximate to the first and second sensors, and
Boyraz discloses “In some embodiments, the perception system 200 generates the terrain map to enable the robotic system 101 to determine where to not travel, regardless of capabilities of the robotic system 101. For example, when delivering objects to a residential delivery address, the robotic system 101 may be configured to avoid traveling on lawns, through gardens, or underneath vehicles, among other scenarios. The robotic system 101 may use the terrain map to identify partitions in the terrain map that are not traversable. For example, the probabilistic 3D map module 216 or the BED map module 218 may identify partitions of generated maps that are traversable or not traversable using state identifiers for each partition. The state identifiers (or labels) may indicate which partitions are traversable and what the terrain is for each partition (for example, concrete—sidewalk, concrete—driveway, grass, mulch, dirt, sand, asphalt, and so forth).” (Boyraz, [0112]).
the method further comprises:
determining, utilizing the fused feature map, a path through the spatial region.
Boyraz discloses “Using the various modules and algorithms described herein, the robotic system 101 may identify or generate a terrain map for the environment of the robotic system 101. The robotic system 101 may dynamically update or regenerate the terrain map as the robotic system 101 is traveling through the environment. The robotic system 101 may use the terrain map and the modules described herein to identify a path through the environment from the first location to the second location.” (Boyraz, [0111]).
Regarding Claim 13:
The method of claim 9, wherein extracting the first features and second features further comprises:
utilizing one or more convolutional neural networks to extract the first features and second features from the first sensor data and second sensor data, respectively.
Boyraz discloses “As part of object tracking, the robotic system 101 may track objects that the robotic system 101 encounters frequently. For example, autonomous systems 101 may come across same types of objects regardless of the particular deliveries the robotic system 101 is serving. For example, the robotic system 101 may often see people, cars, bicycles, wheelchairs, strollers, mailboxes, and the like. As the autonomous systems 101 contact different objects, details of the object (for example, size, speed, type of motion, and so forth) may be stored. Object tracking by the robotic system 101 may utilize multiple sources of data. For example, the sources of data include data from sensors or similar devices. Such a device may comprise a convolutional neural network based object detector that enables the robotic system 101 (for example, via the perception system 200 or similar computing or processing system) to generate or estimate two-dimensional (2D) or three-dimensional (3D boundary or bounding boxes around detected objects.” (Boyraz, [0082]).
Regarding Claim 14:
The method of claim 9, wherein:
the first and second BEV projections are represented in two-dimensional space.
Boyraz discloses “The BEV map module 218 may project the 3D map data received from the probabilistic 3D map module 216 onto a 2D BEV map for use by the robotic system 101 to navigate through the environment of the robotic system 101.” (Boyraz, [0047]).
Regarding Claim 16:
The method of claim 9, wherein:
the first and second sensors comprise cameras and/or light detection and ranging (LiDAR) sensors.
Boyraz discloses “The sensors 202 may include a plurality of sensors, including a plurality of RGB sensors disposed on a front or front surface or a back or back surface of the robotic system 101, a plurality of ultrasonic sensors, a time-of-flight (TOF) sensor, stereo sensors, a LIDAR sensor, and so forth.” (Boyraz, [0032]).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 3, 17 – 19 are rejected under 35 U.S.C. 103 as being unpatentable over Hakan Boyraz, et. al. (US 20210405638 A1), hereinafter referred to as Boyraz, in view of Ryan J. Marcotte (US 20220198198 A1), hereinafter referred to as Marcotte.
Regarding Claim 3:
The feature fusion system of claim 2, wherein:
the circuitry is further configured to:
compare the first BEV projection with the aligned second BEV projection to identify a difference; and
Marcotte discloses “As to generating training data to train the system for determining implicit lane boundaries, the system and method may collect sensor data that observes the movement of vehicles from one lane to another. In one example, the sensor data may be collected from vehicles as they travel from lane to lane. In another example, the sensor data may be collected from one or more fixed sensors that observe a particular set of lanes and how vehicles travel from lane to lane, especially though intersections. As such, the collected sensor data may be utilized to generate a ground truth bird's eye view of an observed road environment, overlay a ground truth grid having cells onto the ground truth bird's eye view, and label the cells of the ground truth grid based on the movement of vehicles as observed. This information may then be utilized to train a neural network of a system that determines implicit lane boundaries.” (Marcotte, [0028]).
re-train the artificial neural network of the STN based on the difference.
Marcotte discloses “As to generating training data to train the system for determining implicit lane boundaries, the system and method may collect sensor data that observes the movement of vehicles from one lane to another. In one example, the sensor data may be collected from vehicles as they travel from lane to lane. In another example, the sensor data may be collected from one or more fixed sensors that observe a particular set of lanes and how vehicles travel from lane to lane, especially though intersections. As such, the collected sensor data may be utilized to generate a ground truth bird's eye view of an observed road environment, overlay a ground truth grid having cells onto the ground truth bird's eye view, and label the cells of the ground truth grid based on the movement of vehicles as observed. This information may then be utilized to train a neural network of a system that determines implicit lane boundaries.” (Marcotte, [0028]).
It would have been obvious to one having ordinary skill in the art at the time of the applicant’s effective filing date to combine the system taught by Boyraz with the training of a neural net taught by Marcotte because Marcotte discloses a method for taking in sensor data and converting it to a BEV, “As such, the collected sensor data may be utilized to generate a ground truth bird's eye view of an observed road environment, overlay a ground truth grid having cells onto the ground truth bird's eye view, and label the cells of the ground truth grid based on the movement of vehicles as observed. This information may then be utilized to train a neural network of a system that determines implicit lane boundaries.” (Marcotte, [0028]), and combining the data such that it can be used to determine implicit features of the road.
Regarding Claim 17:
A feature fusion system for fusing sensor data from a plurality of sensors of an autonomous vehicle, the feature fusion system comprising:
circuitry configured to:
identify first sensor data and second sensor data from first and second sensors of the plurality of sensors, respectively, the first sensor data and the second sensor data being of an environment in which the autonomous vehicle is operating, the autonomous vehicle being a roadway vehicle;
Boyraz discloses “the sensor and data analysis system described herein provides a sensor and data analysis framework that dynamically accepts and employs inputs as available to continuously generate a best possible interpretation of the robotic system's environment. With reference to an illustrative example, the sensor and data analysis system may have access to information from multiple sensors that provide information regarding a current environment (for example, RGB sensors, LIDAR sensors, and depth sensors) and historical information that includes map data defining object placement and potential traversal paths. Accordingly, the sensor and data analysis system may utilize the sensor data from these three sensor systems to generate at least a partial map information from the combination of available sensor data.” (Boyraz, [0016]).
extract first features and second features, by one or more neural networks, based on the first and second sensor data, respectively, the first features and the second features including the same area in the environment;
Boyraz discloses “As part of object tracking, the robotic system 101 may track objects that the robotic system 101 encounters frequently. For example, autonomous systems 101 may come across same types of objects regardless of the particular deliveries the robotic system 101 is serving. For example, the robotic system 101 may often see people, cars, bicycles, wheelchairs, strollers, mailboxes, and the like. As the autonomous systems 101 contact different objects, details of the object (for example, size, speed, type of motion, and so forth) may be stored. Object tracking by the robotic system 101 may utilize multiple sources of data. For example, the sources of data include data from sensors or similar devices. Such a device may comprise a convolutional neural network based object detector that enables the robotic system 101 (for example, via the perception system 200 or similar computing or processing system) to generate or estimate two-dimensional (2D) or three-dimensional (3D boundary or bounding boxes around detected objects.” (Boyraz, [0082]).
convert the first features and second features to first and second bird's eye view (BEV) projections of the first features and second features, respectively;
Boyraz discloses “The CNN based object detector may further enable the perception system 200 to predict a class or confidence score (for example, for each of class and position confidences) for a detected object. The object tracking device may generate various outputs of information, including one or more: [0083] a 2D or 3D position (x,y,z) in a sensor frame, [0084] a 2D or 3D BEV bounding box, [0085] an object class, [0086] one or more confidence probabilities (for example, class or position).” (Boyraz, [0082]).
align, by spatial transform network (STN) the second BEV projection with the first BEV projection into an aligned second BEV projection, wherein the STN comprises:
Boyraz discloses “the perception system 200 may implement a deep continuous integration pointwise integration approach or a spatial transform network for the integration of the panoramic viewpoint and the BEV.” (Boyraz, [0067]).
an artificial neural network configured to estimate a geometric transformation to align the second BEV projection with the first BEV projection; and
Boyraz discloses “Those features are transformed to a common camera frame via extrinsic parameters (e.g., via a transformation matrix between sensor frames to common coordinate frame). After the transformation, the features are aligned (spatially or temporally) to be merged or combined. The perception system 200, for example via the framework of FIG. 5, supports different combination methods, for example averaging, max-pooling, or concatenation, among others.” (Boyraz, [0072]).
a differentiable warping function configured to transform the second BEV projection based on the geometric transformation; and
Boyraz discloses “For example, features are extracted from every RGB sensor viewpoint and every other sensor viewpoint and the information is combined using geometric warping or merging into or with data from other sensor types. For example, LIDAR features are obtained and geometrically warped into the data from the RGB sensors for the semantic segmentation (for example, by the segmentation module 312), the obstacle detection, and the depth completion (for example, by the depth completion module 314) while the features from the RGB sensor data are merged or applied to the LIDAR sensor data for 3D object detection” (Boyraz, [0065]).
compare the first BEV projection with the aligned second BEV projection to identify a difference; and
Marcotte discloses “As to generating training data to train the system for determining implicit lane boundaries, the system and method may collect sensor data that observes the movement of vehicles from one lane to another. In one example, the sensor data may be collected from vehicles as they travel from lane to lane. In another example, the sensor data may be collected from one or more fixed sensors that observe a particular set of lanes and how vehicles travel from lane to lane, especially though intersections. As such, the collected sensor data may be utilized to generate a ground truth bird's eye view of an observed road environment, overlay a ground truth grid having cells onto the ground truth bird's eye view, and label the cells of the ground truth grid based on the movement of vehicles as observed. This information may then be utilized to train a neural network of a system that determines implicit lane boundaries.” (Marcotte, [0028]).
re-train the artificial neural network of the STN based on the difference, wherein a controller of the autonomous vehicle is configured to control operation of the autonomous vehicle based on an output of the re-trained artificial neural network of the STN.
Marcotte discloses “As to generating training data to train the system for determining implicit lane boundaries, the system and method may collect sensor data that observes the movement of vehicles from one lane to another. In one example, the sensor data may be collected from vehicles as they travel from lane to lane. In another example, the sensor data may be collected from one or more fixed sensors that observe a particular set of lanes and how vehicles travel from lane to lane, especially though intersections. As such, the collected sensor data may be utilized to generate a ground truth bird's eye view of an observed road environment, overlay a ground truth grid having cells onto the ground truth bird's eye view, and label the cells of the ground truth grid based on the movement of vehicles as observed. This information may then be utilized to train a neural network of a system that determines implicit lane boundaries.” (Marcotte, [0028]).
It would have been obvious to one having ordinary skill in the art at the time of the applicant’s effective filing date to combine the system taught by Boyraz with the training of a neural net taught by Marcotte because Marcotte discloses a method for taking in sensor data and converting it to a BEV, “As such, the collected sensor data may be utilized to generate a ground truth bird's eye view of an observed road environment, overlay a ground truth grid having cells onto the ground truth bird's eye view, and label the cells of the ground truth grid based on the movement of vehicles as observed. This information may then be utilized to train a neural network of a system that determines implicit lane boundaries.” (Marcotte, [0028]), and combining the data such that it can be used to determine implicit features of the road.
Regarding Claim 18:
The feature fusion system of claim 17, wherein:
the circuitry is further configured to:
generate a fused feature map comprising the aligned second BEV projection and the first BEV projection.
Boyraz discloses “The RGB depth network module 208 generates three (3) outputs based on the data received from the sensors 202 and the localization module 204. These outputs include: (1) 3D object proposal data 270 that is passed to an object tracking and future state prediction module 214; (2) labeled point cloud data 272 (for example, including terrain or obstacle data or semantic classes) to pass to the probabilistic 3D map module 216; and (3) curb traversability prediction data 274 passed to the BEV map module 218. In some embodiments, RGB depth network module 208 may combine data from one or more of the RGB sensors, LIDAR sensors, TOF sensors, or the stereo depth sensors to perform 3D object detection, semantic segmentation, terrain prediction, and depth prediction for partitions of the environment around the robotic system 101. Without limitation, the 3D object can include, but is not limited to one or more of a car, an emergency vehicle, a bicyclist, a pedestrian, a stroller, a wheelchair, a trashcan, a dog, a cat, and various other obstacles.” (Boyraz, [0034]).
Regarding Claim 19:
The feature fusion system of claim 18, wherein:
the fused feature map represents a spatial region proximate to the feature fusion system, and
Boyraz discloses “In some embodiments, the perception system 200 generates the terrain map to enable the robotic system 101 to determine where to not travel, regardless of capabilities of the robotic system 101. For example, when delivering objects to a residential delivery address, the robotic system 101 may be configured to avoid traveling on lawns, through gardens, or underneath vehicles, among other scenarios. The robotic system 101 may use the terrain map to identify partitions in the terrain map that are not traversable. For example, the probabilistic 3D map module 216 or the BED map module 218 may identify partitions of generated maps that are traversable or not traversable using state identifiers for each partition. The state identifiers (or labels) may indicate which partitions are traversable and what the terrain is for each partition (for example, concrete—sidewalk, concrete—driveway, grass, mulch, dirt, sand, asphalt, and so forth).” (Boyraz, [0112]).
the circuitry is further configured to determine, utilizing the fused feature map, a path through the spatial region.
Boyraz discloses “Using the various modules and algorithms described herein, the robotic system 101 may identify or generate a terrain map for the environment of the robotic system 101. The robotic system 101 may dynamically update or regenerate the terrain map as the robotic system 101 is traveling through the environment. The robotic system 101 may use the terrain map and the modules described herein to identify a path through the environment from the first location to the second location.” (Boyraz, [0111]).
Claims 7 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Boyraz, et. al. (US 20210405638 A1), hereinafter referred to as Boyraz, in view of Kumar, et. al (US 20210148709 A1), hereinafter referred to as Kumar.
Regarding Claim 7:
The feature fusion system of claim 1, wherein:
the first and second sensors include a first pair of sensors and a second pair of sensors, the first sensor data are acquired by the first pair of sensors, the second sensor data are acquired by the second pair of sensors, and the circuitry is further configured to:
Boyraz discloses “The sensors 202 may include a plurality of sensors, including a plurality of RGB sensors disposed on a front or front surface or a back or back surface of the robotic system 101, a plurality of ultrasonic sensors, a time-of-flight (TOF) sensor, stereo sensors, a LIDAR sensor, and so forth. The RGB sensors may include, but is not limited to, three RGB sensors with overlapping fields of view… The sensors 202 may also include an RGB sensor disposed at the back or rear of the robotic system 101. The sensors 202 also include a plurality of stereo depth sensors having similar fields of view or placements as the RGB sensors. The stereo depth sensors may be used to generate depth data.” (Boyraz, [0032]).
determine a first disparity in the first features based on the first sensor data and a second disparity in the second features based on the second sensor data;
Boyraz discloses “the perception system 200 may project features (for example, objects, terrain, depth information, and so forth) computed from a point cloud, for example disparity” (Boyraz, [0058]).
convert the first disparity into a first depth of the first features and the second disparity into a second depth of the second features;
Boyraz discloses “The robotic system 101 may use this map to determine a traversability score that measures how traversable each pixel or portion of the map is relative to other pixels or portions of the map. In some embodiments, the robotic system 101 also uses the perception system to label a terrain map or model (referred to herein as a map) with determined surface or terrain types (for example, sidewalk, road, curb, driveway, crosswalk, parking lanes, grass, and so forth).” (Boyraz, [0025]).
generate first three-dimensional (3D) features of the first features based on the first depth and second 3D features of the second features based on the second depth; and
Boyraz discloses “Illustratively, the processing results generated by the sensor and data analysis system may correspond to a three-dimensional (3D) representation of the environment around the robotic system, including dynamic and stationary objects.” (Boyraz, [0019]).
Kumar discloses “Voxelizing 910 the LiDAR point cloud can produce a three-dimensional representation of each of the one or more objects in the physical surroundings detected by the LiDAR sensor 320.” (Kumar, [0119]).
project the first 3D features onto a BEV plane into the first BEV projection and the second 3D features onto the BEV plane into the second BEV projection.
Kumar discloses “FIG. 10 is a flowchart illustrating an exemplary process for object detection according to one embodiment of the present disclosure. As illustrated in this example, object detection in a LiDAR point cloud can begin with placing, by a navigation system 302, a plurality of anchor points in a two-dimensional Bird's Eye View (BEV) of spatial points represented in a segmented ground surface representation of objects detected by a LiDAR sensor 320.” (Kumar, [0123]).
It would have been obvious to one having ordinary skill in the art at the time of the applicant’s effective filing date to combine the system taught by Boyraz with the ability to place simulated 3D objects onto a 2D BEV because if information is collected in 3D, but the map used by the system is 2D, then it would be required for the system to convert the data collected by the sensors into useable data by the system. Kumar discloses a system for projecting the processed 3D data from a LiDAR point cloud onto a simulated two-dimensional BEV.
Regarding Claim 15:
The method of claim 9, wherein:
the first and second sensors include a first pair of sensors and a second pair of sensors, the first sensor data are acquired by the first pair of sensors, the second sensor data are acquired by the second pair of sensors, and the method further comprises:
Boyraz discloses “The sensors 202 may include a plurality of sensors, including a plurality of RGB sensors disposed on a front or front surface or a back or back surface of the robotic system 101, a plurality of ultrasonic sensors, a time-of-flight (TOF) sensor, stereo sensors, a LIDAR sensor, and so forth. The RGB sensors may include, but is not limited to, three RGB sensors with overlapping fields of view… The sensors 202 may also include an RGB sensor disposed at the back or rear of the robotic system 101. The sensors 202 also include a plurality of stereo depth sensors having similar fields of view or placements as the RGB sensors. The stereo depth sensors may be used to generate depth data.” (Boyraz, [0032]).
determining a first disparity in the first features based on the first sensor data and a second disparity in the second features based on the second sensor data;
Boyraz discloses “the perception system 200 may project features (for example, objects, terrain, depth information, and so forth) computed from a point cloud, for example disparity” (Boyraz, [0058]).
converting the first disparity into a first depth of the first features and the second disparity into a second depth of the second features;
Boyraz discloses “The robotic system 101 may use this map to determine a traversability score that measures how traversable each pixel or portion of the map is relative to other pixels or portions of the map. In some embodiments, the robotic system 101 also uses the perception system to label a terrain map or model (referred to herein as a map) with determined surface or terrain types (for example, sidewalk, road, curb, driveway, crosswalk, parking lanes, grass, and so forth).” (Boyraz, [0025]).
generating first three-dimensional (3D) features of the first features based on the first depth and second 3D features of the second features based on the second depth; and
Boyraz discloses “Illustratively, the processing results generated by the sensor and data analysis system may correspond to a three-dimensional (3D) representation of the environment around the robotic system, including dynamic and stationary objects.” (Boyraz, [0019]).
Kumar discloses “Voxelizing 910 the LiDAR point cloud can produce a three-dimensional representation of each of the one or more objects in the physical surroundings detected by the LiDAR sensor 320.” (Kumar, [0119]).
projecting the first 3D features onto a BEV plane into the first BEV projection and the second 3D features onto the BEV plane into the second BEV projection.
Kumar discloses “FIG. 10 is a flowchart illustrating an exemplary process for object detection according to one embodiment of the present disclosure. As illustrated in this example, object detection in a LiDAR point cloud can begin with placing, by a navigation system 302, a plurality of anchor points in a two-dimensional Bird's Eye View (BEV) of spatial points represented in a segmented ground surface representation of objects detected by a LiDAR sensor 320.” (Kumar, [0123]).
It would have been obvious to one having ordinary skill in the art at the time of the applicant’s effective filing date to combine the system taught by Boyraz with the ability to place simulated 3D objects onto a 2D BEV because if information is collected in 3D, but the map used by the system is 2D, then it would be required for the system to convert the data collected by the sensors into useable data by the system. Kumar discloses a system for projecting the processed 3D data from a LiDAR point cloud onto a simulated two-dimensional BEV.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Boyraz, et. al. (US 20210405638 A1), hereinafter referred to as Boyraz, in view of Marcotte (US 20220198198 A1), hereinafter referred to as Marcotte, further in view of Kumar, et. al (US 20210148709 A1), hereinafter referred to as Kumar.
Regarding Claim 20:
The feature fusion system of claim 17, wherein:
the first and second sensors include a first pair of sensors and a second pair of sensors, the first sensor data are acquired by the first pair of sensors, the second sensor data are acquired by the second pair of sensors, and the first pair of sensors and the second pair of sensors comprise cameras and/or light detection and ranging (LIDAR) sensors, and the circuitry is further configured to:
Boyraz discloses “The sensors 202 may include a plurality of sensors, including a plurality of RGB sensors disposed on a front or front surface or a back or back surface of the robotic system 101, a plurality of ultrasonic sensors, a time-of-flight (TOF) sensor, stereo sensors, a LIDAR sensor, and so forth. The RGB sensors may include, but is not limited to, three RGB sensors with overlapping fields of view… The sensors 202 may also include an RGB sensor disposed at the back or rear of the robotic system 101. The sensors 202 also include a plurality of stereo depth sensors having similar fields of view or placements as the RGB sensors. The stereo depth sensors may be used to generate depth data.” (Boyraz, [0032]).
Boyraz discloses “The sensors 202 may include a plurality of sensors, including a plurality of RGB sensors disposed on a front or front surface or a back or back surface of the robotic system 101, a plurality of ultrasonic sensors, a time-of-flight (TOF) sensor, stereo sensors, a LIDAR sensor, and so forth.” (Boyraz, [0032]).
determine a first disparity in the first features based on the first sensor data and a second disparity in the second features based on the second sensor data;
Boyraz discloses “the perception system 200 may project features (for example, objects, terrain, depth information, and so forth) computed from a point cloud, for example disparity” (Boyraz, [0058]).
convert the first disparity into a first depth of the first features and the second disparity into a second depth of the second features;
Boyraz discloses “The robotic system 101 may use this map to determine a traversability score that measures how traversable each pixel or portion of the map is relative to other pixels or portions of the map. In some embodiments, the robotic system 101 also uses the perception system to label a terrain map or model (referred to herein as a map) with determined surface or terrain types (for example, sidewalk, road, curb, driveway, crosswalk, parking lanes, grass, and so forth).” (Boyraz, [0025]).
generate first three-dimensional (3D) features of the first features based on the first depth and second 3D features of the second features based on the second depth; and
Boyraz discloses “Illustratively, the processing results generated by the sensor and data analysis system may correspond to a three-dimensional (3D) representation of the environment around the robotic system, including dynamic and stationary objects.” (Boyraz, [0019]).
Kumar discloses “Voxelizing 910 the LiDAR point cloud can produce a three-dimensional representation of each of the one or more objects in the physical surroundings detected by the LiDAR sensor 320.” (Kumar, [0119]).
project the first 3D features onto a BEV plane into the first BEV projection and the second 3D features onto the BEV plane into the second BEV projection.
Kumar discloses “FIG. 10 is a flowchart illustrating an exemplary process for object detection according to one embodiment of the present disclosure. As illustrated in this example, object detection in a LiDAR point cloud can begin with placing, by a navigation system 302, a plurality of anchor points in a two-dimensional Bird's Eye View (BEV) of spatial points represented in a segmented ground surface representation of objects detected by a LiDAR sensor 320.” (Kumar, [0123]).
It would have been obvious to one having ordinary skill in the art at the time of the applicant’s effective filing date to combine the system taught by Boyraz and Marcotte with the ability to place simulated 3D objects onto a 2D BEV because if information is collected in 3D, but the map used by the system is 2D, then it would be required for the system to convert the data collected by the sensors into useable data by the system. Kumar discloses a system for projecting the processed 3D data from a LiDAR point cloud onto a simulated two-dimensional BEV.
Conclusion
Arata Itoh (US 20220303505 A1)
Itoh was not used because although Itoh teaches the ability to use sensors to create a BEV of the surrounding area, it does not use a Spatial Transform Network to categorize objects.
Benjamin Lund, et. al. (US 20200327343 A1)
Lund was not used because although Lund teaches the ability to locate the position of other vehicles within a space, Lund does not teach a Spatial Transform Network for the locating.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAMES B CHIN whose telephone number is (571)272-4634. The examiner can normally be reached Monday - Friday | 9:00 AM to 5:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wade Miles can be reached at (571) 270-7777. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.B.C./
Examiner, Art Unit 3656
/WADE MILES/Supervisory Patent Examiner, Art Unit 3656
Prosecution Insights