DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/31/2025 has been entered.
Response to Arguments
Applicant’s arguments with respect to the claim(s) have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Prior art Marvasti et al., US 2023/0267720 A1 (Marvasti) has been newly added to assist in teaching the newly added claims and claim amendments. Prior art Morales (Morales et al., “A Combined Voxel and Particle Filter-Based Approach for Fast Obstacle Detection and Tracking in Automotive Applications”) is no longer used within the current Office Action.
Claims 1-13, 15-18, and 21-23 are pending; claims 1, 3, 8, 10, 15, and 16 have been amended; claims 14, 19, and 20 have been canceled; and claims 21-23 have been newly added.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-5, 7-12, 15-18, and 21-23 are rejected under 35 U.S.C. 103 as being unpatentable over Laddah et al., US 2022/0035376 A1 (Laddah), Gomez et al., US 2022/0036650 A1 (Gomez), and further in view of Marvasti et al., US 2023/0267720 A1 (Marvasti).
Regarding claim 1, Laddah teaches a method comprising:
obtaining, by a processing device (computing system including processor(s)) ([0050]), input data (obtaining LIDAR data, radar data, and map data) (Abstract and [0027]) derived from a set of sensors associated with an autonomous vehicle (AV) (obtaining, from one or more sensors of an autonomous vehicle, LIDAR data and radar data) ([0005] and [0023]);
extracting, by the processing device (computing system including processor(s)) ([0050]) from the input data (extracting from the data input) ([0027]), a plurality of sets of features (one or more feature for the data) ([0027]);
performing, by the processing device (computing system including processor(s)) ([0050]), two-dimensional (2D) mapping (generating 2D map data) ([0086]) to obtain a projected bird's-eye view (BEV) (to generate a two-dimensional top-down grid comprising a plurality of cell, such as a Bird’s Eye View (BEV)) ([0086]);
generating, by the processing device (computing system including processor(s)) ([0050]) using the plurality of sets of features (using the spatial features) ([0092] and [0097]), a fused bird's-eye view (BEV) grid (and the output domain can include coordinate frame cells; e.g. x, y BEV grid cells) (Fig. 3; [0092]) (wherein the features can be combined to generate fused feature data) ([0097]);
and providing, by the processing device (computing system including processor(s)) ([0050]), the fused BEV grid for object detection (using the output from the fused data to input into a detection layer to detect one or more objects from the fused sensor data) ([0104]).
Laddah teaches using a Bird’s Eye View (BEV) grid ([0086] and [0092]); wherein the data for sensors as well as map data can be fused together ([0086]); and wherein the features can be combined to generate fused feature data ([0097] and [0103]). However, Laddah does not explicitly state “performing, by the processing device, two-dimensional (2D) mapping to project a feature tensor formed from the plurality of sets of features onto a horizontal surface to obtain a projected bird's-eye view (BEV) feature tensor reflecting aggregated features along a vertical dimension” or a “projected BEV feature tensor”.
Gomez teaches system and methods and vehicles for object detection using pseudo-LiDAR ([0001]); performing, by the processing device (each of the one or more processors may be any device capable of executing machine readable and executable instructions) ([0043]), to project a feature tensor formed from the plurality of sets of features (projecting the feature vectors formed from the plurality of sets of features) (Fig. 4; [0024-0025] and [0030-0031]); and using the projected BEV feature tensor (projecting the feature vectors formed from the plurality of sets of features) (Fig. 4; [0024-0025] and [0030-0031]) to generate a fused BEV grid (combined/concatenated bird’s eye view map) (Fig. 4; Abstract, [0006], and [0029-0031]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Laddah to include combining the BEVs of different resolutions (voxel sizes) since it improves the performance of object detection for different types of objects (Gomez; [0019]).
However, neither explicitly teaches “performing, by the processing device, two-dimensional (2D) mapping” to project a feature tensor formed from the plurality of sets of features “onto a horizontal surface to obtain a projected bird's-eye view (BEV) feature tensor reflecting aggregated features along a vertical dimension”.
Marvasti teaches a mechanism to further improve object detection performance (Abstract); and performing, by the processing device (a device or an apparatus for performing operations) ([0074]), two-dimensional (2D) mapping (the BEV projector unit is used to project the aligned point-clouds onto a 2D image plane) ([0009] and [0032]) to project a feature tensor formed from the plurality of sets of features (a BEV image/tensor is generated based on a plurality of features) ([0009]) onto a horizontal surface to obtain a projected bird's-eye view (BEV) feature tensor reflecting aggregated features along a vertical dimension (wherein the BEV projection includes the target objects such as vehicles or pedestrians on a horizontal surface (the street) along a vertical dimension (from above)) (Fig. 5c; [0017], [0026], and [0060]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of prior arts to include using the 2D mapping to project a feature tensor to obtain a projected BEV since the results show that the approach significantly improves the performance of object detection while keeping the required communication capacity low compared to sharing raw information methods (Marvasti; [0023]).
Regarding claim 2, Laddah teaches wherein: the set of sensors comprises at least one camera and at least one radar (wherein the one or more sensors include cameras, LIDAR, Radio Detection and Ranging (radar)) ([0025] and [0068]); the input data comprises a set of camera data obtained from the at least one camera (wherein the input data can include image data obtained from at least one camera) (Abstract and [0068]) and a set of radar data obtained from the at least one radar (radar data acquired by a Radio Detection and Ranging (radar) system) (Abstract and [0068]); and the plurality of sets of features comprises a set of camera data features generated from the set of camera data and a set of radar data features generated from the set of radar data (wherein the plurality of sets of features are from camera data features and radar data features, from a camera and radar system respectively) (Abstract, [0027], and [0068]).
Regarding claim 3, Laddah teaches wherein generating the fused BEV grid (and the output domain can include coordinate frame cells; e.g. x, y BEV grid cells) (Fig. 3; [0092]) (wherein the features can be combined to generate fused feature data) ([0097]) further comprises: associating each set of features of the plurality of sets of features with a respective set of points (associating features with a set of points) (Fig. 2B; [0086-0087]); generating, using each set of points, a set of BEV grids, the set of BEV grids comprising the first BEV grid and the second BEV grid (generating a two-dimensional grid comprising a plurality of cells, such as a Birds Eye View (BEV) grid) ([0086]); extracting, for each BEV grid of the set of BEV grids, a respective set of BEV grid features (extracting BEV grid features such as a plurality of radar points from a sequence of sweeps that have been graphically represented as arrows and plotted) (Fig. 2B; [0086-0087]); generating, for each BEV grid of the set of BEV grids using the respective set of BEV grid features (wherein multiple BEV grids are created based on the radar sweeps) ([0031-0032]), a resampled BEV grid, wherein the first BEV grid is associated with a first resampled BEV grid and wherein the second BEV grid is associated with a second resampled BEV grid (generating multiple BEV grids based on a sequence of radar sweeps) ([0031-0036] and [0084-0087]); and fusing each resampled BEV grid to generate the fused BEV grid (fusing sweeps to generate a fused BEV grid) ([0084-0087]). Gomez teaches system and methods and vehicles for object detection using pseudo-LiDAR ([0001]); and wherein the fused BEV grid (combined bird’s eye view map) (Abstract and [0006]) is generated based on a first BEV grid having a first scale (a first bird’s eye view map having a first resolution) ([0006]) and a second BEV grid having a second scale different from the first scale (a second bird’s eye view map having a second resolution (which is smaller or larger than the first resolution) (Abstract, [0006], and [0027]); and wherein the combined bird’s eye view map is used for detecting one or more objects using an object detection algorithm ([0007]).
Regarding claim 4, Laddah teaches wherein associating each set of features of the plurality of sets of features with a respective set of points (associating features with a set of points) (Fig. 2B; [0086-0087]) further comprises: transforming a set of radar features of the plurality of sets of features into a set of radar points (transforming radar features into radar points into a common coordinate frame) ([0031-0035]), including transforming from a polar coordinate representation to a Cartesian coordinate representation (wherein transforming the radar points into a common coordinate frame can include a Cartesian grid) ([0031-0035]).
Laddah teaches wherein the input data can include image data obtained from at least one camera (Abstract and [0068]). However, Laddah does not explicitly teach “transforming a set of camera features of the plurality of sets of features into a set of pixel points”.
Gomez teaches system and methods and vehicles for object detection using pseudo-LiDAR ([0001]); and transforming a set of camera features of the plurality of sets of features into a set of pixel points (wherein a BEV grid has a determined resolution and points lying in inside the defined subspace, from which features are extracted to provide a useful representation that represent each “pixel” in a layer of the resulting bird’s eye view map) ([0023-0027]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Laddah to include “pixel points” since it provides a useful representation (Gomez; [0023]).
Regarding claim 5, Laddah teaches further comprising performing, by the processing device (computing system including processor(s)) ([0050]) using the fused BEV grid (and the output domain can include coordinate frame cells; e.g. x, y BEV grid cells) (Fig. 3; [0092]) (wherein the features can be combined to generate fused feature data) ([0097]), the object detection to identify at least one object (using the fused data to input into a detection layer to detect one or more objects from the fused sensor data) ([0104]) using a set of neural networks (using a set of neural networks) (Figs. 3 and 4; [0104] and [0144]). Gomez also teaches further comprising performing, by the processing device (vehicles and systems) (Abstract) using the fused BEV grid (using the combined BEV maps) (Abstract and [0006]), the object detection to identify at least one object using a set of neural networks (identifying objects using the combined BEV map in an object detection algorithm) ([0008]) (wherein the detection algorithm can be a classifier neural network) ([0003] and [0032]).
Regarding claim 7, Laddah teaches further comprising causing, by the processing device (computing system including processor(s)) ([0050]), a driving path of the AV to be modified (obtaining data from a vehicle, and determine and/or modify one or more states of the vehicle including a path of the vehicle based in part on signals or data exchanged with the vehicle) ([0063]) in view of the at least one object (in view of one or more objects proximate to the vehicle) ([0074]).
Regarding claim 8, see the rejection made to claim 1, as well as prior art Laddah for a system (operations computing system) ([0059]) comprising: a memory (comprising one or more memory devices) ([0059]); and a processing device communicative coupled to the memory (the one or more memory devices can store instructions that when executed by the one or more processors to perform operations) ([0059]), the processing device (one or more processors) ([0059]), for they teach all the limitations within this claim.
Regarding claim 9, see the rejection made to claim 2, as well as prior art Laddah for a system (operations computing system) ([0059]) comprising: a memory (comprising one or more memory devices) ([0059]); and a processing device communicative coupled to the memory (the one or more memory devices can store instructions that when executed by the one or more processors to perform operations) ([0059]), the processing device (one or more processors) ([0059]), for they teach all the limitations within this claim.
Regarding claim 10, see the rejection made to claim 3, as well as prior art Laddah for a system (operations computing system) ([0059]) comprising: a memory (comprising one or more memory devices) ([0059]); and a processing device communicative coupled to the memory (the one or more memory devices can store instructions that when executed by the one or more processors to perform operations) ([0059]), the processing device (one or more processors) ([0059]), for they teach all the limitations within this claim.
Regarding claim 11, see the rejection made to claim 4, as well as prior art Laddah for a system (operations computing system) ([0059]) comprising: a memory (comprising one or more memory devices) ([0059]); and a processing device communicative coupled to the memory (the one or more memory devices can store instructions that when executed by the one or more processors to perform operations) ([0059]), the processing device (one or more processors) ([0059]), for they teach all the limitations within this claim.
Regarding claim 12, see the rejection made to claim 5, as well as prior art Laddah for a system (operations computing system) ([0059]) comprising: a memory (comprising one or more memory devices) ([0059]); and a processing device communicative coupled to the memory (the one or more memory devices can store instructions that when executed by the one or more processors to perform operations) ([0059]), the processing device (one or more processors) ([0059]), for they teach all the limitations within this claim.
Regarding claim 15, see the rejection made to claim 2, as well as prior art Laddah for a non-transitory computer-readable storage medium (non-transitory computer readable media) ([0066]) having instructions stored thereon that (the one or more non-transitory computer readable media can store instructions) ([0066]), when executed by a processing device (when executed by the one or more processors) ([0066]), cause the processing device to perform operations (cause the one or more processors to perform operations and functions) ([0066]), for they teach all the limitations within this claim.
Regarding claim 16, Laddah teaches wherein generating the fused BEV grid (and the output domain can include coordinate frame cells; e.g. x, y BEV grid cells) (Fig. 3; [0092]) (wherein the features can be combined to generate fused feature data) ([0097]) further comprises: associating each set of features of the plurality of sets of features with a respective set of points (associating features with a set of points) (Fig. 2B; [0086-0087]); generating, using each set of points, a set of BEV grids, the set of BEV grids comprising the first BEV grid and the second BEV grid (generating a two-dimensional grid comprising a plurality of cells, such as a Birds Eye View (BEV) grid) ([0086]); extracting, for each BEV grid of the set of BEV grids, a respective set of BEV grid features (extracting BEV grid features such as a plurality of radar points from a sequence of sweeps that have been graphically represented as arrows and plotted) (Fig. 2B; [0086-0087]); generating, for each BEV grid of the set of BEV grids using the respective set of BEV grid features (wherein multiple BEV grids are created based on the radar sweeps) ([0031-0032]), a resampled BEV grid, wherein the first BEV grid is associated with a first resampled BEV grid and wherein the second BEV grid is associated with a second resampled BEV grid (generating multiple BEV grids based on a sequence of radar sweeps) ([0031-0036] and [0084-0087]); and fusing each resampled BEV grid to generate the fused BEV grid (fusing sweeps to generate a fused BEV grid) ([0084-0087]). Gomez teaches system and methods and vehicles for object detection using pseudo-LiDAR ([0001]); and wherein the fused BEV grid (combined bird’s eye view map) (Abstract and [0006]) is generated based on a first BEV grid having a first scale (a first bird’s eye view map having a first resolution) ([0006]) and a second BEV grid having a second scale different from the first scale (a second bird’s eye view map having a second resolution (which is smaller or larger than the first resolution) (Abstract, [0006], and [0027]); and wherein the combined bird’s eye view map is used for detecting one or more objects using an object detection algorithm ([0007]).
Regarding claim 17, Laddah teaches wherein associating each set of features of the plurality of sets of features with a respective set of points (associating features with a set of points) (Fig. 2B; [0086-0087]) further comprises: transforming a set of radar features into a set of radar points (transforming radar features into radar points into a common coordinate frame) ([0031-0035]), including transforming from a polar coordinate representation to a Cartesian coordinate representation (wherein transforming the radar points into a common coordinate frame can include a Cartesian grid) ([0031-0035]).
Laddah teaches wherein the input data can include image data obtained from at least one camera (Abstract and [0068]). However, Laddah does not explicitly teach “transforming a set of camera features into a set of pixel points”.
Gomez teaches system and methods and vehicles for object detection using pseudo-LiDAR ([0001]); and transforming a set of camera features into a set of pixel points (wherein a BEV grid has a determined resolution and points lying in inside the defined subspace, from which features are extracted to provide a useful representation that represent each “pixel” in a layer of the resulting bird’s eye view map) ([0023-0027]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Laddah to include “pixel points” since it provides a useful representation (Gomez; [0023]).
Regarding claim 18, Laddah teaches wherein the operations further comprise performing, using the fused BEV grid (and the output domain can include coordinate frame cells; e.g. x, y BEV grid cells) (Fig. 3; [0092]) (wherein the features can be combined to generate fused feature data) ([0097]), the object detection to identify at least one object (using the fused data to input into a detection layer to detect one or more objects from the fused sensor data) ([0104]) using a set of neural networks (using a set of neural networks) (Figs. 3 and 4; [0104] and [0144]). Gomez also teaches further comprising performing, by the processing device (vehicles and systems) (Abstract) using the fused BEV grid (using the combined BEV maps) (Abstract and [0006]), the object detection to identify at least one object using a set of neural networks (identifying objects using the combined BEV map in an object detection algorithm) ([0008]) (wherein the detection algorithm can be a classifier neural network) ([0003] and [0032]).
Regarding claim 21, Marvasti teaches wherein the horizontal surface corresponds to a ground plane in an AV-centric coordinate frame (the coordinate frame being centric to the autonomous vehicle) (Fig. 5c; Abstract), and the vertical dimension is substantially perpendicular to the ground plane (wherein the BEV projection includes the target objects such as vehicles or pedestrians on a horizontal surface (the street) along a vertical dimension (from above), which is substantially perpendicular to the street) (Fig. 5c; [0017], [0026], and [0060]).
Regarding claim 22, Marvasti teaches wherein performing the 2D mapping (the BEV projector unit is used to project the aligned point-clouds onto a 2D image plane) ([0009] and [0032]) to project the feature tensor a BEV image/tensor is generated based on a plurality of features) ([0009]) onto the horizontal surface (onto the street surface) (Fig. 5c; [0017], [0026], and [0060]) comprises: projecting onto the ground plane represented in the AV-centric coordinate frame (the coordinate frame being centric to the autonomous vehicle) (Fig. 5c; Abstract); and summing or averaging feature values across height bins (wherein the height bins can be defined and changed in response to the dataset to achieve a better results) ([0032] and [0036]) along the vertical dimension to obtain the projected BEV feature tensor (wherein a BEV image/tensor is generated having one or more channels and each channel provides the density of reflected points at a specific height bin) ([0009] and [0032]).
Regarding claim 23, Marvasti teaches wherein the summing or averaging of feature values across height bins (wherein the height bins can be defined and changed in response to the dataset to achieve a better results) ([0032] and [0036]) along the vertical dimension (wherein a BEV image/tensor is generated having one or more channels and each channel provides the density of reflected points at a specific height bin) ([0009] and [0032]) comprises applying elevation-dependent weights to down-weight or suppress non-relevant high-elevation pixels (wherein the image can be padded if the user desires to transmit fixed size feature map in terms of height and width) ([0036]).
Claim(s) 6 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Laddah et al., US 2022/0035376 A1 (Laddah), Gomez et al., US 2022/0036650 A1 (Gomez), Marvasti et al., US 2023/0267720 A1 (Marvasti), and further in view of Kim et al., “Low-level Sensor Fusion for 3D Vehicle Detection using Radar Range-Azimuth Heatmap and Monocular Image” (Kim).
Regarding claim 6, Laddah teaches wherein performing object detection (wherein the detection layer can detect one or more objects) ([0104]) further comprises: obtaining a set of predictions generated using the fused BEV grid (sending each detected object through a prediction head (e.g. a convolutional neural network) to generate trajectory predictions using the fused BEV grid) ([0032], [0037], [0040-0042], and [0104-0106]). Gomez teaches combining BEV maps with different resolutions, the best features for detecting different types of objects may be provided to the object recognition algorithm for detecting an object ([0033] and [0040-0041]). Marvasti teaches a mechanism to further improve object detection performance (Abstract); and wherein a user has access to bounding box information ([0058]).
However, none of them explicitly teach “wherein the set of predictions comprises a heatmap prediction and an attribute prediction; generating, from the set of predictions, a set of candidate bounding boxes, each candidate bounding box of the set of candidate bounding boxes corresponding to the at least one object; and selecting, from the set of candidate bounding boxes, at least one bounding box corresponding to the at least one object”.
Kim teaches to detect 3D vehicles by sensor fusion network using the radar range-azimuth heatmap and image data (p. 2; 2nd paragraph); wherein the set of predictions comprises a heatmap prediction (range-azimuth heatmap) (p. 8; Fig. 5) and an attribute prediction (image feature map from the monocular image) (p. 8; Fig. 5); generating, from the set of predictions, a set of candidate bounding boxes, each candidate bounding box of the set of candidate bounding boxes corresponding to the at least one object (generating from prediction results, a set of bounding boxes corresponding to at least one object; i.e., a vehicle) (p. 2; Fig. 1 and 2nd paragraph); and selecting, from the set of candidate bounding boxes, at least one bounding box corresponding to the at least one object (selecting a bounding box for denoting the vehicle for detection and classification) (pages 12-13; Figs. 7 and 8, and Section 6.2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of prior arts to include using a heatmap along with bounding boxes since it allows for the potential to achieve high performance even with inexpensive radar and camera sensors (Kim; p. 14, Section 7).
Regarding claim 13, see the rejection made to claim 6, as well as prior art Laddah for a system (operations computing system) ([0059]) comprising: a memory (comprising one or more memory devices) ([0059]); and a processing device communicative coupled to the memory (the one or more memory devices can store instructions that when executed by the one or more processors to perform operations) ([0059]), the processing device (one or more processors) ([0059]), for they teach all the limitations within this claim.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Yu et al., “Vehicle detection and localization on bird's eye view elevation images using convolutional neural network”: teaches a convolutional neural network- based vehicle detection and localization method using point cloud data acquired by a LIDAR sensor (Abstract); acquired point clouds are transformed into bird's eye view elevation images, where each pixel represents a grid cell of the horizontal x-y plane (Abstract); and intentionally encode each pixel using three channels, namely the maximal, median and minimal height value of all points within the respective grid (Abstract).
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL J VANCHY JR whose telephone number is (571)270-1193. The examiner can normally be reached Monday - Friday 9am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL J VANCHY JR/Primary Examiner, Art Unit 2666 Michael.Vanchy@uspto.gov