DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s)1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Mei et al. (2019/0163990) in view of Urtasun et al (US 2020/0160559)
As to claim 1, Mei et al teaches a computer-implemented method for detecting objects an environment, the method comprising:
generating LIDAR feature data (figure 2; paragraphs 0071 and 0012) paragraph [0011-0012]; figure 2; paragraphs 0071) using a machine-learned LIDAR processing model that processes an input LIDAR point cloud( figure 2; paragraphs [O071, 0012 and 0034]): generating image feature data using a machine-learned image processing model that processes input image data; (generating a lane marking map from the set of lane marking points (processing block 1040); paragraph [0041][0065]); fusing at least a portion of the LIDAR feature data and at least a portion of the image feature data to obtain fused feature data (fusion module 250 as shown in FIG. 3, paragraph [0049-0054]); generating, using the fused feature data, an initial object detection estimate; (paragraph [0064-0065]; figure 3-5). While Mei et al. meets a number of the limitations of the claimed invention, as pointed out more fully above, Mei et al fails to specifically teach” obtaining, using the initial object detection estimate, feature data describing a region of interest associated with the initial object detection estimate, wherein the feature data is selected from one or more of: the LIDAR feature data, the image feature data, and the fused feature data; and generating a refined object detection using a machine-learned refinement model that processes the feature data describing the region of interest.”
Specifically, Urtasun et al et al. teaches end-to-end learnable architecture with multiple machine-learned models that interoperate to reason about 2D and/or 3D object detection as well as one or more auxiliary tasks. According to another aspect of the present disclosure, example systems and methods described herein can perform multi-sensor fusion (e.g., fusing features derived from image data, light detection and ranging (LIDAR) data, and/or other sensor modalities) at both the point-wise and region of interest (ROI)-wise level, resulting in fully fused feature representations ( abstract). Uratsun et al teaches the machine-learned refinement model 205 can be configured to receive respective region of interest (ROI) feature crops from each of the LIDAR feature map and the image feature map, to perform ROI-wise fusion to fuse respective pairs of ROI feature crops to generate fused ROI feature crops, and to generate one or more three-dimensional object detections based on the fused ROI feature crops. Each of the one or more three-dimensional object detections can indicate a location of a detected object within the environment surrounding the autonomous vehicle ( paragraph [0069-0071]).It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to jointly train the machine-learned LIDAR backbone model, the machine-learned image backbone model, the machine-learned refinement model, and the machine-learned mapping model in order to enable improved performance of the autonomous vehicle with regard to control, comfort, safety, fuel efficiency, and/or other metrics.. Therefore, the claimed invention would have been obvious to one of ordinary skill in the art at the time of the invention by applicant.
As to claim 2, Urtasun et al teaches computer-implemented method of claim 1, wherein fusing at least the portion of the LIDAR feature data and at least the portion of the image feature data comprises: performing point-wise feature fusion to fuse at least the portion of the LIDAR feature data and at least the portion of the image feature data (Point-wise and ROI-wise feature fusion can both be applied to achieve full multi-sensor fusion, while multi-task learning provides additional map prior and geometric clues enabling better representation learning and denser feature fusion., paragraph [0032]).
As to claim 3, Urtasun et al teaches the computer-implemented method of claim 1, wherein the machine-learned refinement model comprises one or more fully connected layers ( the multi-sensor ROI features can be fused together and fed into the refinement model 205 with, as an example, two 256-dimension fully connected layers to predict the 2D and 3D box refinements for each 3D detection respectively, paragraph [0075]).
As to claim 4, Urtasun et al teaches the computer-implemented method of claim 1, wherein the initial object detection estimate comprises an initial bounding box for an object, and wherein the refined object detection comprises a refined bounding box for the object (the full model ensemble outputs object classification, 3D box estimation, 2D and 3D box refinement, ground estimation and dense depth, paragraph [0101]).
As to claim 5, Urtasun et al teaches the computer-implemented method of claim 1, wherein the feature data describing the region of interest is obtained from an output layer of the machine-learned LIDAR processing model or an output layer of the machine-learned image processing model( Both retrieved image feature and the BEV geometric feature can be passed into a Multi-Layer Perceptron (MLP) and the output can be fused to BEV feature map by element-wise addition. Note that such point-wise feature fusion is sparse by nature of LIDAR observation, paragraph [0095][0101]).
As to claim 6, Urtasun et al teaches the computer-implemented method of claim 1, wherein generating the refined object detection comprises: combining at least a portion of the image feature data and at least a portion of the LIDAR feature data; and processing the combined portions of the image feature data and the LIDAR feature data using the machine-learned refinement model to generate the refined object detection ( multi-sensor detector can take a LIDAR point cloud 206 and an RGB image 208 as input. The backbone models 202 and 204 form a two-stream structure, where one stream extracts image feature maps, and the other extracts LIDAR BEV feature maps. Point-wise feature fusion 220 can be applied to fuse multiscale image features to BEV stream. The final BEV feature map predicts, in some implementations, dense 3D detections per BEV voxel via 2D convolution. In some implementations, after Non-Maximum Suppression (NMS) and score thresholding, the output of the system can be a small number of high-quality 3D detections and their projected 2D detections. In some implementations, a 2D and 3D box refinement by ROI-wise feature fusion can be applied, where features from both image ROIs and BEV oriented ROIs are combined. After the refinement, the system can output accurate 2D and 3D detections, paragraph [0070]).
As to claim 7, Urtasun et al teaches the computer-implemented method of claim 1, wherein generating the refined object detection comprises: regressing, using the machine-learned refinement model, a relative position refinement (FIG. 4 depicts precise rotated ROI feature extraction that takes orientation cycle into account. In particular, FIG. 4 illustrates, at (1), the rotational periodicity causes reverse of order in feature extraction and at (2), an ROI refine module with two orientation anchors. In some implementations, an ROI can be assigned to 0 or 90 degrees. They share most refining layers except for the output. At (3), FIG. 4 depicts the regression target of relative offsets are re-parametrized with respect to the object orientation axes and at (4) a n×n sized feature is extracted using bilinear interpolation (an example is shown with n=2).
As to claim 8, Urtasun et al teaches the computer-implemented method of claim 1, wherein generating the refined object detection comprises: generating, using a first output layer of the machine-learned refinement model, a refined two-dimensional bounding box associated with the object; and generating, using a second output layer of the machine-learned refinement model, a refined three-dimensional bounding box associated with the object( Referring again to FIG. 2, the illustrated ensemble of models can, in some implementations, be trained jointly and end-to-end. In particular, a multi-task loss can be employed to train the multi-sensor detector end-to-end. In some implementations, the full model ensemble outputs object classification, 3D box estimation, 2D and 3D box refinement, ground estimation and dense depth, paragraph [0097-0101]).
As to claim 9, Urtasun et al teaches the computer-implemented method of claim 1, wherein the machine-learned LIDAR processing model, the machine-learned image processing model, and the machine-learned refinement model were jointly trained end-to-end based on a total loss function that evaluates training object detections output by the machine-learned refinement model (The machine-learned mapping model 216 can be configured to receive the LIDAR point cloud 206 and to process the LIDAR point cloud 206 to generate a ground geometry prediction 218 that describes locations of a ground of the environment within the LIDAR point cloud 206. As indicated above, the machine-learned mapping model 216 can be jointly trained end-to-end with the other models included in the object detection system, all of the models in the illustrated architecture can be jointly trained (e.g., in an end-to-end fashion) using a total loss function that includes multiple different loss function components, where each of the loss function components evaluates the architecture's performance on one of the multiple different tasks/objectives, paragraph [0076-0078].
As to claim 10, Urtasun et al teaches the computer-implemented method of claim 1, wherein the machine-learned image processing model comprises one or more pre-trained image processing layers (Some example implementations use the pre-trained ResNet-18 network to initialize the image backbone model 204, paragraph [0103]).
The limitation of claim 11-20 has been addressed above.
Cited References
Huval (US Patent Pub. No. 2019/0243372 A1) discloses a method for calculating nominal paths for lanes within a geographic region includes: serving a digital frame of a road segment to an annotation portal; at the annotation portal, receiving insertion of a lane marker label, for a lane marker represented in the digital frame, over the digital frame; calculating a nominal path over the road segment and defining a virtual simulator environment for the road segment based on the lane marker label; during a simulation, traversing the virtual road vehicle along the nominal path within the virtual simulator environment and scanning the virtual simulator environment for collisions between the virtual road vehicle and virtual objects within the virtual simulator environment; and, in response to absence of a collision between the virtual road vehicle and virtual objects in the virtual simulator environment, updating a navigation map for the road segment with the nominal path.
Kungaspunta (US Patent Pub. No. 2019/0272446 A1) teaches a system configured to automatically create training datasets for training a segmentation model to recognize features such as lanes on a road. The system may receive sensor data representative of a portion of an environment and map data from a map data store including existing map data for the portion of the environment that includes features present in that portion of the environment. The system may project or overlay the features onto the sensor data to create training datasets for training the segmentation model, which may be a neural network. The training datasets may be communicated to the segmentation model to train the segmentation model to segment data associated with similar features present in different sensor data. The trained segmentation model may be used to update the map data store, and may be used to segment sensor data obtained from other portions of the environment, such as portions not previously mapped.
Singhal (US Patent Pub. No. 2018/0141562 A1) teaches systems of an electrical vehicle and the operations thereof used to select autonomous vehicle operations, including acceleration rate, deceleration rate, steering angle, and inter-vehicle spacing.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NANCY BITAR whose telephone number is (571)270-1041. The examiner can normally be reached Mon-Friday from 8:00 am to 5:00 p.m..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ms. Jennifer Mehmood can be reached at 571-272-2976The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
NANCY . BITAR
Examiner
Art Unit 2664
/NANCY BITAR/Primary Examiner, Art Unit 2664
Prosecution Insights