DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 21-38 are pending.
Claims 1-20 are canceled.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 21-38 are rejected on the ground of nonstatutory double patenting as being unpatentable over the claims of U.S. Patent No. US12050660. Although the claims at issue are not identical, they are not patentably distinct from each other because the limitations in the above indicated claims of the instant application are anticipated by the respective claimed limitations in the listed claims of U.S. Patent No. US12050660. See the claim anticipation mapping below.
Instant application
Claims
Patent US12050660
Claims
21
1
33
15
36
20
22
1, 2
23
1, 3
24
1, 4
25
1, 5
26
1, 6
27
1, 7
28
1, 8
29
1, 9
30
1, 10
31
1, 10, 11
32
1, 10, 11, 12
34
15, 16
35
15, 17
37
20, 7
38
20, 2
Note: the contention that the claims of the instant application are anticipated by the claims of US12050660 under ODP is further supported by Prior Art References Urtasun et al (US20200160559A1), Hotson et al (US20210103755A1) and/or Liang et al (US20200025935A1), as cited in the 35 U.S.C. § 103 rejection below.
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claim(s) 21 and 23-37 is/are rejected under 35 U.S.C. 103 as being unpatentable over Urtasun et al (US20200160559A1) in view of Hotson et al (US20210103755A1).
Regarding claims 21, 33 and 36, Urtasun teaches a method, comprising:
generating, using a first neural network and at least one processor, semantic data associated with a first semantic image from a first image of a first image type, the first neural network configured to extract first features corresponding to a plurality of objects in the first image;
(Urtasun, Fig. 2; "a machine-learned image backbone model configured to receive an image of the environment surrounding the autonomous vehicle and to process the image to generate an image feature map", [0005]; "The machine-learned image backbone model can be configured to receive the image(s) and to process the image(s) to generate an image feature map.", [0023]; generating an image feature map (semantic data associated with a first semantic image) from a camera image (first image of a first image type) using a machine-learned image backbone model (first neural network), where the feature map extracts features corresponding to objects in the image for detection)
embedding, using the at least one processor, second image data of a second image of a second image type with the semantic data associated with the first semantic image to form a fused image,
(Urtasun, Fig. 2; "a machine-learned light detection and ranging (LIDAR) backbone model configured to receive a bird's eye view (BEV) representation of a LIDAR point cloud generated for an environment surrounding an autonomous vehicle and to process the BEV representation of the LIDAR point cloud to generate a LIDAR feature map", [0005]; "perform point-wise fusion to fuse the image feature data with one or more intermediate LIDAR feature maps generated by one or more intermediate layers of the machine-learned LIDAR backbone model.", [0007]; receiving a BEV representation of a LIDAR point cloud (second image data of a second image type) and fusing it with the image feature data; Hotson, "The feature maps are input into a convolutional recurrent neural network or other model which uses multiscale convolutions to generate concatenated feature maps 608.", [0053]; Hotson fills the gap regarding explicitly "embedding... to form a fused image" by teaching the concatenation of the feature maps to form a combined/fused image (concatenated feature map))
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the multiscale convolution teachings from Hotson into the Urtasun framework in order to provide a robust motivation to enhance the precision and contextual depth of sensor fusion in autonomous vehicle perception. The combination of Urtasun and Hotson also teaches other enhanced capabilities.
The combination of Urtasun and Hotson further teaches:
wherein the fused image includes at least a portion of the second image data of the second image embedded with a first feature corresponding to at least a portion of the semantic data extracted from the first image;
(Urtasun, "Point-wise feature fusion 220 can be applied to fuse multiscale image features to BEV stream.", [0070]; Hotson, "The concatenated feature map may include features derived from both an image (e.g., RGB image) and a depth map.", [0021]; "The concatenated feature maps 608 may include all the features of the respective sensor feature maps.", [0053]; the fused image (the concatenated feature map) includes the depth/LIDAR features (second image data) embedded side-by-side with the image features (first feature/semantic data extracted from the first image))
generating, using a second neural network, an annotated fused image from the fused image,
(Urtasun, "a machine-learned refinement model configured to receive respective region of interest (ROI) feature crops from each of the LIDAR feature map and the image feature map, to perform ROI-wise fusion to fuse respective pairs of ROI feature crops to generate fused ROI feature crops, and to generate one or more object detections based on the fused ROI feature crops", [0006]; using a second neural network (the machine-learned refinement model) to generate object detections (annotated fused image data) from the fused features)
wherein the annotated fused image comprises feature data extracted from the fused image, the second neural network configured to extract second features corresponding to the plurality of objects in the first image,
(Urtasun, "wherein each of the one or more object detections indicates a location of a detected object within the environment surrounding the autonomous vehicle", [0005]; "For example, the object detection(s) can be provided in the form of a three-dimensional bounding shape (e.g., bounding box).", [0023]; the refinement model extracts features to output bounding shapes and locations corresponding to the detected objects (annotated feature data))
wherein to train the first neural network, at least one network parameter of the first neural network is modified in response to a loss calculated using a particular annotated fused image generated by the second neural network; and
(Urtasun, "The machine-learned LIDAR backbone model, the machine-learned image backbone model, the machine-learned refinement model, and the machine-learned depth completion model have all been jointly trained end-to-end based on a total loss function that evaluates training object detections output by the machine-learned refinement model", [0005]; "At 760, the computing system can backpropagate the loss function through the machine-learned model to train the model (e.g., by modifying one or more weights associated with the model).", [0120]; Urtasun teaches joint, end-to-end training of the backbone network (first neural network) by modifying its weights (network parameters) via backpropagation of a loss function evaluated on the object detections output by the refinement model (the annotated fused image generated by the second neural network))
navigating a vehicle based on the annotated fused image.
(Urtasun, "At 718, method 700 can include determining a motion plan based on the object detections determined by the machine-learned detector model at 716.", [0112]; "At 720, method 700 can include controlling motion of an autonomous vehicle (e.g., vehicle 102 of FIG. 1) based at least in part on the motion plan determined at 718.", [0113]; controlling the motion and navigating the autonomous vehicle based on the object detections generated by the model)
Regarding claim 23, the combination of Urtasun and Hotson teaches its/their respective base claim(s).
The combination further teaches the method of claim 21, wherein the first image is a camera image.
(Urtasun, "The camera(s) can collect one or more images of an environment surrounding the autonomous vehicle", [0113]; "a camera image 208", [0087]; the first image can be an image collected by a camera)
Regarding claims 24 and 34, the combination of Urtasun and Hotson teaches its/their respective base claim(s).
The combination further teaches the method of claim 21, wherein the second image is a 3D lidar point cloud.
(Urtasun, "the LIDAR system can generate a three-dimensional point cloud for the environment (which will simply be referred to as a LIDAR point cloud).", [0022]; the second image is a 3D point cloud generated by a LIDAR system)
Regarding claims 25 and 35, the combination of Urtasun and Hotson teaches its/their respective base claim(s).
The combination further teaches the method of claim 21, wherein the second neural network is a lidar neural network.
(Urtasun, "The machine-learned LIDAR backbone model 202 can be configured to receive a bird's eye view (BEV) representation 212 of a LIDAR point cloud 206", [0068]; utilizing a LIDAR neural network model as a secondary neural network to process the point cloud data)
Regarding claim 26, the combination of Urtasun and Hotson teaches its/their respective base claim(s).
The combination further teaches the method of claim 21, wherein the second neural network is a prediction neural network.
(Urtasun, Fig. 2; “The machine-learned LIDAR backbone model 202 can be configured to receive a bird's eye view (BEV) representation 212 of a LIDAR point cloud 206 for the environment surrounding the autonomous vehicle. The machine-learned LIDAR backbone model 202 can be configured to process the BEV representation 212 of the LIDAR point cloud 206 to generate a LIDAR feature map”, [0068]; “As examples, the machine-learned models 830 can be or can otherwise include various machine-learned models including, for example, neural networks (e.g., deep neural networks)”; the machine-learned LIDAR backbone model 202 can be categorized as a predictive model because it utilizes machine learning to process input data (LIDAR point clouds) and generate structured outputs (LIDAR feature maps) used for downstream prediction tasks; when the model includes a neural network (NN), the NN becomes a predictive neural network)
Regarding claims 27 and 37, the combination of Urtasun and Hotson teaches its/their respective base claim(s).
The combination further teaches the method of claim 21,
wherein the second image is a 3D point cloud,
(Urtasun, "the LIDAR system can generate a three-dimensional point cloud for the environment", [022]; the second image is a 3D point cloud)
wherein embedding the second image with the semantic data comprises:
transforming the 3D point cloud into a bird's-eye view image; and
(Urtasun, "generate a new LIDAR BEV representation 212 (relative to ground), which is fed to the LIDAR backbone model 202.", [0084]; "The resulting 3D volume can be considered as a BEV representation by treating the height slices as feature channels. This allows reasoning within 2D BEV space", [0071]; transforming the 3D point cloud data into a 2D bird's-eye view (BEV) representation/image)
embedding the bird's-eye view image with the semantic data associated with the first semantic image to form the fused image.
(Urtasun, "point-wise feature fusion can be applied to fuse multi-scale image features from the image stream to the BEV stream.", [0031], "fuse image feature map with BEV feature map", [0095]; fusing or embedding the BEV representation with features/data extracted from the primary image stream)
Regarding claim 28, the combination of Urtasun and Hotson teaches its/their respective base claim(s).
The combination further teaches the method of claim 21, wherein the semantic data associated with a pixel of the first semantic image includes at least one feature embedding.
(Urtasun, "the perception system 124 can determine, for each object, state data 130 that describes a current state of such object. As examples, the state data 130 for each object can describe an estimate of the object's: current location (also referred to as position); current speed; current heading (which may also be referred to together as velocity); current acceleration; current orientation; size/footprint (e.g., as represented by a bounding shape such as a bounding polygon or polyhedron); class of characterization (e.g., vehicle class versus pedestrian class versus bicycle class versus other class); yaw rate; and/or other state information.", [0057]; semantic data including feature embeddings like position, orientation, size/footprint per object, which associates with pixels)
Regarding claim 29, the combination of Urtasun and Hotson teaches its/their respective base claim(s).
The combination further teaches the method of claim 21, wherein the feature data associated with the annotated fused image comprises at least one of width, height, and length of an object, bounding box for the object, object movement, object orientation, or object trajectory prediction.
(Urtasun, "For example, the object detection(s) can be provided in the form of a three-dimensional bounding shape (e.g., bounding box).", [0069]; "compute smooth l1 loss on each dimension of the 3D object (x; y; z; log(w); log(l); log(h); θ)", [0102]; the feature data associated with detected objects includes bounding boxes, as well as dimensional metrics such as width (w), length (l), height (h), and orientation (θ))
Regarding claim 30, the combination of Urtasun and Hotson teaches its/their respective base claim(s).
The combination further teaches the method of claim 21, wherein the semantic data associated with a pixel of the first semantic image includes an object classification score.
(Urtasun, "For object classification loss Lcls, binary cross entropy can be used.", [0102]; semantic data including object classification scores via classification loss)
Regarding claim 31, the combination of Urtasun and Hotson teaches its/their respective base claim(s).
The combination further teaches the method of claim 30, wherein the object classification score associates the pixel with a particular object classification from a plurality of object classifications.
(Urtasun, "For object classification loss Lcls, binary cross entropy can be used.", [0102]; "class of characterization (e.g., vehicle class versus pedestrian class versus bicycle class versus other class)", [0057]; classification scores are associating pixels/objects with classifications like vehicle, pedestrian, bicycle)
Regarding claim 32, the combination of Urtasun and Hotson teaches its/their respective base claim(s).
The combination further teaches the method of claim 31, wherein the plurality of object classifications comprises at least one of a car class, a bicycle class, a pedestrian class, a barrier class, a traffic cone class, a drivable surface class, or a background class.
(Urtasun, "class of characterization (e.g., vehicle class versus pedestrian class versus bicycle class versus other class)", [0057]; classifications including car (vehicle), bicycle, pedestrian, background/other)
Claim(s) 22 and 38 is/are rejected under 35 U.S.C. 103 as being unpatentable over Urtasun et al (US20200160559A1) in view of Hotson et al (US20210103755A1) and further in view of Liang et al (US20200025935A1).
Regarding claims 22 and 38, the combination of Urtasun and Hotson teaches its/their respective base claim(s).
The combination does not expressly disclose but Liang teaches the method of claim 21, wherein the second image is a 3D point cloud, the method further comprising:
(Liang, "For example, a LIDAR system can be configured to capture LIDAR data (e.g., 3D LIDAR point cloud data associated with an environment surrounding an autonomous vehicle).", [0038]; the second image is a 3D point cloud.)
receiving map data associated with a map;
(Liang, "the autonomy computing system 120 can retrieve or otherwise obtain data including the map data 122. The map data 122 can provide detailed information about the surrounding environment of the vehicle 102", [0082]; receiving map data associated with a map)
determining a location of the 3D point cloud within the map; and
(Liang, "When real-time location sensors within the autonomous vehicle determine a current geographic location of the autonomous vehicle, geographic prior data (e.g., geometric ground prior data and/or semantic road prior data) associated with that current geographic location can be retrieved from the map system 206", [0095]; "Specifically, given a LIDAR point cloud (xi, yi, zi), we query the point at location (xi, yi) from the HD map, denoted as (xi, yi, zi 0)", [0122]; querying and determining the location of the 3D point cloud within the map to access associated map data)
embedding 3D point cloud data with annotations associated with the map based on the location of the 3D point cloud within the map to form the fused image.
(Liang, "the semantic road region mask can be extracted from the HD map as a prior knowledge of the scene, which could potentially help guide the network to focus on important regions. Specifically, road layout information can be extracted from HD maps as polygon and rasterize it onto the bird's eye view as a binary road mask channel at the same resolution as the LIDAR 3D grid. The road mask channel can be concatenated together with the LIDAR 3D grid along the Z axis", [0124]; extracting mask annotations associated with the map based on the location and embedding (rasterizing and concatenating) them with the 3D point cloud data to form a fused output)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate Liang into the multi-sensor object detection framework of Urtasun and Hotson in order to embed geographic prior data, such as semantic road masks and geometric ground features from high-definition (HD) maps, directly into the Bird's-Eye View (BEV) LiDAR representation, thereby eliminating translation variance along the Z-axis and providing the neural network with explicit drivable-area spatial context to significantly reduce false positive detections outside of the relevant roadway. The combination of Urtasun, Hotson and Liang also teaches other enhanced capabilities.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANXUN YANG whose telephone number is (571)272-9874. The examiner can normally be reached on MON-FRI: 8AM-5PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, Applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JIANXUN YANG/
Primary Examiner, Art Unit 2662 5/30/2026
Prosecution Insights