Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Remarks
Claim Rejections – 35 U.S.C. 101
Applicant’s amendments have been fully considered and they are persuasive.
The rejection of claims 1-20 under 35 U.S.C. 101 has been withdrawn.
Claim Rejections – 35 U.S.C. 103
Applicant’s prior art arguments have been fully considered and they are only partially persuasive.
Applicant argues (pg. 13) that Xu does not disclose a confidence in the neural network model.
Examiner respectfully disagrees. As Applicant states, Xu describes voxel-wise confidence estimations which are then used to weight terms within its loss functions. Since this loss function is used for training the neural network, it indeed discloses a confidence in the neural network. Examiner suggests specifying the language of the claim.
Applicant argues (pg. 13) that Xu does not disclose a dynamic input resolution for voxelized frames because the frames are allegedly fixed in size.
Examiner agrees. Accordingly, a new reference, Su (“DV-Det: Efficient 3D Point Cloud Object Detection with Dynamic Voxelization”), has been added to the rejection, as further detailed below.
Applicant argues (pg. 14) that Xu2 does not disclose operating an AV to navigate an environment by deploying a specifically trained model for real-time object detection that subsequently controls AV navigation. Instead, Xu2 is allegedly focused on map reliability and estimation.
Examiner respectfully disagrees. Xu2 ¶ [0074]: “using the estimated map data, an operation of the vehicle. For example, the estimated map data 532 may be compared to stored map data and/or trajectory data to assess a reliability and determine whether to traverse along a proposed trajectory.” Xu2 teaches that the map data, with the detected objects, is used to determine the navigation of the AV, as the decision of whether or not to traverse a trajectory is determined. Xu2 ¶ [0031]: “machine learned model that receives various inputs and, based on being trained, outputs the proposed vehicle trajectory.” Xu2 teaches that the model can be trained and output the proposed vehicle trajectory, which is a navigation of the AV.
The remaining arguments are such that Xu2 does not teach the alleged deficiencies of Xu. As argued above and as can be shown in the rejection below, Xu teaches these deficiencies.
The foregoing applies to all independent claims and their dependent claims.
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 5, 10, 12, 14, 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Xu et al. (“SelfVoxeLO: Self-supervised LiDAR Odometry with Voxel-based Deep Neural Networks”) hereinafter known as Xu in view of Xu et al. (US 20220326023 A1) hereinafter known as Xu2 in view of Su et al. (“DV-Det: Efficient 3D Point Cloud Object Detection with Dynamic Voxelization”).
Regarding independent claim 1, Xu teaches:
A method of operating an autonomous vehicle (AV) to navigate an environment, comprising: deploying, at the AV, a neural network model, the neural network model trained by a training process comprising: receiving a 3D light and detection ranging (LIDAR) data to train the neural network model … for detecting objects in LIDAR data; (Xu [Page 1, Paragraph 2]: “Ego-motion estimation from temporal sequences of sensor data, also known as odometry, is of fundamental importance for many robotic vision tasks, including navigation, mapping, virtual/augmented reality, etc. Compared with visual sensors, the LiDAR can capture richer 3D geometric information of the environments and is robust against varying lighting conditions.” Xu teaches receiving LIDAR data for odometry. Because odometry is the measurement of the change in position over time, Xu teaches the detection of objects as objects must be detected for their position to be tracked and measured. Xu [Page 2, Paragraph 2]: “Then, the extracted features are fed into our odometry regression network to predict the final 6-DOF ego-motions” Xu teaches that the LIDAR data is used to train the neural network model for detecting objects.)
converting each frame of the LIDAR data into a voxelized frame to yield a training dataset of voxelized frames; (Xu [Page 2, Paragraph 3]: “Specifically, our network first voxelizes the point clouds into fine-grid voxel cells, and extracts 3D features via 3D convolutional neural networks. Then, the extracted features are fed into our odometry regression network to predict the final 6-DOF ego-motions.” Xu teaches converting each point cloud frame into a voxelized frame. Xu also teaches that this data, when it is formatted properly into features, is provided to the neural network as the training data.)
and training the neural network model based on the training dataset of voxelized frames and a feedback control the feedback control indicating a current confidence in the neural network model and used to: (Xu [Page 2, Paragraph 2]: “our network first voxelizes the point clouds into fine-grid voxel cells, and extracts 3D features via 3D convolutional neural networks. Then, the extracted features are fed into our odometry regression network to predict the final 6-DOF ego-motions … we also propose to estimate the correspondence pair confidences” Xu teaches that the training is based on the voxelized frames, with confidence intervals in the training.)
… before the resized voxelized frames are input into the neural network model during the training; (Xu [Page 2, Paragraph 2]: “our network first voxelizes the point clouds into fine-grid voxel cells, and extracts 3D features via 3D convolutional neural networks. Then, the extracted features are fed into our odometry regression network to predict the final 6-DOF ego-motions” Xu teaches that the training is based on the voxelized frames.)
operating a 3D LIDAR detector disposed at the AV to capture 3D LIDAR data representing a surrounding environment of the AV; (Xu [Page 8, Paragraph 4]: “It is worth mentioning that our method can run in real-time and can be combined with other off-the-shelf mapping algorithms for deployment in practical applications.” Xu teaches that the object detection can be performed in real time in practical settings; in particular – as seen with the Apollo dataset, in the context of an autonomous vehicle in real time.)
operating the deployed neural network model to process the captured 3D LIDAR data to perform real-time detection of objects proximate to the AV; (Xu [Page 8, Paragraph 4]: “It is worth mentioning that our method can run in real-time and can be combined with other off-the-shelf mapping algorithms for deployment in practical applications.” Xu teaches that the object detection can be performed in real time in practical settings; in particular – as seen with the Apollo dataset, in the context of an autonomous vehicle in real time.)
Xu does not teach:
… having residual connections …
…
…
…
…
…
and controlling navigation of the AV through the environment based at least in part on the detected objects
However, Xu2 teaches:
… having residual connections … (Xu2 ¶[0020]: “In addition, a lidar data converter 109b may receive the lidar data 110 and process the lidar data 110, for example, using one or more machine learned models (e.g., using a neural network, such as a residual neural network).” Xu 2 teaches receiving LIDAR data to train the neural network, which may have residual connections.)
…
…
…
…
…
and controlling navigation of the AV through the environment based at least in part on the detected objects (Xu2 ¶ [0074]: “using the estimated map data, an operation of the vehicle. For example, the estimated map data 532 may be compared to stored map data and/or trajectory data to assess a reliability and determine whether to traverse along a proposed trajectory.” Xu2 teaches that the map data, with the detected objects, is used to determine the navigation of the AV, as the decision of whether or not to traverse a trajectory is determined. Xu2 ¶ [0031]: “machine learned model that receives various inputs and, based on being trained, outputs the proposed vehicle trajectory.” Xu2 teaches that the model can be trained and output the proposed vehicle trajectory, which is a navigation of the AV.)
Xu and Xu2 are in the same field of endeavor as the present invention, as the
references are directed to training neural networks for image processing in the context of autonomous vehicles. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine converting LIDAR data into voxels to use for the training data as taught in Xu with using a residual neural network to create the model as taught in Xu2. Xu2 provides this additional functionality. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Xu to include teachings of Xu2 because the combination would allow for the neural network to skip layers during training the model. This has the potential benefit of increasing the accuracy of the model by connecting layers that are most appropriate for connection.
Xu and Xu2 do not explicitly teach:
… dynamically determine an input resolution for voxelized frames from the training dataset;
and resize the voxelized frames according to the determined dynamic input resolution …
However, Su teaches:
… dynamically determine an input resolution for voxelized frames from the training dataset; (Su [Page 6, Paragraph 7]: “Specifically, given a 3D RoI of dimension W ×L× H and the target pooling grid resolution k” Su teaches that the target pooling grid resolution is a variable k that can be given dynamically.)
and resize the voxelized frames according to the determined dynamic input resolution … (Su [Page 6, Paragraph 7]: “fit S_1^i into the 3D regular voxels with resolution k × k × k” Su teaches that the target pooling grid resolution is a variable k that can be given dynamically and that the voxels are resized to fit this resolution.)
Su is in the same field as the present invention, since it is directed to dynamically voxelizing frames. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine training the neural network using voxelized frames as taught in Xu as modified by Xu2 with dynamically voxelizing the frames as taught in Su. Su provides this additional functionality. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Xu as modified by Xu2 to include teachings of Su because the combination would allow for the voxelization of frames to be done on the fly. This has the potential benefit of speeding up the voxelization, as the process of downsampling and creating the voxels can be done dynamically, without needing to store large amounts of overhead memory.
Regarding dependent claim 3, Xu and Xu2 teach:
The method of claim 1,
Xu teaches:
wherein the neural network model is implemented in a perception stack in the AV for detecting objects proximate to the AV. (Xu [Page 8, Paragraph 1]: “The Apollo dataset is closer to actual autonomous driving scenarios with more moving objects.” Xu teaches that the neural network model has significant generality, as it can be used on a dataset with autonomous driving and moving objects proximate to the autonomous vehicle.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 5, Xu and Xu2 teach:
The method of claim 1,
Xu teaches:
wherein a number of frames in the 3D LIDAR data is equal to a number of frames in the training dataset of voxelized frames. (Xu [Page 4, Paragraph 2]: “Then, the points … in the point cloud P are dispensed to the respective cells and we refer to these cells as voxels in the following.” Xu teaches that each point cloud frame is converted to a voxelized frame. Therefore, there is a 1 to 1 correspondence between point cloud frames and voxelized frames.)
The reasons to combine are substantially similar to those of claim 1.
Claim 10 is substantially similar to claim 1, but has the additional elements:
Regarding independent claim 10, Xu2 teaches:
A system comprising: a storage configured to store instructions; (Xu2 ¶[0101]: “Memory 618 and 638 can store an operating system and one or more software applications, instructions” Xu teaches storage that can store instructions.)
a processor configured to execute the instructions and cause the processor to: (Xu2 ¶[0100]: “The processor(s) 616 of the vehicle 602 and the processor(s) 636 of the computing device(s) 634 can be any suitable processor capable of executing instructions” Xu teaches a processor that can execute instructions.)
The reasons to combine are substantially similar to those of claim 1.
Claims 12, 14 are rejected on the same grounds under 35 U.S.C. 103 as claims 3, 5 as they are substantially similar, respectively. Mutatis mutandis.
Claim 19 is substantially similar to claim 1, but has the additional elements:
Regarding independent claim 19, Xu2 teaches:
A non-transitory computer readable medium comprising instructions, (Xu2 ¶[0101]: “Memory 618 and 638 are examples of non-transitory computer-readable media. Memory 618 and 638 can store an operating system and one or more software applications, instructions” Xu teaches non-transitory storage that can store instructions.)
the instructions, when executed by a computing system, cause the computing system to: (Xu2 ¶[0100]: “The processor(s) 616 of the vehicle 602 and the processor(s) 636 of the computing device(s) 634 can be any suitable processor capable of executing instructions” Xu teaches a processor that can execute instructions.)
The reasons to combine are substantially similar to those of claim 1.
Claim 20 is rejected on the same grounds under 35 U.S.C. 103 as claim 3 as they are substantially similar. Mutatis mutandis.
Claims 2, 11 are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of Xu2 in view of He et al. (“Deep Residual Learning for Image Recognition”) hereinafter known as He.
Regarding dependent claim 2, Xu and Xu2 teach:
The method of claim 1,
Xu and Xu2 do not explicitly teach:
wherein the neural network model comprises a 23-layer residual neural network model and the residual connections skip at least one layer of the 23-layer residual neural network model.
However, He teaches:
wherein the neural network model comprises a 23-layer residual neural network model and the residual connections skip at least one layer of the 23-layer residual neural network model. (He [Page 5, Table 1] He teaches residual neural networks 34 layers. Therefore, He teaches 23 layers, as layers of a neural network are modular. He [Page 2, Paragraph 3] “Shortcut connections [2, 34, 49] are those skipping one or more layers.” He teaches shortcut connections that skip one or more layers of the residual neural network.)
He is in the same field as the present invention, since it is directed to deep residual learning for image recognition. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine using LIDAR data to detect objects as taught in Xu as modified by Xu2 with using a deep residual neural network as taught in He. He provides this additional functionality. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Xu as modified by Xu2 to include teachings of He because the combination would allow for more layers to be implemented into the neural network model. This has the potential benefit of converging on a model, as the additional layers allow the model to be more accurately trained.
Claim 11 is rejected on the same grounds under 35 U.S.C. 103 as claim 2 as they are substantially similar. Mutatis mutandis.
Claims 6, 7, 15, 16, are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of Xu2 in view of Su in view of Dixit (“Max Pooling, Why use it and its advantages” https://medium.com/geekculture/max-pooling-why-use-it-and-its-advantages-5807a0190459) hereinafter known as Dixit.
Regarding dependent claim 6, Xu and Xu2 teach:
The method of claim 5,
Xu teaches:
wherein training the neural network model based on the training dataset of voxelized frames and the feedback control to control input from the training dataset of voxelized frames into the neural network model comprises: … to resize the voxelized frame based on the dynamic input resolution; (Xu [Page 3, Figure 2]: “The input LiDAR point clouds are voxelized and then fed to the 3D geometric feature encoding module, where the points are pre-downsampled… the confidence estimation module predict the ego-motion (Rpred , t pred) and the voxel-wise confidence estimations” Xu teaches a pre-downsampling factor, or a dynamic resolution, that is based on the confidence estimations, that is used to resize the voxelized frame.)
Xu and Xu2 do not explicitly teach:
controlling a max pool layer based on a stride size …
However, Dixit teaches:
controlling a max pool layer based on a stride size … (Dixit [Page 2, Figure 1]: Dixit teaches a max pool of a stride of 2.)
Dixit is in the same field as the present invention, since it is directed to using max pooling with a stride size to create an output tensor. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine using LIDAR data to detect objects as taught in Xu as modified by Xu2 as modified by Su with using max pooling with a stride size as taught in Dixit. Dixit provides this additional functionality. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Xu as modified by Xu2 as modified by Su to include teachings of Dixit because the combination would allow for the downsizing of a frame in the dataset. This has the potential benefit of speeding up the computations in the neural network, as the lower resolution of the data would be quicker to calculate.
Regarding dependent claim 7, Xu, Xu2, and Dixit teach:
The method of claim 6,
Dixit teaches:
wherein the max pool layer … (Dixit [Page 2, Figure 1]: Dixit teaches a max pool of a stride of 2.)
Xu teaches:
… is part of a training module executed in a distributed system that provides the resized voxelized frame into the neural network model. (Xu [Page 3, Figure 2]: “The input LiDAR point clouds are voxelized and then fed to the 3D geometric feature encoding module, where the points are pre-downsampled… the confidence estimation module predict the ego-motion (Rpred , t pred) and the voxel-wise confidence estimations… The sub-figure (a) and (b) demonstrate our spherical reprojection loss and transformation residual loss, which are elaborated in Sec. 3.2.” Xu teaches from the calculated loss that the resized voxelized frame is input to the neural network to eventually get the estimated error. Note that the training module that provides this can use pooling techniques other than max pool (as described by specs [0066] of present invention) and thus is able to be combined with Dixit’s teachings of max pool.)
The reasons to combine are substantially similar to those of claim 6.
Claims 15-16 are rejected on the same grounds under 35 U.S.C. 103 as claims 6-7 as they are substantially similar, respectively. Mutatis mutandis.
Claims 8, 9, 17, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of Xu2 in view of Dixit in view of Ioffe et al. (“Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”) hereinafter known as Ioffe.
Regarding dependent claim 8, Xu, Xu2, and Dixit teach:
The method of claim 7,
Xu teaches:
… features extracted from a frame of the training dataset of voxelized frames… (Xu [Page 2, Paragraph 3]: “Specifically, our network first voxelizes the point clouds into fine-grid voxel cells, and extracts 3D features via 3D convolutional neural networks.” Xu teaches extracting the features from the voxelized frames, that were originally from the LIDAR data.)
Xu, Xu2, and Dixit do not explicitly teach:
wherein the neural network model is trained by the distributed system without synchronized batch normalization, wherein each node is the distributed system normalizes … to reduce internal covariate shift.
However, Ioffe teaches:
wherein the neural network model is trained by the distributed system without synchronized batch normalization, wherein each node is the distributed system normalizes … to reduce internal covariate shift. (Ioffe [Page 2, Paragraph 3]: “Batch Normalization, that takes a step towards reducing internal covariate shift, and in doing so dramatically accelerates the training of deep neural nets. It accomplishes this via a normalization step that fixes the means and variances of layer inputs,” Ioffe teaches a distributed system that does not use synchronous batch normalization; instead, it uses standard batch normalization, which only normalizes the data within the local system. This has the additional effect of reducing the internal covariate shift.)
Ioffe is in the same field as the present invention, since it is directed to performing some type of normalization to reduce internal covariate shift. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine using LIDAR data to detect objects as taught in Xu as modified by Xu2 as modified by Dixit with using batch normalization to reduce internal covariate shift as taught in Ioffe. Ioffe provides this additional functionality. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Xu as modified by Xu2 as modified by Dixit to include teachings of Ioffe because the combination would allow for reducing internal covariate shift. This has the potential benefit of improving the training stability and convergence speed of the neural network model.
Regarding dependent claim 9, Xu, Xu2, Dixit, and Ioffe teach:
The method of claim 8,
Xu teaches:
wherein a first voxelized frame in the training dataset has a first resolution and a second voxelized frame in the training dataset has a second resolution that is different from the first resolution. (Xu [Page 4, Paragraph 2]: “divide the space into equal-sized cells c^i with sizes of (D/n_D, W/n_W , H/n_H)” Xu teaches that the resolution is indeed dynamic as the factor that is being used to divide the space are the components of the variable n. Since n represents a given points coordinate, and are dynamic because each point will change position between point cloud frames during motion, the resolution is also dynamic. Note that different frames have different resolutions as well because the points that make up the frames have different location points.)
The reasons to combine are substantially similar to those of claim 8.
Claims 17-18 are rejected on the same grounds under 35 U.S.C. 103 as claims 8-9 as they are substantially similar, respectively. Mutatis mutandis.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYU HYUNG HAN whose telephone number is (703) 756-5529. The examiner can normally be reached on MF 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached on (571) 270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Kyu Hyung Han/
Examiner
Art Unit 4114
/BEN M RIFKIN/Primary Examiner, Art Unit 2123