DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03/06/2026 has been entered.
Claims 1-2, 5-11 and 13-15 are currently pending and examined below. Claims 1, 10 and 15 have been amended.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 03/06/2026 and 05/05/2026 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements have been considered by the examiner.
Response to Amendment
Applicant's arguments filed 03/06/2026 have been fully considered but they are not persuasive.
In particular, in pages 5-6 of the Applicant’s Argument, the Applicant argues that with respect to Claim 1 “Voodarla merely teaches how to determine the location coordinates of the camera (x and y coordinates) and the rotation around the z-axis based on the stereo images. The aim of Voodarla is to locate the vehicle on a map, but not to modify information from the vehicle's odometry unit by using translational and rotational motion information provided by a neural network based on stereo images from a stereo camera.”
The Examiner respectfully disagrees. Voodarla discloses in [0078]-[0080] and [0085] to use a variational autoencoder (VAE), a neural network, to encode stereo image pairs into latent variables that represent motion and derive visual odometry including translation/rotation in global coordinates of the vehicle, disclosing odometry data modified by using translational and rotational motion information provided by a neural network based on stereo images from a stereo camera.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 5, 8, 10-11 and 15 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Voodarla et al. (US 20220180106 A1; hereinafter Voodarla).
Regarding claim 1, Voodarla discloses:
A method ([0049] operation of second computer 150 of the mapping vehicle 145) for determining information about ego-motion of a vehicle ([0071] mapping vehicle 145)([0033] Determining a path can include solving a localization problem. Localization includes determining a three degree-of-freedom (DoF) pose for the vehicle with respect to a map of the environment around the vehicle, [0078] Stereo visual odometry is a technique for determining a three DoF (3DOF) pose 622 for the mapping vehicle 145),
wherein the vehicle includes a stereo camera system with at least two cameras for capturing stereo images ([0071] pairs of stereo images 602) of an area surrounding the vehicle ([0071] pairs of stereo images 602 acquired as a mapping vehicle 145 equipped with stereo video sensors travels along a roadway 204, 206 to be mapped), and an artificial neural network ([0078] variational autoencoder (VAE)) for processing image information provided by the stereo camera system ([0078] VAE can be trained to input image data),
wherein, during movement of the vehicle, the stereo camera system captures, by the at least two cameras, image sequences which contain a plurality of image information at different points in time ([0071] pairs of stereo images 602 acquired as a mapping vehicle 145 equipped with stereo video sensors travels along a roadway 204, 206 to be mapped, [0079] The VAE determines corresponding feature points in sequential images and calculates the change in location of the sensor between images),
wherein the artificial neural network receives the image information ([0078] VAE can be trained to input image data) and generates based on said image information stereo images with distance information ([0078] Stereo visual odometry is a technique for determining a three DoF (3DOF) pose 622 for the mapping vehicle 145 based on determining changes in the locations of feature points extracted from the images as the mapping vehicle 145 moves through a scene),
wherein, based on the image information of the stereo camera system, the artificial neural network provides odometry data at an output interface ([0078] Visual odometry can be performed by a trained variational autoencoder (VAE)),
wherein the odometry data comprises information regarding the translational motion of the vehicle along three spatial axes of a Cartesian coordinate system ([0080] “Visual odometry processor 608 determines three DoF poses 622 based on a plurality of pairs of stereo images 602 acquired as the mapping vehicle 145 travels along the path to be topologically mapped. The three DoF pose 622 locates the mapping vehicle 145 with respect to global coordinates.”), namely the longitudinal, vertical, and lateral velocity of the vehicle ([0085] “the vehicle motion vector can include positions in x, y, z, yaw, pitch, roll, yaw rate, pitch rate, roll rate, heading velocity and heading acceleration that can be determined by fitting a polynomial function to successive 2D locations included in the vehicle motion vector with respect to the ground surface, for example”), and
wherein the odometry data further comprises information about the rotational motion of the vehicle about the three spatial axes of the Cartesian coordinate system ([0080] “Visual odometry processor 608 determines three DoF poses 622 based on a plurality of pairs of stereo images 602 acquired as the mapping vehicle 145 travels along the path to be topologically mapped. The three DoF pose 622 locates the mapping vehicle 145 with respect to global coordinates.”), namely the pitch, yaw and roll speed of the vehicle ([0085] “the vehicle motion vector can include positions in x, y, z, yaw, pitch, roll, yaw rate, pitch rate, roll rate, heading velocity and heading acceleration that can be determined by fitting a polynomial function to successive 2D locations included in the vehicle motion vector with respect to the ground surface, for example”).
Regarding claim 2, Voodarla discloses:
wherein the artificial neural network analyzes change of the image information over time in the image sequences ([0079] The VAE determines corresponding feature points in sequential images and calculates the change in location of the sensor between images. A three DoF pose for the camera can be determined by triangulating two or more sets of feature points to determine translation and rotation to determine a frame of reference for the sensor in global coordinates.) and generates odometry data based on the change of image information ([0080] “Visual odometry processor 608 determines three DoF poses 622 based on a plurality of pairs of stereo images 602 acquired as the mapping vehicle 145 travels along the path to be topologically mapped. The three DoF pose 622 locates the mapping vehicle 145 with respect to global coordinates.”, [0085] “the vehicle motion vector can include positions in x, y, z, yaw, pitch, roll, yaw rate, pitch rate, roll rate, heading velocity and heading acceleration that can be determined by fitting a polynomial function to successive 2D locations included in the vehicle motion vector with respect to the ground surface, for example”).
Regarding claim 5, Voodarla discloses:
wherein the artificial neural network compensates for changes in calibration parameters of the stereo camera system ([0073] The pairs of stereo images 602 are processed by point cloud processor (PCP) 604 to form an averaged stereo point cloud image by determining three dimensional locations of corresponding feature points based on stereo disparity between the pairs of stereo images 602. PCP 604 can be a CNN as discussed above in relation to FIG. 4.).
Regarding claim 8, Voodarla discloses:
wherein the artificial neural network is a pre-trained neural network which is trained by training data in the form of stereo images of a scene and associated calibration parameters, the training data indicating how the stereo images change depending on modifications of the calibration parameters of the cameras of the stereo camera system ([0073] The pairs of stereo images 602 are processed by point cloud processor (PCP) 604 to form an averaged stereo point cloud image by determining three dimensional locations of corresponding feature points based on stereo disparity between the pairs of stereo images 602. PCP 604 can be a CNN as discussed above in relation to FIG. 4.).
Regarding claim 10, Voodarla discloses:
A system ([0049] second computer 150 of the mapping vehicle 145) for determining information about ego-motion of a vehicle ([0071] mapping vehicle 145)([0033] Determining a path can include solving a localization problem. Localization includes determining a three degree-of-freedom (DoF) pose for the vehicle with respect to a map of the environment around the vehicle, [0078] Stereo visual odometry is a technique for determining a three DoF (3DOF) pose 622 for the mapping vehicle 145), the system comprising:
a stereo camera system with at least two cameras for capturing stereo images ([0071] pairs of stereo images 602) of an area surrounding the vehicle ([0071] pairs of stereo images 602 acquired as a mapping vehicle 145 equipped with stereo video sensors travels along a roadway 204, 206 to be mapped) and an artificial neural network ([0078] variational autoencoder (VAE)) for processing image information provided by the stereo camera system ([0078] VAE can be trained to input image data), the stereo camera system being configured to capture image sequences containing a plurality of image information at different points in time by the at least two cameras during movement of the vehicle ([0071] pairs of stereo images 602 acquired as a mapping vehicle 145 equipped with stereo video sensors travels along a roadway 204, 206 to be mapped, [0079] The VAE determines corresponding feature points in sequential images and calculates the change in location of the sensor between images),
wherein the artificial neural network is configured to receive the image information ([0078] VAE can be trained to input image data) and generate based on the image information stereo images with distance information ([0078] Stereo visual odometry is a technique for determining a three DoF (3DOF) pose 622 for the mapping vehicle 145 based on determining changes in the locations of feature points extracted from the images as the mapping vehicle 145 moves through a scene),
wherein the artificial neural network comprises an output interface at which odometry data is provided, which is calculated by the artificial neural network based on the image sequences ([0078] Visual odometry can be performed by a trained variational autoencoder (VAE)),
wherein the odometry data comprises information regarding the translational motion of the vehicle along three spatial axes of a Cartesian coordinate system ([0080] “Visual odometry processor 608 determines three DoF poses 622 based on a plurality of pairs of stereo images 602 acquired as the mapping vehicle 145 travels along the path to be topologically mapped. The three DoF pose 622 locates the mapping vehicle 145 with respect to global coordinates.”), namely the longitudinal, vertical, and lateral velocity of the vehicle ([0085] “the vehicle motion vector can include positions in x, y, z, yaw, pitch, roll, yaw rate, pitch rate, roll rate, heading velocity and heading acceleration that can be determined by fitting a polynomial function to successive 2D locations included in the vehicle motion vector with respect to the ground surface, for example”), and
wherein the odometry data further comprises information about the rotational motion of the vehicle about the three spatial axes of the Cartesian coordinate system ([0080] “Visual odometry processor 608 determines three DoF poses 622 based on a plurality of pairs of stereo images 602 acquired as the mapping vehicle 145 travels along the path to be topologically mapped. The three DoF pose 622 locates the mapping vehicle 145 with respect to global coordinates.”), namely the pitch, yaw and roll speed of the vehicle ([0085] “the vehicle motion vector can include positions in x, y, z, yaw, pitch, roll, yaw rate, pitch rate, roll rate, heading velocity and heading acceleration that can be determined by fitting a polynomial function to successive 2D locations included in the vehicle motion vector with respect to the ground surface, for example”).
Regarding claim 11, Voodarla discloses:
wherein the artificial neural network is configured to analyze change of the image information over time in the image sequences and to generate odometry data based on said change of image information ([0079] The VAE determines corresponding feature points in sequential images and calculates the change in location of the sensor between images. A three DoF pose for the camera can be determined by triangulating two or more sets of feature points to determine translation and rotation to determine a frame of reference for the sensor in global coordinates.).
Regarding claim 15, Voodarla discloses:
A vehicle ([0071] mapping vehicle 145) comprising the system according to claim 10 (see rejection set forth above for claim 10, claim 15 is rejected for substantially the same reasons and based on the same teaching of Voodarla cited).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 6-7, 9 and 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Voodarla in view of Anonymous, "Depth Camera D435i", February 24, 2021, https://web.archive.org/web/20210224151334/https://www.intelrealsense.com/depth-camera-d435i, 8 pages (hereinafter D2).
Regarding claim 6, Voodarla does not specifically disclose:
wherein the cameras of the stereo camera system comprise inertial sensors which determine changes in the movement of the cameras.
However, D2 discloses:
wherein the cameras of the stereo camera system comprise inertial sensors which determine changes in the movement of the cameras (The Intel® RealSense™ D435i places an IMU into our cutting‑edge stereo depth camera; page 3, line 3).
Voodarla and D2 are considered to be analogous to the claimed invention because they are in the same field of stereo camera. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Voodarla’s stereo camera to further incorporate D2’s stereo camera for the advantage of adding a IMU into the stereo camera which results in camera calibration easier (D2’s page 1, lines 4-7).
Regarding claim 7, Voodarla does not specifically disclose:
wherein calibration parameters of the stereo camera system are adjusted by information provided by the inertial sensors of the cameras of the stereo camera system.
However, D2 discloses:
wherein calibration parameters (calibration; page 1, lines 4-7) of the stereo camera system are adjusted by information provided by the inertial sensors of the cameras of the stereo camera system (Adding an IMU allows your application to refine its depth awareness in any situation where the camera moves. This opens the door for rudimentary SLAM and tracking applications allowing better point-cloud alignment. It also allows improved environmental awareness for robotics and drones. The use of an IMU makes registration and calibration easier; page 1, lines 4-7).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Voodarla’s stereo camera to further incorporate D2’s stereo camera for the advantage of adding a IMU into the stereo camera to adjust parameters which results in camera calibration easier (D2’s page 1, lines 4-7).
Regarding claim 9, Voodarla discloses:
wherein the artificial neural network is post-trained on the basis of calibration information generated from information of sensors of the cameras ([0073] The pairs of stereo images 602 are processed by point cloud processor (PCP) 604 to form an averaged stereo point cloud image by determining three dimensional locations of corresponding feature points based on stereo disparity between the pairs of stereo images 602. PCP 604 can be a CNN as discussed above in relation to FIG. 4.).
Voodarla does not specifically disclose:
inertial sensors of the cameras.
However, D2 discloses:
inertial sensors of the cameras (The Intel® RealSense™ D435i places an IMU into our cutting‑edge stereo depth camera; page 3, line 3).
Voodarla and D2 are considered to be analogous to the claimed invention because they are in the same field of stereo camera. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Voodarla’s stereo camera to further incorporate D2’s stereo camera for the advantage of adding a IMU into the stereo camera to calibrate the images and then train the neural networks using calibrated images which results in camera images refinement (D2’s page 1, lines 4-7).
Regarding claim 13, Voodarla does not specifically disclose:
wherein the cameras of the stereo camera system comprise inertial sensors which are designed to determine changes in movement of the cameras.
However, D2 discloses:
wherein the cameras of the stereo camera system comprise inertial sensors which are designed to determine changes in movement of the cameras (The Intel® RealSense™ D435i places an IMU into our cutting‑edge stereo depth camera; page 3, line 3).
Voodarla and D2 are considered to be analogous to the claimed invention because they are in the same field of stereo camera. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Voodarla’s stereo camera to further incorporate D2’s stereo camera for the advantage of adding a IMU into the stereo camera which results in camera calibration easier (D2’s page 1, lines 4-7).
Regarding claim 14, Voodarla discloses:
wherein the artificial neural network is post-trained on the basis of calibration information generated from information of sensors of the cameras ([0073] The pairs of stereo images 602 are processed by point cloud processor (PCP) 604 to form an averaged stereo point cloud image by determining three dimensional locations of corresponding feature points based on stereo disparity between the pairs of stereo images 602. PCP 604 can be a CNN as discussed above in relation to FIG. 4.).
Voodarla does not specifically disclose:
inertial sensors of the cameras.
However, D2 discloses:
inertial sensors of the cameras (The Intel® RealSense™ D435i places an IMU into our cutting‑edge stereo depth camera; page 3, line 3).
Voodarla and D2 are considered to be analogous to the claimed invention because they are in the same field of stereo camera. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Voodarla’s stereo camera to further incorporate D2’s stereo camera for the advantage of adding a IMU into the stereo camera to calibrate the images and then train the neural networks using calibrated images which results in camera images refinement (D2’s page 1, lines 4-7).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAYSUN WU whose telephone number is (571)272-1528. The examiner can normally be reached Monday-Friday 8AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hunter Lonsberry can be reached on (571)272-7298. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PAYSUN WU/Examiner, Art Unit 3665
/DONALD J WALLACE/Primary Examiner, Art Unit 3665
Prosecution Insights