DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This office action is in response to an application filed on 11/11/2024. The applicant submits an Information Disclosure Statement dated 03/21/2025. The applicant does not make a claim for Foreign priority. The applicant makes a claim for Domestic priority to an application filed on 12/18/2020.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 – 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract idea of a mental concept of evaluation without significantly more. The claims are evaluated with respect to the MPEP and 2019 Subject Matter Guidance. Example 40 is used as the reference for evaluating the claims.
STEP 1
The claims recite a computer-implemented method, a non-transitory computer-readable medium, and a computing system. The claims pass the first step 1 by stating one of the four statutory categories.
STEP 2A PRONG I
Independent claim 1 is reproduced below with the abstract idea mental concept elements identified in italics and the pre/post solution activity identified in bold. The analysis of claim 1 is representative of the other independent claims 12 and 20.
Claim 1
A computer-implemented method comprising:
obtaining first sensor data captured by a first sensor of a vehicle during a given period of operation of the vehicle;
obtaining second sensor data captured by a second sensor of the vehicle during the given period of operation of the vehicle;
utilizing a first odometry-based technique to derive a first trajectory comprising a first series of poses based on the first sensor data captured by the first sensor;
utilizing a second odometry-based technique to derive a second trajectory comprising a second series of poses based on the second sensor data captured by the second sensor;
utilizing an optimization technique to align the first and second trajectories;
determining a translation and rotation between the first and second sensors based on the aligned first and second trajectories; and
based on the determined translation and rotation between the first and second sensors, carrying out an action that facilitates recalibration of the first and second sensors.
The inventive concept is determined to be recalibration of a first sensor relative to a second sensor pursuant to MPE 2106.07. The operation steps may be performed in the mind as the claims fail to identify objective data that may not be collected from observation. The features of first sensor, second sensor, period of operation of the vehicle, position determination, translation, and rotation are so broadly claimed, therefore, the features do not identify data that cannot be collected from observation. In addition, the operation clause feature of “carrying out an action that facilitates recalibration of the first and second sensors” does not identify with specificity how the operation is performed or what aspects of the first and second sensor are adjusted based upon the calibration operation.
With respect to MPEP 2106.04(a)(2)(III) the structural features and operations may be performed in the mind pursuant to 2106.04(a)(2)(III)(B or C). The independent claims and dependent claims are not definitive or specific in scope as to the structure components, the data collected, or the operations used to determine whether calibration is necessary, and the operations to adjust the sensors if misalignment is detected.
With respect to the 2019 Guidance and example 40, the claims fail to comply with the guidance by stating the specific data collected for a specific analysis of defined thresholds for which it is determined that a recalibration is necessary. In addition, the claims do not identify how the recalibration is performed with respect to first sensor or the second sensor. The dependent claims 2 – 4 are directed to identifying a misalignment, claims 5 – 9 are directed to executing an undefined realignment without defining what is changed with respect to the first and second sensors. Dependent claims 10 identifies the first and second sensor. Dependent claim 11 is further defining whether a misalignment exists. The analysis applies to the other dependent claims. All the operations are broadly claimed in a manner that does not identify specific data elements and operations that are too complex to be performed in the mind. Therefore, the human mind may observe the sensors that are not operating properly when there is not output of the data processing.
Therefore, the claims fail step 2A Prong I.
STEP 2A PRONG II
This judicial exception is not integrated into a practical application because the claims fail to satisfy the requirements of MPEP 2106.04(d)(1). The claims fail to show an improvement in technology due to the lack of specificity of the determining miscalibration or operations to realign the sensors. The claims further fail to comply with the 2019 Guidance by stating with specificity the data collected and the operation performed that is new or an improvement in determining whether to execute a recalibration. Thus, the claims fail Step 2A Prong II.
STEP 2B
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claims fail to state any of the factors articulated in MPEP 2106.05(a-h). The claims fail to show how the calibration is performed and how the operation is new or an improvement not known with the art. The claims are broadly claimed without scope so that one of ordinary skill in the art would know the thresholds for which the determination is made that the sensors are misaligned. In addition, the claims do not identify how the recalibration is performed or what performs the recalibration. Therefore, the claims fail Step 2B.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claims 1, 5, 6, 8, 9, 12, 16, 17, and 20 contain the phrase “optimization technique” which is interpreted based upon the specification and the claims to not have a definitive scope. The independent claims assert the function is to align first and second trajectories but doesn’t claim how that occurs with respect the first sensor and second sensor data. The dependent claims claim different operations and factors that affect the process. However, none of the dependent claims identify the entire scope of the feature, therefore, one of ordinary skill in the art would interpret the feature as a relative terminology under MPEP 2173.05(b) and conclude the feature is relative terminology and indefinite.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1 – 10, 12 – 17, 19, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Wheeler US 2019/0122386 in view of Garin US 2017/0031032.
As per claim 1, A computer-implemented method comprising:
obtaining first sensor data captured by a first sensor of a vehicle during a given period of operation of the vehicle; (Wheeler paragraph 0061 discloses, “The localize API receives inputs comprising one or more of, location provided by GPS, vehicle motion data provided by IMU, LIDAR scanner data, and camera images. The localize API returns an accurate location of the vehicle as latitude and longitude coordinates. The coordinates returned by the localize API are more accurate compared to the GPS coordinates used as input, for example, the output of the localize API may have precision range from 5-10 cm. In one embodiment, the vehicle computing system 120 invokes the localize API to determine location of the vehicle periodically based on the LIDAR using scanner data, for example, at a frequency of 10 Hz. The vehicle computing system 120 may invoke the localize API to determine the vehicle location at a higher rate (e.g., 60 Hz) if GPS/IMU data is available at that rate. The vehicle computing system 120 stores as internal state, location history records to improve accuracy of subsequent localize calls. The location history record stores history of location from the point-in-time, when the car was turned off/stopped.”)
obtaining second sensor data captured by a second sensor of the vehicle during the given period of operation of the vehicle; (Wheeler paragraph 0061 discloses, “The localize API receives inputs comprising one or more of, location provided by GPS, vehicle motion data provided by IMU, LIDAR scanner data, and camera images. The localize API returns an accurate location of the vehicle as latitude and longitude coordinates. The coordinates returned by the localize API are more accurate compared to the GPS coordinates used as input, for example, the output of the localize API may have precision range from 5-10 cm. In one embodiment, the vehicle computing system 120 invokes the localize API to determine location of the vehicle periodically based on the LIDAR using scanner data, for example, at a frequency of 10 Hz. The vehicle computing system 120 may invoke the localize API to determine the vehicle location at a higher rate (e.g., 60 Hz) if GPS/IMU data is available at that rate. The vehicle computing system 120 stores as internal state, location history records to improve accuracy of subsequent localize calls. The location history record stores history of location from the point-in-time, when the car was turned off/stopped.”)
utilizing a first odometry-based technique to derive a first trajectory comprising a first series of poses based on the first sensor data captured by the first sensor; (Wheeler paragraph 0050 discloses, “A LIDAR surveys the surroundings of the vehicle by measuring distance to a target by illuminating that target with a laser light pulses, and measuring the reflected pulses.”)
utilizing a second odometry-based technique to derive a second trajectory comprising a second series of poses based on the second sensor data captured by the second sensor; (Wheeler paragraph 0050 discloses, “The GPS navigation system determines the position of the vehicle based on signals from satellites. An IMU is an electronic device that measures and reports motion data of the vehicle such as velocity, acceleration, direction of movement, speed, angular rate, and so on using a combination of accelerometers and gyroscopes or other measuring instruments.”)
utilizing an optimization technique to align the first and second trajectories; (Wheeler paragraph 0100 discloses, “the HD map system receives 1130 sensor data from sensors of the vehicle including the camera sensor and lidar sensor, for example, data captured as the vehicle drives along various routes. The HD map system generates 1140 HD maps using the received sensor data and the lidar-to-camera transforms determined by calibrating the sensors of the vehicle.”)
determining a translation and rotation between the first and second sensors based on the aligned first and second trajectories; (Wheeler paragraph 0119 discloses, “The sensor calibration module 290 repeats the steps 1630, 1640, 1650, 1660, and 1670. The sensor calibration module 290 transforms 1630 checkerboard points by small amounts, by varying translation in x, y and rotation around z.”) and
based on the determined translation and rotation between the first and second sensors, carrying out an action that facilitates recalibration of the first and second sensors. (Garin paragraph 0053 teaches, “In some embodiments, the pose of the camera may be used to recalibrate sensors in IMU 170, and/or to compensate for and/or remove biases from measurements of sensors 185 and/or sensors in IMU 170. For example, IMU 170 and/or sensors 185 may output measured information in synchronization with the capture of each image frame by camera(s) 180 by UE 100. When the camera pose can be estimated accurately, for example, based on the images (e.g. successful detection of one or more corresponding feature points in images) then the VIO estimated camera pose may be used to apply corrections to measurements by IMU 170 and/or sensors 185 and/or to recalibrate IMU 170/sensors 185, so that measurements by IMU 170/sensors 185 may more closely track the VIO determined pose.”)
Wheeler discloses a lidar to camera calibration for generating high-definition maps. Wheeler does not does not disclose carrying out a recalibration based upon determined translation and rotation between a first and second sensor. Garin teaches carrying out a recalibration based upon determined translation and rotation between a first and second sensor. Therefore, at the time of filing, it would have been obvious to one of ordinary skill in the art to incorporate the teachings of Garin et.al. into the invention of Wheeler. Such incorporation is motivated by the need to ensure accurate data used to by a vehicle for various uses.
As per claim 2, The computer-implemented method of claim 1, wherein the action comprises at least one of (i) causing issuance of a notification to recalibrate the first or second sensors, (ii) causing the vehicle to pull over, or (iii) updating a previously determined calibration between the first and second sensors. (Wheeler paragraph 0189 discloses, “The application quantifies the amount of drift based on the received user input and sends an alert for re-calibration if needed.”)
As per claim 3, The computer-implemented method of claim 1, wherein the first trajectory is represented in a first local coordinate frame for the first sensor having a first point of origin that corresponds to an initial pose in the first series, and wherein the second trajectory is represented in a second local coordinate frame for the second sensor having a second point of origin that corresponds to an initial pose in the second series. (Wheeler paragraph 0067 discloses, “The calibration module 290 performs various actions related to calibration of sensors of an autonomous vehicle, for example, lidar-to-camera calibration or lidar-to-lidar calibration. Lidar and cameras of an autonomous vehicle record data in their own coordinate systems. In an embodiment, the HD map system 100 determines a rigid 3d transform (a rotation+translation) to convert data from a coordinate system to another.”)
As per claim 4, The computer-implemented method of claim 1, wherein: utilizing the first odometry-technique to derive the first trajectory based on the first sensor data comprises determining, based on the first sensor data, a relative change in position and orientation of the first sensor between capture times for the first sensor data; (Wheeler paragraph 0100 discloses, “the HD map system receives 1130 sensor data from sensors of the vehicle including the camera sensor and lidar sensor, for example, data captured as the vehicle drives along various routes. The HD map system generates 1140 HD maps using the received sensor data and the lidar-to-camera transforms determined by calibrating the sensors of the vehicle. For example, the lidar-to-camera transform is used for correlating the data captured by lidar and camera sensors and combining the data to obtain a consistent view of the surroundings of the vehicle. The vehicle uses 1150 the HD map for various purposes including guiding the vehicle, displaying map data and other applications related to driving of the vehicle or self-driving.”)
utilizing the second odometry-technique to derive the second trajectory based on the second sensor data comprises determining, based on the second sensor data, a relative change in position and orientation of the second sensor between capture times for the second sensor data. (Wheeler paragraph 0100 discloses, “the HD map system receives 1130 sensor data from sensors of the vehicle including the camera sensor and lidar sensor, for example, data captured as the vehicle drives along various routes. The HD map system generates 1140 HD maps using the received sensor data and the lidar-to-camera transforms determined by calibrating the sensors of the vehicle. For example, the lidar-to-camera transform is used for correlating the data captured by lidar and camera sensors and combining the data to obtain a consistent view of the surroundings of the vehicle. The vehicle uses 1150 the HD map for various purposes including guiding the vehicle, displaying map data and other applications related to driving of the vehicle or self-driving.”)
As per claim 5, The computer-implemented method of claim 1, wherein the optimization technique comprises a least squares optimization technique. (Wheeler paragraph 0161 teaches, “With a set of corresponding points in the two coordinate system, the system determines a least squares solution for the rigid transform between lidar and camera coordinates. In some embodiments, this process receives the coordinates of corners from multiple frames.”)
As per claim 6, The computer-implemented method of claim 1, wherein the optimization technique iteratively evaluates different candidate alignments between the first and second trajectories until a best-fit alignment between the first and second trajectories is determined. (Wheeler paragraph 0112 discloses, “The sensor calibration module 290 fits 1540 the checkerboard geometry from boundary points. In an embodiment, the sensor calibration module 290 fits two perpendicular lines that best fit the upper side and lower side boundary points.” And paragraph 0167 discloses, “the overall process for performing calibration of sensors of a vehicle based on edgel detection, according to an embodiment. The edgel based calibration module 950 receives 2110 a lidar scan captured by the lidar of the vehicle and a camera image captured by a camera of the vehicle. The lidar scan and the camera image are obtained from a frame captured at the same time. Accordingly, the lidar scan and the camera image substantially represent the same scene or surroundings of the vehicle or at least have a significant overlap in the portion of the scene captured by the lidar and the camera. If there is a time difference between the capture of the lidar scan and the camera image, the edgel based calibration module 950 performs a temporal correction, for example, by transforming the 3D points to a position corresponding to the time of capture of the image.”)
As per claim 7, The computer-implemented method of claim 6, wherein the best-fit alignment between the first and second trajectories comprises an alignment that minimizes a difference between the first series of poses and the second series of poses. (Wheeler paragraph 0164 discloses, “The HD map system achieves higher calibration accuracy by using frames where the car is stopped at intersections, to prevent other sources of error (e.g., pose error) from affecting calibration. One advantage of these solutions is that they are capable of online calibration during driving of the vehicle. In some scenarios, due to the high variance in real world data, the process may not converge all the time, and may result in lower precision even when the process does converge.” And paragraph 0167 discloses, “the overall process for performing calibration of sensors of a vehicle based on edgel detection, according to an embodiment. The edgel based calibration module 950 receives 2110 a lidar scan captured by the lidar of the vehicle and a camera image captured by a camera of the vehicle. The lidar scan and the camera image are obtained from a frame captured at the same time. Accordingly, the lidar scan and the camera image substantially represent the same scene or surroundings of the vehicle or at least have a significant overlap in the portion of the scene captured by the lidar and the camera. If there is a time difference between the capture of the lidar scan and the camera image, the edgel based calibration module 950 performs a temporal correction, for example, by transforming the 3D points to a position corresponding to the time of capture of the image.”)
As per claim 8, The computer-implemented method of claim 1, wherein the optimization technique is carried out based on a constraint that, for each respective pose in the first series, there is a same fixed difference in position and orientation between the respective pose in the first series and a counterpart pose in the second series that corresponds to a same time during the given period of operation. (Wheeler paragraph 0164 discloses, “The HD map system achieves higher calibration accuracy by using frames where the car is stopped at intersections, to prevent other sources of error (e.g., pose error) from affecting calibration. One advantage of these solutions is that they are capable of online calibration during driving of the vehicle. In some scenarios, due to the high variance in real world data, the process may not converge all the time, and may result in lower precision even when the process does converge.” And paragraph 0167)
As per claim 9, The computer-implemented method of claim 1, wherein the optimization technique is carried out based on one or more additional constraints that require a given level of complexity in a motion path of the vehicle during the given period of operation. (Wheeler paragraph 0167)
As per claim 10, The computer-implemented method of claim 1, wherein the first sensor is a LiDAR unit or a camera, and wherein the second sensor is an Inertial Measurement Unit (IMU). (Wheeler paragraph 0061 discloses, “The localize API receives inputs comprising one or more of, location provided by GPS, vehicle motion data provided by IMU, LIDAR scanner data, and camera images.”)
As per claim 12, A non-transitory computer-readable medium comprising program instructions stored thereon that are executable to cause a computing system to:
obtain first sensor data captured by a first sensor of a vehicle during a given period of operation of the vehicle; (Wheeler paragraph 0061 discloses, “The localize API receives inputs comprising one or more of, location provided by GPS, vehicle motion data provided by IMU, LIDAR scanner data, and camera images. The localize API returns an accurate location of the vehicle as latitude and longitude coordinates. The coordinates returned by the localize API are more accurate compared to the GPS coordinates used as input, for example, the output of the localize API may have precision range from 5-10 cm. In one embodiment, the vehicle computing system 120 invokes the localize API to determine location of the vehicle periodically based on the LIDAR using scanner data, for example, at a frequency of 10 Hz. The vehicle computing system 120 may invoke the localize API to determine the vehicle location at a higher rate (e.g., 60 Hz) if GPS/IMU data is available at that rate. The vehicle computing system 120 stores as internal state, location history records to improve accuracy of subsequent localize calls. The location history record stores history of location from the point-in-time, when the car was turned off/stopped.”)
obtain second sensor data captured by a second sensor of the vehicle during the given period of operation of the vehicle; (Wheeler paragraph 0061 discloses, “The localize API receives inputs comprising one or more of, location provided by GPS, vehicle motion data provided by IMU, LIDAR scanner data, and camera images. The localize API returns an accurate location of the vehicle as latitude and longitude coordinates. The coordinates returned by the localize API are more accurate compared to the GPS coordinates used as input, for example, the output of the localize API may have precision range from 5-10 cm. In one embodiment, the vehicle computing system 120 invokes the localize API to determine location of the vehicle periodically based on the LIDAR using scanner data, for example, at a frequency of 10 Hz. The vehicle computing system 120 may invoke the localize API to determine the vehicle location at a higher rate (e.g., 60 Hz) if GPS/IMU data is available at that rate. The vehicle computing system 120 stores as internal state, location history records to improve accuracy of subsequent localize calls. The location history record stores history of location from the point-in-time, when the car was turned off/stopped.”)
utilize a first odometry-based technique to derive a first trajectory comprising a first series of poses based on the first sensor data captured by the first sensor; (Wheeler paragraph 0050 discloses, “A LIDAR surveys the surroundings of the vehicle by measuring distance to a target by illuminating that target with a laser light pulses, and measuring the reflected pulses.”)
utilize a second odometry-based technique to derive a second trajectory comprising a second series of poses based on the second sensor data captured by the second sensor; (Wheeler paragraph 0050 discloses, “The GPS navigation system determines the position of the vehicle based on signals from satellites. An IMU is an electronic device that measures and reports motion data of the vehicle such as velocity, acceleration, direction of movement, speed, angular rate, and so on using a combination of accelerometers and gyroscopes or other measuring instruments.”)
utilize an optimization technique to align the first and second trajectories; (Wheeler paragraph 0100 discloses, “the HD map system receives 1130 sensor data from sensors of the vehicle including the camera sensor and lidar sensor, for example, data captured as the vehicle drives along various routes. The HD map system generates 1140 HD maps using the received sensor data and the lidar-to-camera transforms determined by calibrating the sensors of the vehicle.”)
determine a translation and rotation between the first and second sensors based on the aligned first and second trajectories; (Wheeler paragraph 0119 discloses, “The sensor calibration module 290 repeats the steps 1630, 1640, 1650, 1660, and 1670. The sensor calibration module 290 transforms 1630 checkerboard points by small amounts, by varying translation in x, y and rotation around z.”) and
based on the determined translation and rotation between the first and second sensors, carry out an action that facilitates recalibration of the first and second sensors. (Garin paragraph 0053 teaches, “In some embodiments, the pose of the camera may be used to recalibrate sensors in IMU 170, and/or to compensate for and/or remove biases from measurements of sensors 185 and/or sensors in IMU 170. For example, IMU 170 and/or sensors 185 may output measured information in synchronization with the capture of each image frame by camera(s) 180 by UE 100. When the camera pose can be estimated accurately, for example, based on the images (e.g. successful detection of one or more corresponding feature points in images) then the VIO estimated camera pose may be used to apply corrections to measurements by IMU 170 and/or sensors 185 and/or to recalibrate IMU 170/sensors 185, so that measurements by IMU 170/sensors 185 may more closely track the VIO determined pose.”)
Wheeler discloses a lidar to camera calibration for generating high-definition maps. Wheeler does not does not disclose carrying out a recalibration based upon determined translation and rotation between a first and second sensor. Garin teaches carrying out a recalibration based upon determined translation and rotation between a first and second sensor. Therefore, at the time of filing, it would have been obvious to one of ordinary skill in the art to incorporate the teachings of Garin et.al. into the invention of Wheeler. Such incorporation is motivated by the need to ensure accurate data used to by a vehicle for various uses.
As per claim 13, The computer-readable medium of claim 12, wherein the action comprises at least one of (i) causing issuance of a notification to recalibrate the first or second sensors, (ii) causing the vehicle to pull over, or (iii) updating a previously determined calibration between the first and second sensors. (Wheeler paragraph 0189 discloses, “The application quantifies the amount of drift based on the received user input and sends an alert for re-calibration if needed.”)
As per claim 14, The computer-readable medium of claim 12, wherein the first trajectory is represented in a first local coordinate frame for the first sensor having a first point of origin that corresponds to an initial pose in the first series, and wherein the second trajectory is represented in a second local coordinate frame for the second sensor having a second point of origin that corresponds to an initial pose in the second series. (Wheeler paragraph 0067 discloses, “The calibration module 290 performs various actions related to calibration of sensors of an autonomous vehicle, for example, lidar-to-camera calibration or lidar-to-lidar calibration. Lidar and cameras of an autonomous vehicle record data in their own coordinate systems. In an embodiment, the HD map system 100 determines a rigid 3d transform (a rotation+translation) to convert data from a coordinate system to another.”)
As per claim 15, The computer-readable medium of claim 12, wherein: utilizing the first odometry-technique to derive the first trajectory based on the first sensor data comprises determining, based on the first sensor data, a relative change in position and orientation of the first sensor between capture times for the first sensor data; (Wheeler paragraph 0100 discloses, “the HD map system receives 1130 sensor data from sensors of the vehicle including the camera sensor and lidar sensor, for example, data captured as the vehicle drives along various routes. The HD map system generates 1140 HD maps using the received sensor data and the lidar-to-camera transforms determined by calibrating the sensors of the vehicle. For example, the lidar-to-camera transform is used for correlating the data captured by lidar and camera sensors and combining the data to obtain a consistent view of the surroundings of the vehicle. The vehicle uses 1150 the HD map for various purposes including guiding the vehicle, displaying map data and other applications related to driving of the vehicle or self-driving.”)
utilizing the second odometry-technique to derive the second trajectory based on the second sensor data comprises determining, based on the second sensor data, a relative change in position and orientation of the second sensor between capture times for the second sensor data. (Wheeler paragraph 0100 discloses, “the HD map system receives 1130 sensor data from sensors of the vehicle including the camera sensor and lidar sensor, for example, data captured as the vehicle drives along various routes. The HD map system generates 1140 HD maps using the received sensor data and the lidar-to-camera transforms determined by calibrating the sensors of the vehicle. For example, the lidar-to-camera transform is used for correlating the data captured by lidar and camera sensors and combining the data to obtain a consistent view of the surroundings of the vehicle. The vehicle uses 1150 the HD map for various purposes including guiding the vehicle, displaying map data and other applications related to driving of the vehicle or self-driving.”)
As per claim 16, The computer-readable medium of claim 12, wherein the optimization tecimique comprises a least squares optimization technique. (Wheeler paragraph 0161 teaches, “With a set of corresponding points in the two coordinate system, the system determines a least squares solution for the rigid transform between lidar and camera coordinates. In some embodiments, this process receives the coordinates of corners from multiple frames.”)
As per claim 17, The computer-readable medium of claim 12, wherein the optimization technique is carried out based on a constraint that, for each respective pose in the first series, there is a same fixed difference in position and orientation between the respective pose in the first series and a counterpart pose in the second series that corresponds to a same time during the given period of operation. (Wheeler paragraph 0164 discloses, “The HD map system achieves higher calibration accuracy by using frames where the car is stopped at intersections, to prevent other sources of error (e.g., pose error) from affecting calibration. One advantage of these solutions is that they are capable of online calibration during driving of the vehicle. In some scenarios, due to the high variance in real world data, the process may not converge all the time, and may result in lower precision even when the process does converge.” And paragraph 0167)
As per claim 19, The computer-readable medium of claim 12, wherein the first sensor is a LiDAR unit or a camera, and wherein the second sensor is an Inertial Measurement Unit (IMU). (Wheeler paragraph 0061 discloses, “The localize API receives inputs comprising one or more of, location provided by GPS, vehicle motion data provided by IMU, LIDAR scanner data, and camera images.
As per claim 20, A computing system comprising:
at least one processor; (Wheeler paragraph 0038 discloses, “FIG. 24 illustrates an embodiment of a computing machine that can read instructions from a machine-readable medium and execute the instructions in a processor or controller.”)
a non-transitory computer-readable medium; and program instructions stored on the non-transitory computer-readable medium that are executable by the at least one processor such that the computing system is capable of:
obtaining first sensor data captured by a first sensor of a vehicle during a given period of operation of the vehicle; (Wheeler paragraph 0061 discloses, “The localize API receives inputs comprising one or more of, location provided by GPS, vehicle motion data provided by IMU, LIDAR scanner data, and camera images. The localize API returns an accurate location of the vehicle as latitude and longitude coordinates. The coordinates returned by the localize API are more accurate compared to the GPS coordinates used as input, for example, the output of the localize API may have precision range from 5-10 cm. In one embodiment, the vehicle computing system 120 invokes the localize API to determine location of the vehicle periodically based on the LIDAR using scanner data, for example, at a frequency of 10 Hz. The vehicle computing system 120 may invoke the localize API to determine the vehicle location at a higher rate (e.g., 60 Hz) if GPS/IMU data is available at that rate. The vehicle computing system 120 stores as internal state, location history records to improve accuracy of subsequent localize calls. The location history record stores history of location from the point-in-time, when the car was turned off/stopped.”)
obtaining second sensor data captured by a second sensor of the vehicle during the given period of operation of the vehicle; (Wheeler paragraph 0061 discloses, “The localize API receives inputs comprising one or more of, location provided by GPS, vehicle motion data provided by IMU, LIDAR scanner data, and camera images. The localize API returns an accurate location of the vehicle as latitude and longitude coordinates. The coordinates returned by the localize API are more accurate compared to the GPS coordinates used as input, for example, the output of the localize API may have precision range from 5-10 cm. In one embodiment, the vehicle computing system 120 invokes the localize API to determine location of the vehicle periodically based on the LIDAR using scanner data, for example, at a frequency of 10 Hz. The vehicle computing system 120 may invoke the localize API to determine the vehicle location at a higher rate (e.g., 60 Hz) if GPS/IMU data is available at that rate. The vehicle computing system 120 stores as internal state, location history records to improve accuracy of subsequent localize calls. The location history record stores history of location from the point-in-time, when the car was turned off/stopped.”)
utilizing a first odometry-based technique to derive a first trajectory comprising a first series of poses based on the first sensor data captured by the first sensor; (Wheeler paragraph 0050 discloses, “A LIDAR surveys the surroundings of the vehicle by measuring distance to a target by illuminating that target with a laser light pulses, and measuring the reflected pulses.” And paragraph 0092 discloses, “A perspective-n-point technique receives input comprising a set of N 3D points in a reference frame and their corresponding 2D image projections as well as the calibrated intrinsic camera parameters, and determines the 6 DOF pose of the camera in the form of its rotation and translation with respect to the world. Given a pose of the camera, the perspective-n-point technique can be used to determine the calibrated intrinsic camera parameters and therefore used for performing calibration of the camera. The parameters of the camera that are calibrated include intrinsic properties of the camera such as the focal length, principal image point, skew parameter, and other parameters.”)
utilizing a second odometry-based technique to derive a second trajectory comprising a second series of poses based on the second sensor data captured by the second sensor; (Wheeler paragraph 0050 discloses, “The GPS navigation system determines the position of the vehicle based on signals from satellites. An IMU is an electronic device that measures and reports motion data of the vehicle such as velocity, acceleration, direction of movement, speed, angular rate, and so on using a combination of accelerometers and gyroscopes or other measuring instruments.” And paragraph 0092 discloses, “A perspective-n-point technique receives input comprising a set of N 3D points in a reference frame and their corresponding 2D image projections as well as the calibrated intrinsic camera parameters, and determines the 6 DOF pose of the camera in the form of its rotation and translation with respect to the world. Given a pose of the camera, the perspective-n-point technique can be used to determine the calibrated intrinsic camera parameters and therefore used for performing calibration of the camera. The parameters of the camera that are calibrated include intrinsic properties of the camera such as the focal length, principal image point, skew parameter, and other parameters.”)
utilizing an optimization technique to align the first and second trajectories; (Wheeler paragraph 0100 discloses, “the HD map system receives 1130 sensor data from sensors of the vehicle including the camera sensor and lidar sensor, for example, data captured as the vehicle drives along various routes. The HD map system generates 1140 HD maps using the received sensor data and the lidar-to-camera transforms determined by calibrating the sensors of the vehicle.”)
determining a translation and rotation between the first and second sensors based on the aligned first and second trajectories; (Wheeler paragraph 0119 discloses, “The sensor calibration module 290 repeats the steps 1630, 1640, 1650, 1660, and 1670. The sensor calibration module 290 transforms 1630 checkerboard points by small amounts, by varying translation in x, y and rotation around z.”) and
based on the determined translation and rotation between the first and second sensors, carrying out an action that facilitates recalibration of the first and second sensors. (Garin paragraph 0053 teaches, “In some embodiments, the pose of the camera may be used to recalibrate sensors in IMU 170, and/or to compensate for and/or remove biases from measurements of sensors 185 and/or sensors in IMU 170. For example, IMU 170 and/or sensors 185 may output measured information in synchronization with the capture of each image frame by camera(s) 180 by UE 100. When the camera pose can be estimated accurately, for example, based on the images (e.g. successful detection of one or more corresponding feature points in images) then the VIO estimated camera pose may be used to apply corrections to measurements by IMU 170 and/or sensors 185 and/or to recalibrate IMU 170/sensors 185, so that measurements by IMU 170/sensors 185 may more closely track the VIO determined pose.”)
Wheeler discloses a lidar to camera calibration for generating high-definition maps. Wheeler does not does not disclose carrying out a recalibration based upon determined translation and rotation between a first and second sensor. Garin teaches carrying out a recalibration based upon determined translation and rotation between a first and second sensor. Therefore, at the time of filing, it would have been obvious to one of ordinary skill in the art to incorporate the teachings of Garin et.al. into the invention of Wheeler. Such incorporation is motivated by the need to ensure accurate data used to by a vehicle for various uses.
Claims 11 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Wheeler US 2019/0122386 in view of Garin US 2017/0031032 in view of Liu US 10,390,003.
As per claim 11, The computer-implemented method of claim 1, wherein the determined translation and rotation between the first and second sensors comprises a first estimate of the translation and rotation between the first and second sensors, the method further comprising:
accessing a combined map comprising (i) a first map layer corresponding to a first type of the first sensor and (ii) a second map layer corresponding to a second type of the second sensor, wherein the first and second map layers are aligned such that there is a known transformation between a first coordinate frame of the first map layer and a second coordinate frame of the second map layer; (Wheeler paragraph 0100 discloses, “The HD map system generates 1140 HD maps using the received sensor data and the lidar-to-camera transforms determined by calibrating the sensors of the vehicle.”)
localizing the first sensor within the first coordinate frame of a first map layer of the combined map; (Liu Col 6 lines 38 – 58)
localizing the second sensor within the second coordinate frame of a second map layer of the combined map; (Liu Col 6 lines 38 – 58)
based on the known transformation between the first coordinate frame and the second coordinate frame, determining respective poses for the first sensor and the second sensor in a common coordinate frame; (Liu Col 20 lines 13 – 23) and
determining a second estimate of the translation and rotation between the first and second sensors based on the respective poses for the first sensor and the second sensor in the common coordinate frame, (Liu Col 20 lines 13 – 23)
wherein the second estimate of the translation and rotation between the first and second sensors is combined with the first estimate of the translation and rotation between the first and second sensors. (Liu Col 6 lines 38 – 58)
Wheeler discloses a lidar to camera calibration for generating high-definition maps. Wheeler does not does not disclose localizing sensor data within a frame. Liu teaches localizing sensor data within a frame. Therefore, at the time of filing, it would have been obvious to one of ordinary skill in the art to incorporate the teachings of Liu et.al. into the invention of Wheeler. Such incorporation is motivated by the need to ensure accurate data used to by a vehicle for various uses.
As per claim 18, The computer-readable medium of claim 12, wherein the determined translation and rotation between the first and second sensors comprises a first estimate of the translation and rotation between the first and second sensors, and wherein the computer-readable medium further comprises program instructions stored thereon that are executable to cause the computing system to:
access a combined map comprising (i) a first map layer corresponding to a first type of the first sensor and (ii) a second map layer corresponding to a second type of the second sensor, wherein the first and second map layers are aligned such that there is a known transformation between a first coordinate frame of the first map layer and a second coordinate frame of the second map layer; (Wheeler paragraph 0100 discloses, “The HD map system generates 1140 HD maps using the received sensor data and the lidar-to-camera transforms determined by calibrating the sensors of the vehicle.”)
localize the first sensor within the first coordinate frame of a first map layer of the combined map; (Liu Col 6 lines 38 – 58)
localize the second sensor within the second coordinate frame of a second map layer of the combined map; (Liu Col 6 lines 38 – 58)
based on the known transformation between the first coordinate frame and the second coordinate frame, determine respective poses for the first sensor and the second sensor in a common coordinate frame; (Liu Col 20 lines 13 – 23) and
determine a second estimate of the translation and rotation between the first and second sensors based on the respective poses for the first sensor and the second sensor in the common coordinate frame, (Liu Col 20 lines 13 – 23)
wherein the second estimate of the translation and rotation between the first and second sensors is combined with the first estimate of the translation and rotation between the first and second sensors. (Liu Col 6 lines 38 – 58)
Wheeler discloses a lidar to camera calibration for generating high-definition maps. Wheeler does not does not disclose localizing sensor data within a frame. Liu teaches localizing sensor data within a frame. Therefore, at the time of filing, it would have been obvious to one of ordinary skill in the art to incorporate the teachings of Liu et.al. into the invention of Wheeler. Such incorporation is motivated by the need to ensure accurate data used to by a vehicle for various uses.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TYLER D PAIGE whose telephone number is (571)270-5425. The examiner can normally be reached M-F 7:00am - 6:00pm (mst).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kito Robinson can be reached at 5712703921. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TYLER D PAIGE/Primary Examiner, Art Unit 3664
Prosecution Insights