DETAILED ACTION
Response to Restriction Requirement
Applicant’s amendment received 6/12/26 has been accepted and entered. In response to a restriction requirement, applicant elected, without traverse, invention I for the restriction, corresponding to claims 1-16. Accordingly, claims 17-20 are withdrawn. With respect to the election of species, Applicant elected, without traverse: Group I: species ii; Group II: species i; Group III: species i. Applicant identifies claims 1-5 and 12 as encompassing the elected species such that claims 6-11 and 13-20 are withdrawn. Accordingly, claims 1-5 and 12 are examined herein and claims 6-11 and 13-20 are withdrawn
Claim Objections
Claims 6-11 and 13-20 are objected to because of the following informalities: these claims are withdrawn and should be indicated with the identifier (withdrawn). Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. 20200142426 to Gist et al. (Gist) in view of U.S. Patent Application Publication No. 20210033255 to Kuffner (Kuff)
With respect to claim 1, Gist discloses a system for calibrating a set of sensors on a vehicle, the system comprising:
(i.e., sensor module 102, FIG. 1; ¶ 16 calibrating a position of the pod on the vehicle . . . calibrating the position of the removable pod on the vehicle comprises measuring a set of vehicle dimensions using the plurality of sensors . . . calibrating the position of the pod on the vehicle comprises confirming the position of the removable pod by receiving positioning environmental data from the plurality of sensors; claim 6)
a plurality of sensing modules that are detached from each other, each of the plurality of sensing modules includes:
(i.e., sensor module 102, FIG. 1; 202 Fig. 2; ¶ 13 removable pod attached to a vehicle in an environment using a pod mount, where the plurality of sensors includes at least an inertial measurement unit (IMU), a three dimension (3D) positioning sensor, a camera, and a Light Detection and Ranging (LIDAR) unit; 202, FIG. 2-3; ¶ 46 “several removable pods 202 can be mounted to vehicle 302 to increase the density of data collection. For example, a car can have a removable pod 202 mounted to its front right side and an additional removable pod 202 can be mounted to its front left side . . . three removable pods 202 mounted along the top to capture additional environmental data a single removable pod 202 could potentially miss”)
a respective mounting plate,
(206, FIG. 2 and corresponding description)
a respective sensor fixed on the respective mounting plate, and
(FIG. 1, sensors 104, 106, 108, 112, 114 all mounted to single common pod mounting plate 206, and corresponding description; i.e., ¶¶ 13, 48, 60)
a respective calibration device1 fixed on the respective mounting plate,
(i.e., at least one of 104, 106, 108, 112, 114, FIG. 1; claims 6 and 8; ¶ 16
wherein the respective sensor of each of the plurality of sensing modules is configured to capture respective sensing data of a surrounding of the vehicle; and
(¶ 48 At block 402, the system collects environmental data using removable pod hardware. In many implementations, environmental data includes a variety of information collected about a vehicle's surrounding environment. Environmental data can be collected by one or more sensors in the removable pod including an IMU, a 3D positioning sensor (such as GPS), one or more cameras, a LIDAR unit, etc.)
Gist fails to explicitly disclose use of the calibration device with a first calibration target.
Kuff is from the same field of endeavor because Kuff also discloses
a system for calibrating a set of sensors on a vehicle (¶ 1 “systems and methods for automatically calibrating sensors on a vehicle”)
a plurality of sensing modules that are detached from each other (i.e., (1) mounted in headlamp unit 136A with sensors 144A, 146A 148A; (2) sensors 144B, 146B, 148B, mounted on common headlamp unit 136B; ¶ 19)
wherein the modules can be placed anywhere on the vehicle (¶¶ 24-25 The vehicle headlamp units 136 may further include one or more sensors such as one or more cameras 144, 148 and/or one or more LIDAR sensors 146. However, in some embodiments, the one or more cameras 144, 148 and the one or more LIDAR sensors 146 may be installed in various locations on the vehicle for capturing data from an environment of the vehicle 110”);
wherein a processor configured to generate one or more calibrated position parameters associated with the plurality sensing modules based on measurements generated using the respective calibration device of each of the plurality of sensing modules and a first calibration target
(Kuff, ¶¶ 38-39 calibration pattern 155B comprising a target pattern . . . perceived by the sensors 144, 146, 148 being calibrated . . . Based on the location of each of the features within the calibration pattern as sensed by the sensors being calibrated the calibrations of the sensors may be established or updated; 34-35 electronic control unit 130 may analyze each of the data streams from the sensors 144, 146, 148 and implement one or more mathematical operations to determine the precise location of each the sensors 144, 146, 148 with respect to each other and/or with respect to a frame of reference of the vehicle 110 . . . detection processing algorithms to detect overlapping features within the calibration pattern 155 that are captured by the various sensors 144, 146, 148 to converge on extrinsic calibration values for each of the sensors 144, 146, 148; ¶ 44 block 508 . . . when calibrating . . . data stream that indicates the perceived distances for each sequence of the pattern of dots . . . generate a calibration factor . . . frequencies may relate to relative changes in distance . . . sense the various relative changes; ¶ 45-47 correlate . . . calibration pattern and with other portions of the calibration pattern captured pattern by the other sensors being calibrated to configured the view angle, focal length, distance, position and the like for each camera; ¶¶ 36-40, sizes, shapes of pattern; Fig. 3A-4 and corresponding description)
Accordingly, it would have been obvious to one of ordinary skill in the art at the time of effective filing date to generate one or more calibrated position parameters associated with the plurality sensing modules using a calibration target as taught by Kuff in order to dynamically update calibration of the sensors after they have experienced movement of position since various factors result in de-calibrating the sensor modules during use (Kuff, ¶ 16) without the need of external computing devices (Kuff, ¶ 16). In addition, use of a common calibration target allows the sensor modules to be calibrated with respect to one another, i.e., relative positioning) (Kuff, ¶ 46).
With respect to claim 2, Gist in view of Kuff disclose the plurality of sensing modules are placed to form a constellation2 around the first calibration target.
(Gist as modified by Kuff; i.e., sensor module 102, FIG. 1; 202 Fig. 2; ¶ 13 removable pod attached to a vehicle in an environment using a pod mount, where the plurality of sensors includes at least an inertial measurement unit (IMU), a three dimension (3D) positioning sensor, a camera, and a Light Detection and Ranging (LIDAR) unit; 202, FIG. 2-3; ¶ 46 “several removable pods 202 can be mounted to vehicle 302 to increase the density of data collection. For example, a car can have a removable pod 202 mounted to its front right side and an additional removable pod 202 can be mounted to its front left side . . . three removable pods 202 mounted along the top to capture additional environmental data a single removable pod 202 could potentially miss”)
(Kuff, ¶¶ 33-34 calibration pattern on a surface 152 . . . surface 152 may be . . . a hood 162 of the vehicle 110; 24-25 The vehicle headlamp units 136 may further include one or more sensors such as one or more cameras 144, 148 and/or one or more LIDAR sensors 146. However, in some embodiments, the one or more cameras 144, 148 and the one or more LIDAR sensors 146 may be installed in various locations on the vehicle for capturing data from an environment of the vehicle 110”)
(Kuff, ¶¶ 38-39 calibration pattern 155B comprising a target pattern . . . perceived by the sensors 144, 146, 148 being calibrated . . . Based on the location of each of the features within the calibration pattern as sensed by the sensors being calibrated the calibrations of the sensors may be established or updated; 34-35, i.e., overlapping fields of view from each sensor module -- electronic control unit 130 may analyze each of the data streams from the sensors 144, 146, 148 and implement one or more mathematical operations to determine the precise location of each the sensors 144, 146, 148 with respect to each other and/or with respect to a frame of reference of the vehicle 110 . . . detection processing algorithms to detect overlapping features within the calibration pattern 155 that are captured by the various sensors 144, 146, 148 to converge on extrinsic calibration values for each of the sensors 144, 146, 148; ¶ 44 block 508 . . . when calibrating . . . data stream that indicates the perceived distances for each sequence of the pattern of dots . . . generate a calibration factor . . . frequencies may relate to relative changes in distance . . . sense the various relative changes; ¶ 45-47 first portion and the second portion may include overlapping portions of the calibration pattern . . . correlate . . . calibration pattern and with other portions of the calibration pattern captured pattern by the other sensors being calibrated to configured the view angle, focal length, distance, position and the like for each camera; ¶¶ 36-40, sizes, shapes of pattern; Fig. 3A-4 and corresponding description)
With respect to claim 3, Gist in view of Kuff discloses wherein the respective sensor of each of the plurality of sensing modules is oriented outwardly towards the surrounding of the vehicle, and the respective calibration device of each of the plurality of sensing modules is oriented toward the first calibration target.
(Gist as modified by Kuff; i.e., sensor module 102, FIG. 1; 202 Fig. 2; ¶ 13 removable pod attached to a vehicle in an environment using a pod mount, where the plurality of sensors includes at least an inertial measurement unit (IMU), a three dimension (3D) positioning sensor, a camera, and a Light Detection and Ranging (LIDAR) unit; 202, FIG. 2-3; ¶ 46 “several removable pods 202 can be mounted to vehicle 302 to increase the density of data collection. For example, a car can have a removable pod 202 mounted to its front right side and an additional removable pod 202 can be mounted to its front left side . . . three removable pods 202 mounted along the top to capture additional environmental data a single removable pod 202 could potentially miss”)
(Kuff, ¶¶ 33-34 calibration pattern on a surface 152 . . . surface 152 may be . . . a hood 162 of the vehicle 110; 24-25 The vehicle headlamp units 136 may further include one or more sensors such as one or more cameras 144, 148 and/or one or more LIDAR sensors 146. However, in some embodiments, the one or more cameras 144, 148 and the one or more LIDAR sensors 146 may be installed in various locations on the vehicle for capturing data from an environment of the vehicle 110”)
With respect to claim 4, Gist in view of Kuff discloses the plurality of sensing modules and the first calibration target are placed on top of the vehicle.
(Gist as modified by Kuff; i.e., sensor module 102, FIG. 1; 202 Fig. 2; ¶ 13 removable pod attached to a vehicle in an environment using a pod mount, where the plurality of sensors includes at least an inertial measurement unit (IMU), a three dimension (3D) positioning sensor, a camera, and a Light Detection and Ranging (LIDAR) unit; 202, FIG. 2-3; ¶ 46 “several removable pods 202 can be mounted to vehicle 302 to increase the density of data collection. For example, a car can have a removable pod 202 mounted to its front right side and an additional removable pod 202 can be mounted to its front left side . . . three removable pods 202 mounted along the top to capture additional environmental data a single removable pod 202 could potentially miss”)
(Kuff, ¶¶ 33-34 calibration pattern on a surface 152 . . . surface 152 may be . . . a hood 162 of the vehicle 110)
With respect to claim 5, Gist in view of Kuff discloses the measurements include distances or relative positions between the first calibration target and the respective calibration devices of the plurality of sensing modules.
(Kuff, ¶¶ 33-34 calibration pattern on a surface 152 . . . surface 152 may be . . . a hood 162 of the vehicle 110; 24-25 The vehicle headlamp units 136 may further include one or more sensors such as one or more cameras 144, 148 and/or one or more LIDAR sensors 146. However, in some embodiments, the one or more cameras 144, 148 and the one or more LIDAR sensors 146 may be installed in various locations on the vehicle for capturing data from an environment of the vehicle 110”)
(Kuff, ¶¶ 38-39 calibration pattern 155B comprising a target pattern . . . perceived by the sensors 144, 146, 148 being calibrated . . . Based on the location of each of the features within the calibration pattern as sensed by the sensors being calibrated the calibrations of the sensors may be established or updated; 34-35, i.e., overlapping fields of view from each sensor module -- electronic control unit 130 may analyze each of the data streams from the sensors 144, 146, 148 and implement one or more mathematical operations to determine the precise location of each the sensors 144, 146, 148 with respect to each other and/or with respect to a frame of reference of the vehicle 110 . . . detection processing algorithms to detect overlapping features within the calibration pattern 155 that are captured by the various sensors 144, 146, 148 to converge on extrinsic calibration values for each of the sensors 144, 146, 148; ¶ 44 block 508 . . . when calibrating . . . data stream that indicates the perceived distances for each sequence of the pattern of dots . . . generate a calibration factor . . . frequencies may relate to relative changes in distance . . . sense the various relative changes; ¶ 45-47 first portion and the second portion may include overlapping portions of the calibration pattern . . . correlate . . . calibration pattern and with other portions of the calibration pattern captured pattern by the other sensors being calibrated to configured the view angle, focal length, distance, position and the like for each camera; ¶¶ 36-40, sizes, shapes of pattern; Fig. 3A-4 and corresponding description; 44 “pattern of dots at predetermined intervals that relate to specific time-of-flight or distances that would be perceived by the LIDAR sensor. The receiving portion of the LID AR sensor may generate a data stream that is transmitted to the electronic control unit that indicates the perceived distances for each sequence of the pattern of dots”)
With respect to claim 12, Gist in view of Kuff discloses the first calibration target is: a feature of the vehicle, a physical object positioned on the vehicle, a passive calibration pattern placed on a surface of the vehicle or an active calibration pattern that is projected on a surface of the vehicle.
(Kuff, abstract, 508, FIG. 5; ¶¶ 17, 23, 26, 28, 33-39; 43-46)
Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US 10818034 to FORD is cited to disclose a passive calibration target on the surface of vehicle, as shown in FIG. 2 for calibration of sensors on the vehicle.
PNG
media_image1.png
1463
1355
media_image1.png
Greyscale
US 20200191927 is cited to disclose: FIG. 2 – lidar 12 and camera 10 focus on the same geometric point to determining a calibration parameter
[0033] The vehicle-mounted ECU 100 calibrates the external parameter of the camera 10. In particular, the vehicle-mounted ECU 100 of the first embodiment executes calibration processing for calibrating the external parameter that is set between the camera 10 and the Lidar 12. The vehicle-mounted ECU 100 performs sensor fusion of the camera 10 and the Lidar 12 using the calibrated external parameter and associates the coordinates indicating the same geometric point to improve detection accuracy of the optical sensor.
[0045] The calculation unit 54 generates the external parameter of the camera 10 relative to the Lidar 12 based on the marker image data and the point group information. For example, the position coordinates of the feature points of the calibration marker extracted from the point group information are treated as substantially true values by the calculation unit 54. Then, the calculation unit 54 generates the external parameter that brings the position coordinates of the feature points of the calibration marker extracted from the marker image data close to the true values.
US 20220050191 is cited to disclose:
¶ 26 LIDAR unit 215 could include one or more laser sources, a laser scanner, and one or more detectors, among other system components. Cameras 211 may include one or more devices to capture images of the environment surrounding the ADV. Cameras 211 may be still cameras and/or video cameras. A camera may be mechanically movable, for example, by mounting the camera on a rotating and/or tilting a platform
NODAR US 20210327092 is cited to disclose, two rooftop cameras, same as instant application, FIG. 20B does calibration, it finds range to object 2014 -- (cmos sensor 2010) and a "calibration device” on each camera/calibration module
[0097] FIG. 20A shows a flow diagram of the implementation 1606b, in which the absolute range calibration engine 1606 uses an object with known dimensions, according to some embodiments of the present technology. FIG. 20B illustrates an example of imaging optics for imaging an object 2006 with known dimensions, according to some embodiments of the present technology. In some embodiments, because the dimensions of the object 2006 are known and because focal lengths of camera lenses of the cameras 1300, 1302 are known, the range or distance to the object (target) 2006 may be determined by an equation 2020. More specifically, according to an equation 2020, a ratio of a width W of the object (see 2012) to a range R to the object (see 2014) is equal to a ratio of a focal length F of a camera lens 2008 (see 2016) to a width H of an image (see 2018) sensed by a camera sensor 2010 (e.g., a CMOS sensor). In some embodiments, a known object can be recognized in an automated fashion using known detection technology (e.g., a traffic sign recognition system, license plate detector, an object detector, etc.). In some embodiments, a known object may be specified manually by a manual input of a bounding box and a distance to the known object. An example of an object detector for detecting stop signs is shown in FIG. 24.
Calibration device could be added:
[0129] (18) The stereo vision system of any of configurations (1) to (17), further comprising: a close-range sensor system configured to provide 3D data for objects in a range that is closer than a minimum depth-range of the first and second camera sensors, wherein the at least one processor is configured to combine the 3D data produced from the first and second sensor signals with the 3D data provided by the close-range sensor system. [0130] (19) The stereo vision system of configuration (18), wherein the close-range sensor system is comprised of a pair of camera sensors having a wider field-of-view and a shorter baseline distance than the first and second camera sensors. [0131] (20) The stereo vision system of configuration (18), wherein the close-range sensor system is comprised of a third camera sensor that forms a trinocular stereo system with the first and second camera sensors, such that the first and second camera sensors have a shorter baseline length than the first and third camera sensors. [0132] (21) The stereo vision system of configuration (18), wherein the close-range sensor system is comprised of a time-of-flight camera. [0133] (22) The stereo vision system of any of configurations (1) to (21), further comprising an active illumination device configured to emit visible or infrared radiation towards a field of view of the first and second camera sensors. [0134] (23) The stereo vision system of configuration (22), wherein the active illumination device is configured to: alternate between emitting radiation and not emitting radiation, and emit radiation in synchronization with an exposure interval of the first camera sensor and an exposure interval of the second camera sensor. [0135] (24) The stereo vision system of configuration (22), wherein the active illumination component is comprised of any one or any combination of: a vertical cavity surface emitting laser array, a radiation lamp that emits in a visible spectrum range, and a radiation lamp that emits in a near-infrared spectrum range. [0136] (25). The stereo vision system of any of configurations (1) to (24), wherein the at least one processor is configured to: compute structure-from-motion data from the first sensor signal and from the second sensor signal, and estimate, using the structure-from-motion data, 3D positions of objects that are closer than a minimum depth-range of the first and second camera sensors.
20210080286 TuSimple is cited to disclose: calibration of cameras, looking at same calibration point, each camera is paired with a laser detection (calibration device)
US 11682140 is cited to disclose: The vehicle 100 can include multiple ToF sensors 120 throughout the body of the vehicle 100. In some cases, each stereo camera system 104 can be accompanied by one or more associated ToF sensors 120.
The compute device 128 of the vehicle 100 can be connected to the stereo camera system 104, such as the primary camera 104A and/or the secondary camera 104B, and to the ToF sensor 120 to capture scenes for the ToF stereo calibration refinement system 180.
As discussed above, in some implementations, the ToF disparities 148 can be calculated by transforming the ToF data (e.g., sensor data 124) into a coordinate frame of the stereo camera system 104 using a rigid-body transform, where it is projected into the stereo image data 112/116 using the intrinsics matrix of the stereo camera system 104 to obtain a pixel (x, y coordinate) with an associated distance z. The distance is then converted to a pseudo-disparity (e.g., ToF disparity 148) using stereo calibration,
In some embodiments, prior to generating the stereo image data, the processor 132/188 can be caused by the instructions from the memory 136 of the ToF stereo calibration refinement system 180 to calibrate multiple camera parameters of the primary camera 104A, the secondary camera 104B, and/or the ToF sensor 120, for three-dimensional (3D) image processing. Camera parameters can include intrinsic parameters and extrinsic parameters. The intrinsic parameters can include data representing internal features of the applicable camera, such as the focal length, and the extrinsic parameters can include data representing orientation and/or location of the camera. In some implementations, the camera calibration can include estimating the intrinsic and extrinsic parameters using one or more captured images of a scene.
The ToF sensor 120 onboard the vehicle 120 can be caused by the processor 132 to capture and/or generate sensor data 124 representing a scene, such as the same scene depicted in the stereo image data as described above. “Sensor data,” as used in this disclosure, can refer to data captured and/or produced from a ToF sensor. Sensor data 124 can include time and distance measurements. For example, the ToF sensor 120 can measure the time it takes for a wave of light to travel from the ToF Sensor 120 to an object within a scene and back to the ToF sensor 120. The measurement can be derived from wave propagation, which can enable the ToF sensor 120 to capture 3D images alongside stereoscopic cameras such as the primary camera 104A and the secondary camera 104B. With ToF technology, the wave of light is emitted from a modulated source, such as a laser, and the light beams reflected off one or more objects are then captured by a sensor or camera. The distance can thereby be determined by means of the time delay Δt between when the light is emitted and when the reflected light is received. The time delay is proportional to twice the distance between the camera and the object (round trip). Therefore, the distance can be estimated as
The memory 136 of the ToF stereo calibration refinement system 180 also stores instructions to cause the processor 132 to receive the sensor data 124 from the ToF sensor 120. The memory 136 also stores instructions to cause the processor 132 to project the sensor data 124 into a representation of the scene to generate multiple ToF distance measurements 144. In some implementations, projecting the sensor data 124 can include stacking the sensor data 124 on an image of the scene, the image including the stereo image data from the first stereo image data 112 and/or the second stereo image data 116. The sensor data 124 can include a variety of images, colors, values, or the like, for each pixel of an image. In some cases, the representation of the scene can include the image of the scene captured by the primary camera 104A and/or the secondary camera 104B. For example, the first stereo image data 112 and second stereo image data 116 can include disparities, where the difference is primarily due to angle rotation/position of the primary camera 104A and the secondary camera 104B. A “ToF distance measurement,” as used in this disclosure, can refer to the distance traveled by a light wave and/or signal emitted from the position of a ToF sensor to an object within a scene. In some cases, the ToF sensor 120 can have a range of about 250 meters, or about 500 meters, or more than 500 meters, and the ToF sensor 120 can generate multiple ToF distance measurements 144 for objects within its range. The sensor data 124 can be represented as, or converted into, an image projection 140 containing the ToF distance measurements 144 and/or disparities such as ToF disparities 148. An “image projection,” as used in this disclosure, can refer to a projection of sensor data and/or distance measurements onto an image. In some implementations, the image projection 140 can include a representation of the ToF distance measurements 144 onto the stereo image data, where the stereo image data can include the first stereo image data 112 and/or second stereo image data 116 In some cases, the stereo image data can include a disparity map of the first stereo image data 112 and the second stereo image data 116 captured by the primary camera 104A and the secondary camera 104B respectively. In some implementations, the processor 132 can be caused by the memory 136 to generate the image projection 140 using a projection function/matrix, for example to convert meters found in ToF distance measurements 144 to pixels.
The memory 136 of the ToF stereo calibration refinement system 180 further stores instructions to cause the processor 132 to identify multiple ToF disparities 148 based on the ToF distance measurements 144. A “ToF disparity,” as used in this disclosure, can refer to disparities in the transit time and/or length of distance for a wave/signal to travel after reflecting off an object in a scene. For example, the ToF sensor 120 can measure ToF distance measurements 144 from the sensor data 124 while the primary camera 104A and the secondary camera 104B can generate two disparate image data and measurements. In some cases, the memory 136 of the ToF stereo calibration refinement system 180 stores instructions to identify differences between the ToF distance measurements 144 and one or more disparities in the first stereo image 112 and the second stereo image 116. The ToF disparities 148 can include the difference in measurement values from the binocular disparity from the two stereo image data such as the first stereo image data 112 and the second stereo image data 116. In some cases, the disparity measurements as represented by the ToF disparities 148 from the ToF sensor 120 can be identified using an up-sampling method associated with either the primary camera 104A or the secondary camera 104B. In some cases, the memory 136 stores instructions to further cause the processor 132 to filter the ToF distance measurements 144 prior to identifying the ToF disparities 148. In some cases, the ToF distance measurements 144 can be filtered using a machine learning model and/or a classifier. For example, the sensor data 124 and the stereo image data from the primary camera 104A and the secondary camera 104B can include disparities in measurements. The ToF stereo calibration refinement system 180 can use the machine learning model and/or classifier to filter out the data from the ToF sensor 120 and/or the primary camera 104A and the secondary camera 104B that are furthest from a predefined expected value of the measurements. In some cases, the ToF stereo calibration refinement system 180 can implement a histogram for distance measurements found in (or based on) the stereo image data and the sensor data 124, to filter out the distance measurements and to identify the ToF disparities 148.
In some embodiments, the ToF disparities 148 can include a fusion of measurements from the stereo image data produced by the primary camera 104A and the secondary camera 104B and the sensor data 124 produced by the ToF sensor 120. For example, the ToF disparities 148 can be used to estimate the depth of objects in the scene using the stereo image data, where the stereo image data can include the first stereo image data 112, the second stereo image data 116, and/or the disparity map produced from the rectification of the first stereo image data 112 and the second stereo image data 116. For example, depth and disparity in the image projection 140 are interconnected in standard parallel stereo by:
(39) Z=TfD
where Z is the depth in Euclidian coordinates, T is the baseline, f is the focal length and D is the disparity. In some implementations, the ToF distance measurements 144 extracted from the sensor data 124 can be converted to the disparity with the primary camera 104A and the secondary camera 104B using the baseline and focal lengths of the primary camera 104A and the secondary camera 104B. As described above, ToF sensor 120 can use the timing of reflections of emitted light wave/signal(s), either using a pulsed or continuous wave operation to retrieve a time measurement. For example, the pulsed operation can include sending a pulse in which the peak of the reflection of the pulse off an object is measured by the ToF sensor 120 to detect a yield of time and distance, using:
(40) t=2zc,z=t2c
where t is time, z is distance, and c is the speed of light. The ToF disparities 148 can be used to compensate for the limited range and accuracy of stereo cameras. For example, stereo cameras have extremely high accuracy up close (e.g. ≤3 m), but reduced accuracy at longer ranges due to the quadratic function of triangulation, where the resolution of the stereo cameras drops quadratically with distance. Unlike stereo cameras, the ToF sensor 120 uses an active technique since it projects light to the scene instead of ambient light. Therefore, the ToF sensor 120 can capture data even in dim light conditions. ToF sensor 120 also contains higher processing power than the stereo cameras and support greater distance ranges (e.g. 0.5 m-500 m). However, ToF sensors 120 are also limited in its capabilities of capturing depth for up-close objects. For example, the accuracy of the ToF sensor 120 is largely dependent on the distance from the object in the scene as it is generally estimated at 1% of that value. For example, if an object is 5 meters away, the ToF sensor 120 can achieve an accuracy of about 5 cm. This places the ToF sensor 120 somewhere in between stereo cameras (with precision of about 5-10% of the distance) and other structured light sensors, which the most accurate technology that can have an accuracy of as little as 1 mm. In some implementations, the ToF sensor 120 can include a radar which has an accuracy of =1-5 meters at both the near and the far end of its range. The ToF stereo calibration refinement system 180 can use sensor data 124 from the ToF sensor 120 to measure distances of objects in the scene that are further in range, while the stereo cameras can be used to provide accurate measurements of close-range objects in the scene.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KENNETH J MALKOWSKI whose telephone number is (313)446-4854. The examiner can normally be reached 8:00 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Faris Almatrahi can be reached at 313-446-4821. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KENNETH J MALKOWSKI/Primary Examiner, Art Unit 3667
1 No limiting definition is provided for “calibration device”, however this element can include “sensing or imaging device ( e.g., a Siamese camera, a laser range finder) (spec. ¶ 61); a camera sensor wherein the camera sensor can be both the sensor and the calibration device (¶ 66 camera sensors 110 a-b may be configured to serve the roles of the calibration devices l09a-b in diagram 200a); any device that is capable of estimating a distance or relative position (¶ 73); or a sensor producing imaging data depicting a size or a shape of the calibration target from a point of view at the first sensor or the second sensor (original claim 11).
2 No limiting definition is provided in the specification; the BRI of the term can include a group of things, i.e., two or more objects. See Merriam-Webster definition of “constellation” (“an assemblage, collection, or group of usually related persons, qualities, or things”; “pattern, arrangement”) see https://www.merriam-webster.com/dictionary/constellations
Prosecution Insights