DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 26 and 30 are objected to because of the following informalities:
Claim 26 recites “The ToF camera system according to claim 22, wherein the weighted sensor unit temperature difference (
c
T
S
e
n
∙
Δ
T
S
e
n
) each as an offset value of the generated depth images (
D
).”
This is not a sentence as it does not contain a verb and therefore is grammatically incorrect
Claim 30 recites “temperature measuring unit (20) of the illumination unit”
Claim 30 should be amended to recite “temperature measuring unit of the illumination unit”
Appropriate correction is required.
Claim Rejections - 35 USC § 112
Claims 20, 25, and 26 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The claims are considered indefinite because there is a question or doubt as to whether the mathematical expressions inside the parenthesis are (a) merely exemplary of the remainder of the claim, and therefore not required, or (b) a required feature of the claims. In the present instance, Claim 20 recites “the weighted sensor unit temperature difference (
c
T
S
e
n
∙
Δ
T
S
e
n
).” Claim 20 has a total of five similar structures of plain English followed by mathematical expressions. Additionally, Claims 25 and 26 use similar structure with mathematical expressions inside parenthesis recited after plain English. Therefore, Claims 20, 25, and 26 are considered indefinite as for failing to particularly point out and distinctly claim the subject matter regarded as the invention.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 15 and 30 are rejected under 35 U.S.C. 103 as being unpatentable over Schoenlieb et al. (US 2022/0074856 A1) in view of Kanko et al. (US 2021/0323086 A1).
Regarding Claim 15, Schoenlieb teaches a time of flight, ToF, camera system ([0031] a two-dimensional sensing method based on a ToF camera) for measuring depth images of an environment ([0031] The result of the sensing may, e.g., be a two-dimensional image in which the pixels respectively indicate the amount of light reflected by the object), wherein the ToF camera system comprises:
an illumination unit comprising a light source for emitting modulated light signals for illuminating objects of the environment ([0032] illuminating the scene comprising the object 201 with the modulated light 202 of the illumination element 230 based on an illumination signal);
a sensor unit comprising an image sensor for acquiring light signals reflected from the objects ([0031] The result of the sensing may, e.g., be a two-dimensional image in which the pixels respectively indicate the amount of light reflected by the object);
a processing unit for generating the depth images based on the acquired reflected light signals ([0029] the processing circuit 240 may further output data indicative of the reflectivity value (e.g. a two-dimensional image)); and
a correction unit for correcting temperature-related measurement errors of the depth images ([0042] Based on at least one of the measured temperature at the illumination element 220 and the measured light strength and/or rise time of the modulated light 202 emitted by the illumination element 220, at least one of the illumination signal, the reference signal and the time offset between the illumination signal and the reference signal may be varied in order to compensate for temperature-dependent drifts in operation of the illumination element 220);
wherein the sensor unit comprises a sensor unit temperature measuring unit for measuring a current temperature (
T
S
e
n
) at the sensor unit and/or the illumination unit comprises an illumination unit temperature measuring unit for measuring a current temperature (
T
I
L
L
) at the illumination unit ([0042] The apparatus 200 may comprise one or more temperature sensors for measuring the temperature at the illumination element 220); and
wherein the correction unit is adapted to perform the correcting in dependence of the current temperature (
T
S
e
n
) at the sensor unit and/or the current temperature (
T
I
L
L
) at the illumination unit ([0042] Based on at least one of the measured temperature at the illumination element 220 and the measured light strength and/or rise time of the modulated light 202 emitted by the illumination element 220, at least one of the illumination signal, the reference signal and the time offset between the illumination signal and the reference signal may be varied in order to compensate for temperature-dependent drifts in operation of the illumination element 220).
Schoenlieb is not relied upon as teaching that the correction unit is adapted to perform the correcting in dependence of a set exposure time (exp) and a set frame rate (fr).
However, Kanko teaches that the correction unit is adapted to perform the correcting in dependence of a set exposure time (exp) and a set frame rate (fr) ([0099] The 2D camera hardware may also have adjustable properties (e.g., exposure time, frame rate, field of view) that are configurable according to the type of calibration measurement being performed).
Schoenlieb and Kanko are considered to be analogous to the claimed invention because they are both in the same field of time-of-flight (ToF) depth imaging and sensor calibration. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the correction unit of Schoenlieb to include the adjustable properties (e.g., exposure time, frame rate) of Kanko with a reasonable expectation of success. This modification would have been motivated by the desire to improve the accuracy and robustness of temperature-related measurement error correction across varying operational settings. By integrating Kanko’s teaching of configuring exposure time and frame rate into Schoenlieb’s correction unit, the system can account for the impact these operational parameters have on temperature-dependent drifts, thereby enhancing the reliability of the depth images. A person of ordinary skill in the art would recognize that accounting for exposure time and frame rate in the correction algorithm would yield the predictable result of more precise and consistent depth measurements under a broader range of operating conditions.
Regarding Claim 30, Schoenlieb teaches a time of flight, ToF, method for measuring depth images (
D
) of an environment ([0031] a two-dimensional sensing method based on a ToF camera… The result of the sensing may, e.g., be a two-dimensional image in which the pixels respectively indicate the amount of light reflected by the object), wherein the ToF method comprises the steps of:
emitting modulated light signals for illuminating objects of the environment, by a light source of an illumination unit ([0032] illuminating the scene comprising the object 201 with the modulated light 202 of the illumination element 230 based on an illumination signal);
acquiring light signals reflected by the objects, by an image sensor of a sensor unit ([0031] The result of the sensing may, e.g., be a two-dimensional image in which the pixels respectively indicate the amount of light reflected by the object);
generating the depth images (
D
) based on the acquired reflected light signals, by a processing unit ([0029] the processing circuit 240 may further output data indicative of the reflectivity value (e.g. a two-dimensional image)); and
correcting temperature-related measurement errors of the depth images (
D
), by a correction unit ([0042] Based on at least one of the measured temperature at the illumination element 220 and the measured light strength and/or rise time of the modulated light 202 emitted by the illumination element 220, at least one of the illumination signal, the reference signal and the time offset between the illumination signal and the reference signal may be varied in order to compensate for temperature-dependent drifts in operation of the illumination element 220);
wherein the correction is performed in dependence of a current temperature (
T
S
e
n
) at the sensor unit measured by a sensor unit temperature measuring unit of the sensor unit and/or a current temperature (
T
I
L
L
) at the illumination unit measured by an illumination unit temperature measuring unit (20) of the illumination unit ([0042] Based on at least one of the measured temperature at the illumination element 220 and the measured light strength and/or rise time of the modulated light 202 emitted by the illumination element 220, at least one of the illumination signal, the reference signal and the time offset between the illumination signal and the reference signal may be varied in order to compensate for temperature-dependent drifts in operation of the illumination element 220,
Schoenlieb is not relied upon as teaching that the correction is performed in dependence of a set exposure time (
e
x
p
) and a set frame rate (
f
r
).
However, Kanko teaches that the correction is performed in dependence of a set exposure time (
e
x
p
) and a set frame rate (
f
r
) ([0099] The 2D camera hardware may also have adjustable properties (e.g., exposure time, frame rate, field of view) that are configurable according to the type of calibration measurement being performed).
Schoenlieb and Kanko are considered to be analogous to the claimed invention because they are both in the same field of time-of-flight (ToF) depth imaging and sensor calibration. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of correcting temperature-related measurement errors of Schoenlieb to include the adjustable properties (e.g., exposure time, frame rate) of Kanko with a reasonable expectation of success. This modification would have been motivated by the desire to improve the accuracy and robustness of temperature-related measurement error correction across varying operational settings. By integrating Kanko’s teaching of configuring exposure time and frame rate into Schoenlieb’s method, the system can account for the impact these operational parameters have on temperature-dependent drifts, thereby enhancing the reliability of the depth images. A person of ordinary skill in the art would recognize that accounting for exposure time and frame rate in the correction method would yield the predictable result of more precise and consistent depth measurements under a broader range of operating conditions.
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Schoenlieb et al. (US 2022/0074856 A1) and Kanko et al. (US 2021/0323086 A1) in further view of Ono et al. (US 2022/0137224 A1).
Regarding Claim 21, Schoenlieb is not relied upon as teaching that the sensor unit is adapted to alternately acquire the light signals reflected from the objects with a shorter exposure time (
e
x
p
S
) and a longer exposure time (
e
x
p
L
); the processing unit is adapted to generate the first depth images (
D
S
) based on the reflected light signals acquired with the shorter exposure time (
e
x
p
S
) and the second depth images (
D
L
) based on the reflected light signals acquired with the longer exposure time (
e
x
p
L
) and to generate the depth images (
D
) based on the first depth images (
D
S
) and the second depth images (
D
L
); and the correction unit is adapted to correct the temperature-related measurement errors of the first depth images (
D
S
) and the second depth images (
D
L
) each separately.
However, Ono teaches that the sensor unit is adapted to alternately acquire the light signals reflected from the objects with a shorter exposure time (
e
x
p
S
) and a longer exposure time (
e
x
p
L
) ([0208] One frame includes a frame for performing distance measurement by capturing an image through long-time exposure (this frame will be hereinafter referred to as the subframe L), and a frame for performing distance measurement by capturing an image through short-time exposure (this frame will be hereinafter referred to as the subframe S) [0213] The subframe L and the subframe S are sequentially supplied from the light receiving unit 12 (FIG. 1) to the HDR combining unit 201. Note that, although the subframe S is supplied after the subframe L in the explanation continued below, the subframe L may be supplied after the subframe S. In other words, distance measurement by short-time exposure may be performed after distance measurement by long-time exposure, or distance measurement by long-time exposure may be performed after distance measurement by short-time exposure Examiner Note: If one frame includes a frame for long exposure time and a frame for short exposure time, it would be obvious to perform these measurements alternately);
the processing unit is adapted to generate the first depth images (
D
S
) based on the reflected light signals acquired with the shorter exposure time (
e
x
p
S
) and the second depth images (
D
L
) based on the reflected light signals acquired with the longer exposure time (
e
x
p
L
) ([0201] one image by performing a combining process in which a long-time exposure image and a short-time exposure image are successively and individually captured) and to generate the depth images (
D
) based on the first depth images (
D
S
) and the second depth images (
D
L
) ([0201] a technique for successively capturing a plurality of images with different exposure times and combining the images is known as a method for expanding the dynamic range); and
the correction unit is adapted to correct the temperature-related measurement errors of the first depth images (
D
S
) and the second depth images (
D
L
) each separately ([0221] Note that, in a case where the imaging by long-time exposure is not saturated, there is a high possibility that the imaging by short-time exposure is not saturated either. Therefore, the detection signals obtained by the long-time exposure imaging and the detection signals obtained by the short-time exposure imaging may be combined by a predetermined method, such as multiplying each of the detection signals by a weighting coefficient and adding up the resultant detection signals, for example. The distance D may then be calculated from the combined signal Examiner Note: weighting coefficient could be combined with the temperature-related measurement error correction of Schoenlieb).
Schoenlieb, Kanko, and Ono are considered to be analogous to the claimed invention because they are all in the same field of time-of-flight (ToF) depth imaging and sensor calibration. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the ToF system of Schoenlieb (as modified by Kanko) to include the alternating short and long exposure time acquisition and combination as taught by Ono with a reasonable expectation of success. This modification would have been motivated by the desire to increase the dynamic range of the depth images while maintaining accurate temperature-related error correction. By integrating Ono’s teaching of successively capturing images with different exposure times into the system of Schoenlieb (as modified by Kanko), the system can generate depth images from combined exposure data that are independently corrected for temperature-dependent drifts. A person of ordinary skill in the art would recognize that applying temperature correction to separate depth images obtained from different exposure times would yield the predictable result of more accurate and reliable depth measurements in environments with high dynamic range.
Claims 22 and 31 are rejected under 35 U.S.C. 103 as being unpatentable over Schoenlieb et al. (US 2022/0074856 A1) and Marra (US 2018/0180471 A1) in further view of Dehghani (US 2022/0377217 A1).
Regarding Claim 22, Schoenlieb teaches a time of flight, ToF, camera system ([0031] a two-dimensional sensing method based on a ToF camera) for measuring depth images (
D
) of an environment ([0031] The result of the sensing may, e.g., be a two-dimensional image in which the pixels respectively indicate the amount of light reflected by the object), wherein the ToF camera system comprises:
an illumination unit comprising a light source for emitting modulated light signals for illuminating objects of the environment ([0032] illuminating the scene comprising the object 201 with the modulated light 202 of the illumination element 230 based on an illumination signal);
a sensor unit comprising an image sensor for acquiring light signals reflected from the objects ([0031] The result of the sensing may, e.g., be a two-dimensional image in which the pixels respectively indicate the amount of light reflected by the object);
a processing unit for generating the depth images (
D
) based on the acquired reflected light signals ([0029] the processing circuit 240 may further output data indicative of the reflectivity value (e.g. a two-dimensional image)); and
a correction unit for correcting temperature-related measurement errors of the generated depth images (
D
) ([0042] Based on at least one of the measured temperature at the illumination element 220 and the measured light strength and/or rise time of the modulated light 202 emitted by the illumination element 220, at least one of the illumination signal, the reference signal and the time offset between the illumination signal and the reference signal may be varied in order to compensate for temperature-dependent drifts in operation of the illumination element 220);
wherein the illumination unit comprises an illumination unit temperature measuring unit for measuring a current temperature (
T
I
L
L
) at the illumination unit ([0042] The apparatus 200 may comprise one or more temperature sensors for measuring the temperature at the illumination element 220); and
wherein the correction unit is adapted to perform the correcting in dependence of the current temperature (
T
I
L
L
) at the illumination unit ([0042] Based on at least one of the measured temperature at the illumination element 220 and the measured light strength and/or rise time of the modulated light 202 emitted by the illumination element 220, at least one of the illumination signal, the reference signal and the time offset between the illumination signal and the reference signal may be varied in order to compensate for temperature-dependent drifts in operation of the illumination element 220) independently of time (([0042] Based on at least one of the measured temperature at the illumination element Examiner Note: Applicant specification [0029] states “the correction can also be performed independently of time -i.e., without determining temperature change rates” and Schoenlieb does the correction based on the measured temperature at the illumination element 220).
Schoenlieb is not relied upon as teaching that the sensor unit comprises a sensor unit temperature measuring unit for measuring a current temperature (
T
S
e
n
) at the sensor unit; and that the correction unit is adapted to perform the correcting in dependence of the current temperature (
T
S
e
n
) at the sensor unit directly on the generated depth images (
D
).
However, Marra teaches that the sensor unit comprises a sensor unit temperature measuring unit for measuring a current temperature (
T
S
e
n
) at the sensor unit; and that the correction unit is adapted to perform the correcting in dependence of the current temperature (
T
S
e
n
) at the sensor unit ([0024] The voltage compensation unit preferably is configured as a multi-channel unit for individually adapting the bias voltage for different groups of avalanche photodiode elements in accordance with different operating temperatures measured at different positions. The term multi-channel also includes a plurality of voltage compensation units. Then, the bias voltage may optimally be adapted even in case of the operating temperature not being the same everywhere on the light receiver. It is conceivable that the temperature is measured at more positions than there are available channels for different bias voltage compensations in that some measurements are combined by averaging).
Schoenlieb and Marra are considered to be analogous to the claimed invention because they are both in the same field of time-of-flight depth imaging and sensor temperature compensation. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the sensor unit of Schoenlieb to include the sensor unit temperature measuring unit and corresponding temperature-dependent correction unit of Marra with a reasonable expectation of success. This modification would have been motivated by the desire to improve the precision of depth measurements by accounting for thermal fluctuations at the sensor level, in addition to the illumination unit. By integrating Marra’s teaching of measuring the current temperature at the sensor unit and performing correction based on that temperature into the Schoenlieb ToF system, the system can more comprehensively compensate for temperature-dependent drifts across the entire light receiver. A person of ordinary skill in the art would recognize that utilizing localized temperature measurements at the sensor to adjust compensation would yield the predictable result of enhanced accuracy and stability of the generated depth images, particularly in scenarios with varying operational temperatures.
Marra is not relied upon as teaching that the correction unit is adapted to perform the correcting directly on the generated depth images (
D
).
However, Dehghani teaches that the correction unit is adapted to perform the correcting directly on the generated depth images (
D
) ([0313] the system may further comprise an image post processing unit configured to update the depth map based on temperature fluctuations associated with an operation of the TOF light source or the TOF sensor, thereby enhancing depth sensing accuracy and surgical safety.).
Schoenlieb (as previously modified by Marra) and Dehghani are considered to be analogous to the claimed invention because they are both in the same field of time-of-flight (ToF) depth imaging and sensor calibration. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Schoenlieb (as previously modified by Marra) to include performing the correcting directly on the generated depth images of Dehghani with a reasonable expectation of success. This modification would have been motivated by the desire to enhance depth sensing accuracy. By integrating Dehghani‘s teaching of configuring an image post processing unit to update the depth map based on temperature fluctuations into Schoenlieb’s (as previously modified by Marra) system, the system can correct temperature-related measurement errors directly on the final depth data output. A person of ordinary skill in the art would recognize that applying temperature compensation directly to the generated depth maps as a post-processing operation would yield the predictable result of more accurate and reliable depth sensing measurements.
Regarding Claim 31, Schoenlieb teaches a time of flight, ToF, method ([0031] a two-dimensional sensing method based on a ToF camera) for measuring depth images (
D
) of an environment ([0031] The result of the sensing may, e.g., be a two-dimensional image in which the pixels respectively indicate the amount of light reflected by the object), the ToF method comprising the steps of:
emitting modulated light signals for illuminating objects of the environment, by a light source of an illumination unit ([0032] illuminating the scene comprising the object 201 with the modulated light 202 of the illumination element 230 based on an illumination signal);
acquiring light signals reflected by the objects, by an image sensor of a sensor unit ([0031] The result of the sensing may, e.g., be a two-dimensional image in which the pixels respectively indicate the amount of light reflected by the object);
generating the depth images (
D
) based on the acquired reflected light signals, by a processing unit ([0029] the processing circuit 240 may further output data indicative of the reflectivity value (e.g. a two-dimensional image)); and
correcting temperature-related measurement errors of the depth images (
D
), by a correction unit ([0042] Based on at least one of the measured temperature at the illumination element 220 and the measured light strength and/or rise time of the modulated light 202 emitted by the illumination element 220, at least one of the illumination signal, the reference signal and the time offset between the illumination signal and the reference signal may be varied in order to compensate for temperature-dependent drifts in operation of the illumination element 220);
wherein the correction is performed in dependence of a current temperature (
T
I
L
L
) at the illumination unit measured by an illumination unit temperature measuring unit of the illumination unit ([0042] Based on at least one of the measured temperature at the illumination element 220 and the measured light strength and/or rise time of the modulated light 202 emitted by the illumination element 220, at least one of the illumination signal, the reference signal and the time offset between the illumination signal and the reference signal may be varied in order to compensate for temperature-dependent drifts in operation of the illumination element 220) and independently of time (([0042] Based on at least one of the measured temperature at the illumination element Examiner Note: Applicant specification [0029] states “the correction can also be performed independently of time -i.e., without determining temperature change rates” and Schoenlieb does the correction based on the measured temperature at the illumination element 220).
Schoenlieb is not relied upon as teaching that the sensor unit comprises a sensor unit temperature measuring unit for measuring a current temperature (
T
S
e
n
) at the sensor unit; and that the correction unit is adapted to perform the correcting in dependence of the current temperature (
T
S
e
n
) at the sensor unit directly on the generated depth images (
D
).
However, Marra teaches that the sensor unit comprises a sensor unit temperature measuring unit for measuring a current temperature (
T
S
e
n
) at the sensor unit; and that the correction unit is adapted to perform the correcting in dependence of the current temperature (
T
S
e
n
) at the sensor unit ([0024] The voltage compensation unit preferably is configured as a multi-channel unit for individually adapting the bias voltage for different groups of avalanche photodiode elements in accordance with different operating temperatures measured at different positions. The term multi-channel also includes a plurality of voltage compensation units. Then, the bias voltage may optimally be adapted even in case of the operating temperature not being the same everywhere on the light receiver. It is conceivable that the temperature is measured at more positions than there are available channels for different bias voltage compensations in that some measurements are combined by averaging).
Schoenlieb and Marra are considered to be analogous to the claimed invention because they are both in the same field of time-of-flight depth imaging and sensor temperature compensation. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Schoenlieb to incorporate the sensor unit temperature measuring unit and corresponding temperature-dependent correction unit of Marra with a reasonable expectation of success. This modification would have been motivated by the desire to improve the precision of depth measurements by accounting for thermal fluctuations at the sensor level, in addition to the illumination unit. By integrating Marra’s teaching of measuring the current temperature at the sensor unit and performing correction based on that temperature into the Schoenlieb ToF method, the method can more comprehensively compensate for temperature-dependent drifts across the entire light receiver. A person of ordinary skill in the art would recognize that utilizing localized temperature measurements at the sensor to adjust compensation would yield the predictable result of enhanced accuracy and stability of the generated depth images, particularly in scenarios with varying operational temperatures.
Marra is not relied upon as teaching that the correction unit is adapted to perform the correcting directly on the generated depth images (
D
).
However, Dehghani teaches that the correction unit is adapted to perform the correcting directly on the generated depth images (
D
) ([0313] the system may further comprise an image post processing unit configured to update the depth map based on temperature fluctuations associated with an operation of the TOF light source or the TOF sensor, thereby enhancing depth sensing accuracy and surgical safety.).
Schoenlieb (as previously modified by Marra) and Dehghani are considered to be analogous to the claimed invention because they are both in the same field of time-of-flight (ToF) depth imaging and sensor calibration. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Schoenlieb (as previously modified by Marra) to include performing the correcting directly on the generated depth images of Dehghani with a reasonable expectation of success. This modification would have been motivated by the desire to enhance depth sensing accuracy. By integrating Dehghani‘s teaching of configuring an image post processing unit to update the depth map based on temperature fluctuations into Schoenlieb’s (as previously modified by Marra) system, the method can correct temperature-related measurement errors directly on the final depth data output. A person of ordinary skill in the art would recognize that applying temperature compensation directly to the generated depth maps as a post-processing operation would yield the predictable result of more accurate and reliable depth sensing measurements.
Claims 27 and 28 are rejected under 35 U.S.C. 103 as being unpatentable over Schoenlieb et al. (US 2022/0074856 A1), Marra (US 2018/0180471 A1), and Dehghani (US 2022/0377217 A1) in further view of Ono et al. (US 2022/0137224 A1).
Regarding Claim 27, Schoenlieb is not relied upon as teaching that the sensor unit is adapted to alternately acquire the light signals reflected from the objects with a shorter exposure time (
e
x
p
S
) and a longer exposure time (
e
x
p
L
); the processing unit is adapted to generate the first depth images (
D
S
) based on the reflected light signals acquired with the shorter exposure time (
e
x
p
S
) and the second depth images (
D
L
) based on the reflected light signals acquired with the longer exposure time (
e
x
p
L
) and to generate the depth images (
D
) based on the first depth images (
D
S
) and the second depth images (
D
L
); and the correction unit is adapted to correct the temperature-related measurement errors of the first depth images (
D
S
) and the second depth images (
D
L
) each separately.
However, Ono teaches that the sensor unit is adapted to alternately acquire the light signals reflected from the objects with a shorter exposure time (
e
x
p
S
) and a longer exposure time (
e
x
p
L
) ([0208] One frame includes a frame for performing distance measurement by capturing an image through long-time exposure (this frame will be hereinafter referred to as the subframe L), and a frame for performing distance measurement by capturing an image through short-time exposure (this frame will be hereinafter referred to as the subframe S) [0213] The subframe L and the subframe S are sequentially supplied from the light receiving unit 12 (FIG. 1) to the HDR combining unit 201. Note that, although the subframe S is supplied after the subframe L in the explanation continued below, the subframe L may be supplied after the subframe S. In other words, distance measurement by short-time exposure may be performed after distance measurement by long-time exposure, or distance measurement by long-time exposure may be performed after distance measurement by short-time exposure Examiner Note: If one frame includes a frame for long exposure time and a frame for short exposure time, it would be obvious to perform these measurements alternately);
the processing unit is adapted to generate the first depth images (
D
S
) based on the reflected light signals acquired with the shorter exposure time (
e
x
p
S
) and the second depth images (
D
L
) based on the reflected light signals acquired with the longer exposure time (
e
x
p
L
) ([0201] one image by performing a combining process in which a long-time exposure image and a short-time exposure image are successively and individually captured) and to generate the depth images (
D
) based on the first depth images (
D
S
) and the second depth images (
D
L
) ([0201] a technique for successively capturing a plurality of images with different exposure times and combining the images is known as a method for expanding the dynamic range); and
the correction unit is adapted to correct the temperature-related measurement errors of the first depth images (
D
S
) and the second depth images (
D
L
) each separately ([0221] Note that, in a case where the imaging by long-time exposure is not saturated, there is a high possibility that the imaging by short-time exposure is not saturated either. Therefore, the detection signals obtained by the long-time exposure imaging and the detection signals obtained by the short-time exposure imaging may be combined by a predetermined method, such as multiplying each of the detection signals by a weighting coefficient and adding up the resultant detection signals, for example. The distance D may then be calculated from the combined signal Examiner Note: weighting coefficient could be combined with the temperature-related measurement error correction of Schoenlieb).
Schoenlieb (as previously modified by Marra and Dehghani), and Ono are considered to be analogous to the claimed invention because they are all in the same field of time-of-flight (ToF) depth imaging and sensor calibration. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the ToF system of Schoenlieb (as previously modified by Marra and Dehghani) to include the alternating short and long exposure time acquisition and combination as taught by Ono with a reasonable expectation of success. This modification would have been motivated by the desire to increase the dynamic range of the depth images while maintaining accurate temperature-related error correction. By integrating Ono’s teaching of successively capturing images with different exposure times into the system of Schoenlieb (as previously modified by Marra and Dehghani), the system can generate depth images from combined exposure data that are independently corrected for temperature-dependent drifts. A person of ordinary skill in the art would recognize that applying temperature correction to separate depth images obtained from different exposure times would yield the predictable result of more accurate and reliable depth measurements in environments with high dynamic range.
Regarding Claim 28, Schoenlieb is not relied upon as teaching that the correction unit is adapted to perform the correcting additionally in dependence of the respective exposure time (
e
x
p
S
;
e
x
p
L
).
However, Ono teaches that the correction unit is adapted to perform the correcting additionally in dependence of the respective exposure time (
e
x
p
S
;
e
x
p
L
) ([0200]-[0201] An electronic shutter operation or the like is performed to adjust the exposure time by controlling the charge accumulation period in the photodiodes and optimize sensitivity, in accordance with a change in external light or the like. Further, for image sensors, a technique for successively capturing a plurality of images with different exposure times and combining the images is known as a method for expanding the dynamic range. That is, this is a technique for generating one image by performing a combining process in which a long-time exposure image and a short-time exposure image are successively and individually captured, the long-time exposure image is used for a dark image region, and the short-time exposure image is used for a bright image region that is likely to be overexposed in a long-time exposure image. By combining a plurality of different exposure images in this manner, it is possible to obtain an image with a wide dynamic range without overexposure, or a high dynamic range image (HDR image)).
Schoenlieb (as previously modified by Marra and Dehghani), and Ono are considered to be analogous to the claimed invention because they are all in the same field of time-of-flight (ToF) depth imaging and sensor calibration. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the correction unit of Schoenlieb (as previously modified by Marra and Dehghani) to perform the correcting additionally in dependence of the respective exposure time as taught by Ono with a reasonable expectation of success. This modification would have been motivated by the desire to improve the precision of temperature-related error correction by accounting for the specific operational settings used to capture the images. By integrating Ono’s teaching of adjusting sensitivity and processing based on different charge accumulation periods (exposure time) into the existing system, the system can more accurately refine the temperature compensation for each specific subframe. A person of ordinary skill in the art would recognize that correlating temperature correction factors with the exposure time sued for each image would yield the predictable result of further enhanced accuracy and stability in the final, combined depth images.
Prior Art Made of Record
This prior art made of record and not relied upon is considered pertinent to applicant's disclosure: Miura et al. (US 2020/0028326 A1) and Moore et al. (US 2013/0328894 A1)
Allowable Subject Matter
Claims 16-19, 23-24, 29, and 32-34 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
The prior art made of record does not teach or disclose the following limitations in the following claims:
Regarding Claims 16, 23, and 24, weighting a sensor unit temperature difference (
Δ
T
S
e
n
) between the current temperature (
T
S
e
n
) and a reference temperature (
T
S
e
n
r
e
f
) at the sensor unit with a sensor unit temperature correction coefficient (
c
T
S
e
n
) and/or weighting an illumination unit temperature difference (
Δ
T
I
L
L
) between the current temperature at the illumination unit (
T
I
L
L
) and a reference temperature (
T
I
L
L
r
e
f
) at the illumination unit with an illumination unit temperature correction coefficient (
c
T
I
L
L
).
Regarding Claims 17 and 32, weighting an exposure time difference (
Δ
e
x
p
) between the set exposure time (
e
x
p
) and a reference exposure time (
e
x
p
r
e
f
) with an exposure time correction coefficient (
c
e
x
p
).
Regarding Claim 18 and 33, weighting a frame rate difference (
Δ
f
r
) between the set frame rate (
f
r
) and a reference frame rate (
f
r
r
e
f
) with a frame rate correction coefficient (
c
f
r
).
Regarding Claims 19 and 34, weighting a product of an exposure time difference (
Δ
e
x
p
) between the set exposure time (
e
x
p
) and a reference exposure time (
e
x
p
r
e
f
) and a frame rate difference (
Δ
f
r
) between the set frame rate (
f
r
) and a reference frame rate (
f
r
r
e
f
) with an exposure frame rate correction coefficient (
c
e
x
p
,
f
r
).
Regarding Claim 29, the correction unit is adapted to perform the correcting additionally in dependence of the respective difference (
Δ
e
x
p
L
S
) between the longer exposure time (
e
x
p
L
) and the shorter exposure time (
e
x
p
S
).
Claims 20 and 25-26 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
The prior art made of record does not teach or disclose the following limitations in the following claims:
Regarding Claim 20, using:
the weighted sensor unit temperature difference (
c
T
S
e
n
∙
Δ
T
S
e
n
);
the weighted illumination unit temperature difference (
c
T
I
L
L
∙
Δ
T
I
L
L
);
the weighted exposure time difference (
c
e
x
p
∙
Δ
e
x
p
);
the weighted frame rate difference (
c
f
r
∙
Δ
f
r
); and/or
the weighted product (
c
e
x
p
,
f
r
∙
Δ
e
x
p
∙
Δ
f
r
) of the exposure time difference (
Δ
e
x
p
) and the frame rate difference (
Δ
f
r
) each as an offset value of the generated depth images (
D
).
Regarding Claim 25, using the weighted illumination unit temperature difference (
c
T
I
L
L
∙
Δ
T
I
L
L
).
Regarding Claim 26, the weighted sensor unit temperature difference (
c
T
S
e
n
∙
Δ
T
S
e
n
) each as an offset value of the generated depth images (
D
).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EVAN H HAUT whose telephone number is (571)272-7927. The examiner can normally be reached Monday-Thursday 10am-3pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Helal Algahaim can be reached at (571) 272-9358. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/E.H.H./Patent Examiner, Art Unit 3645
/HELAL A ALGAHAIM/SPE , Art Unit 3645
Prosecution Insights