Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. KR10-2023-0008689, filed on 01/20/2023.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 12/07/2023 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
The drawings are objected to because Figure 4 fails to visually distinguish between the “Reference signal” and the “Distorted signal” on the graph, and Figure 5 fails to visually distinguish between “When MPI is present” and “When MPI is not present” on the graph.
The drawings are objected to under 37 CFR 1.83(a). The drawings must show every feature of the invention specified in the claims. Therefore, the subject matters of claims 1-5 must be shown or the feature(s) canceled from the claim(s). No new matter should be entered. See rejections under 112(a) below.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The specification is objected to as failing to provide proper antecedent basis for the claimed subject matter. See 37 CFR 1.75(d)(1) and MPEP § 608.01(o). Correction of the following is required: See rejections under 112(a) below.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a) and 112(b):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-5 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Regarding claim 1, although reference is made in Para [0048]-[0061], particularly Para [0048], [0054], and [0061] of the specification to “calculating a predicted amplitude of the reflection modulation signal using a value of the offset“, the specification does not show possession to the claimed subject matter at there is no level of detail to describe the process to satisfy the written description. As the process is not described, the limitation is therefore not predictable in the art.
Claims 2-5 are rejected due to claim dependency.
Claims 8 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 8, the limitation "substituting one of the intensity of the direct reflection path and the intensity of the indirect reflection path" is unclear. There is insufficient antecedent basis for this limitation in the claim. Which should be considered as “the intensity” for the “direct reflection path” and the “indirect reflection path”?
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5 are rejected under 35 U.S.C. 103 as being unpatentable over Schweizer et al. (US 20120033045 A1, "Schweizer") in view of Hitomi (WO 2022058280 A1, "Hitomi”) and Ott (US 20220066004 A1, “Ott”).
Regarding claim 1, Schweizer teaches a method of detecting a multi-path interference component in a time-of-flight (ToF) camera, the method comprising (Schweizer, Para [0032]-[0033], Fig 4, where the detecting pixels calculate distance R through time of flight):
(Schweizer, Para [0043], Fig 4, where the modulation frequencies from a multipath light source are 15 and 30 MHz);
calculating a measured amplitude and an offset of the collected reflection modulation signal (Schweizer, Para [0039]-[0040], Fig 5A-5B, where the amplitude A and offset B are calculated using the sampled modulations);
calculating a predicted amplitude of the reflection modulation signal using a value of the offset (Schweizer, Para [0057], Fig 5A-5B, where the error from the indirect measurement (measured amplitude) and therefore offset value is compared to a phase and amplitude of a direct measurement having an assumed amplitude (predicted amplitude) as disclosed in Para [0070]);
determining whether the multi-path interference component is included in the collected reflection modulation signal by comparing the predicted amplitude and the measured amplitude (Schweizer, Para [0056]-[0057], where the error included in the stray light and thus interference vector is caused by the indirect measurement and dependent on a comparison with the indirect (measured) amplitude and the direct (predicted) amplitude).
However, Schweizer does not teach collecting a reflection modulation signal emitted from a light source that returns to the ToF camera after being reflected by a subject at a plurality of different times,
generating distortion image data according to a difference between the offset and the measured amplitude of the measured reflection modulation signal; and
generating a confidence map as a reciprocal number of the distortion image data.
On the other hand, Hitomi teaches a multitude of demodulation signals in a ToF system at different phases and frames and therefore different times (Hitomi, Pp. 13 lin. 15- 26, Fig 4, where modulation signal LMS consists of four demodulation signals in an indirect ToF system, each captured with four different frames and therefore a plurality of different times)
Accordingly, it would have been obvious of one of ordinary skill in the art, before the effective filing date of the invention to have modified the method of detecting a multi-path interference component of Schweizer in view of Hitomi, by using demodulating signals that are shifted to obtain signal depth information (Hitomi, Pp.4 lin. 15-24).
However Schweizer in view of Hitomi still does not teach generating distortion image data according to a difference between the offset and the measured amplitude of the measured reflection modulation signal; and
generating a confidence map as a reciprocal number of the distortion image data.
On the other hand Ott teaches the difference between amplitude and offset to generate distortion data by taking the difference of amplitudes with respect to the offset for detecting distortion. In addition, Ott teaches using a reciprocal of the scaling factor which is a part of the distortion detection as confidence information (Ott, Para [0109] Fig 3 and Fig 23, where the reciprocal of the scaling factor is used as a confidence indicator).
Accordingly, it would have been obvious of one of ordinary skill in the art, before the effective filing date of the invention to have further modified the method of detecting a multi-path interference component of Schweizer in view of Hitomi and Ott by using multi-phase demodulation signals to improve saturation detection via distortion determination (Ott, Para [022] and [0033]).
Regarding claim 2, Schweizer in view of Hitomi and Ott teaches the method of claim 1, wherein the measured amplitude is an amplitude of a first modulation signal of the two modulation signals that is more sensitive to the multi-path interference component of the reflection modulation signal than a second modulation signal of the two modulation signals (Schweizer, Para [0050], Fig 5B, where a camera with modulation frequency of 30 MHz has a smaller non ambiguity range and therefore is more sensitive to interference over a smaller range).
Regarding claim 3, Schweizer in view of Hitomi and Ott teaches the method of claim 2, wherein the first modulation signal has a higher frequency than the second modulation signal (Schweizer, Para [0050], Fig 5B, where a camera with modulation frequency of 30 MHz has a smaller non ambiguity range and therefore is more sensitive to interference over a smaller range. The subsequent modulation frequency is 15 MHz which is smaller than a first modulation frequency of 30 MHz).
Regarding claim 4, Schweizer in view of Hitomi and Ott teaches the method of claim 1, wherein determining whether the multi-path interference component is included in the measured reflection modulation signal comprises determining that multi-path distortion is present in the collected reflection modulation signal when the amplitude of the predicted reflection modulation signal and the amplitude of the measured reflection modulation signal are different from each other (Schweizer, Para [0056]-[0057], Fig 5A-5B, where the error (included in disturbance from stray light) from the indirect measurement (measured amplitude) and therefore offset value is compared to a phase and amplitude of a direct measurement having an assumed amplitude (predicted amplitude) as disclosed in Para [0070]).
Regarding claim 5, Schweizer in view of Hitomi and Ott teaches the method of claim 1, wherein:
the reflection modulation signal comprises four reflection modulation signals each having a phase difference of 90° each other (Schweizer, Para [0052], Fig 5A-5B, where the phase of a modulated signal is reconstructed based on four samples with 90 degree separation), and
the amplitude of the reflection modulation signal and the offset of the reflection modulation signal are calculated using intensities of the four reflection modulation signals (Hitomi, Pp. 15 lin. 29-31, Fig 2 and Fig 5, where the phase amplitude in respect to a spatial profile corresponds to an intensity profile).
Claims 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Schweizer, Freedman et al. (US 20150193938 A1, “Freedman”), Hitomi, and Ebrahimi Afrouzi et al. (US 20200225673 A1, “Ebrahimi Afrouzi”)
Regarding claim 6, Schweizer teaches a method of correcting a multi-path interference component in a time-of-flight (ToF) camera, the method comprising (Schweizer, Para [0032]-[0033], Fig 4, where the detecting pixels calculate distance R through time of flight):
collecting a reflection modulation signal corresponding to a modulation signal emitted by a light source towards a subject when the reflection modulation signal returns to the ToF camera after being reflected by the subject (Schweizer, Para [0032]-[0033], Fig 4, where the detecting pixels calculate distance R through time of flight and therefore a reflected modulated signal as disclosed in);
separating a direct reflection component produced via a direct reflection path, which is included in the reflection modulation signal, and an indirect reflection component produced via an indirect reflection path, which is included in the reflection modulation signal, from each other (Schweizer, Para [0057], Fig 5A-5B, where for modulated light ML1A and ML1B there is an indirect and direct measurement path);
generating a dual path model by generating a cost function comprising a data term and a normalization term, based on the direct reflection path and the indirect reflection path (Freedman, Para [0040], Fig 4, where measurement values 404 in a mapping process 402 are mapped to normalized measurements (normalization term) which computes backscattering values 416 (data term)) ;
On the other hand Schweizer does not teach optimizing the dual path model by calculating values of a plurality of variables that minimize the cost function; and
generating a corrected depth map by applying the values of plurality of variables to a depth image comprising the direct reflection component and the indirect reflection component,
wherein the data term is a term for calculating a difference between k-th frame data that is measured by the ToF camera using the direct reflection component and the indirect reflection component as a model and reconstructed k-th frame data, where k in a natural number, and
wherein the normalization term comprises a value that is determined according to an amplitude of the direct reflection path and a length of the direct reflection path, after applying a total variation (TV) algorithm to the amplitude of the direct reflection path and the length of the direct reflection path and then applying penalty constants to the amplitude of the direct reflection path and the length of the direct reflection path, respectively.
On the other hand Freedman teaches a dual path model using normalized measurements and backscattering values (Freedman, Para [0040], Fig 4, where measurement values 404 in a mapping process 402 are mapped to normalized measurements (normalization term) which computes backscattering values 416 (data term) . The values are composed of amplitude and phase values) to optimize and therefore minimize the cost to the model (Freedman, Para [0041], Fig 4, where the multipath computation 414 a plurality of measurement values are searched in an optimization process), to generate a depth map (Freedman, Para [0036], Fig 3-4, where the resulting depth map 314 is a corrected depth map with corrected depth values as disclosed in Para [0027]) with indexed measurement values to tell the difference between direct and indirect frame data (Freedman, Para [0040], Fig 4, where a single index in a look up table is a measurement value in a time of flight camera. By separating the indices to discrete 4D measurement values, indirect and direct frame data are differentiated).
Accordingly, it would have been obvious of one of ordinary skill in the art, before the effective filing date of the invention to have modified the method of detecting a multi-path interference component of Schweizer in view of Freedman by using an optimization process to generate more accurate depth values (Freedman, Para [0036]).
However, Schweizer in view of Freedman still does not teach wherein the data term is a term for calculating a difference between k-th frame data
direct reflection path and then applying penalty constants to the amplitude of the direct reflection path and the length of the direct reflection path, respectively.
On the other hand Hitomi teaches k-th frame data through image data of different phase shifted signals in a frame (Hitomi, Pp. 12 lin. 25-28, Fig 3, where image data includes image data of one frame, where a multi-phase sensor outputs image data for different phase shifted signals).
Accordingly, it would have been obvious of one of ordinary skill in the art, before the effective filing date of the invention to have further modified the method of detecting a multi-path interference component of Schweizer in view of Freedman and Hitomi by using demodulating signals that are shifted to obtain signal depth information (Hitomi, Pp.4 lin. 15-24).
However, Schweizer in view of Freedman and Hitomi does not teach
On the other hand, Ebrahimi Afrouzi teaches the use of total variation to minimize distance with a penalty term, where an Euler-Lagrange is used for minimization with the amplitude and phase value disclosed in Freedman (Ebrahimi Afrouzi, Para [0252], Fig 65, where total variation is used for minimization problem and lambda is the penalty term).
Accordingly, it would have been obvious of one of ordinary skill in the art, before the effective filing date of the invention to have further modified the method of detecting a multi-path interference component of Schweizer in view of Freedman, Hitomi and Ebrahimi Afrouzi by using total variation and an Euler-Lagrange equation to remove noise from an image while maintaining the integrity, detail, and structure of the image (Ebrahimi Afrouzim, Para [0252])
Regarding claim 7, Schweizer in view of Freedman, Hitomi, and Ebrahimi Afrouzi teaches the method of claim 6, wherein the reconstructed k-th frame data (Hitomi, Pp. 12 lin. 25-28, Fig 3, where image data includes image data of one frame, where a multi-phase sensor outputs image data for different phase shifted signals) is an equation for representing the amplitude and the length of the direct reflection path, an amplitude and a length of the indirect reflection path, and a wavelength and a phase shift of the k-th frame by using an Euler’s formula periodic function (Ebrahimi Afrouzi, Para [0252], Fig 65, where the Euler-Lagrange equation can be used for each amplitude and phase value as disclosed in Freedman, therefore making it periodic).
Claims 8 is rejected under 35 U.S.C. 103 as being unpatentable over Schweizer, Freedman, Hitomi, Ebrahimi Afrouzi and Semenov et al. (RU 2778496 C1, “Semenov”).
Regarding Claim 8, Schweizer, Freedman, Hitomi and Ebrahimi Afrouzi teaches (Hitomi, Pp. 13 lin. 15- 26, Fig 4, where modulation signal LMS consists of four demodulation signals in an indirect ToF system, each captured with four different frames and therefore a plurality of different times).
However, Schweizer, Freedman, Hitomi and Ebrahimi Afrouzi does not teach the method of claim 7, wherein optimizing the dual path model further comprises substituting one of the intensity of the direct reflection path and the intensity of the indirect reflection path with a relation equation between the other of the intensity of the direct reflection path and the intensity of the indirect reflection path and
On the other hand, Semenov teaches the substitution of a direct path intensity for an indirect path intensity (Semenov, Para [0124]-[0129], Fig 3A-3D, where the intensity of a pixel assigned to be from a direct path is substituted into an indirect path intensity).
Accordingly, it would have been obvious of one of ordinary skill in the art, before the effective filing date of the invention to have further modified the method of detecting a multi-path interference component of Schweizer in view of Freedman, Hitomi, Ebrahimi Afrouzi and Semenov, to substitute intensities to improve image quality (Semenov, Para [0019] and [0124]-[0129].
Claims 9 is rejected under 35 U.S.C. 103 as being unpatentable over Schweizer, Ott, Freedman, Schafer et al. (US 20210166124 A1, “Schafer”) and Hitomi.
Regarding claim 9, Schweizer teaches, a method of correcting a multi-path interference component in a time-of-flight (ToF) camera, the method comprising (Schweizer, Para [0032]-[0033], Fig 4, where the detecting pixels calculate distance R through time of flight):
wherein the emitted modulation signal comprises two modulation signals having respective different frequencies (Schweizer, Para [0043], Fig 4, where the modulation frequencies from a multipath light source are 15 and 30 MHz), and
(Schweizer, Para [0039]-[0040], Fig 5A-5B, where the amplitude A and offset B are calculated using the sampled modulations) (Schweizer, Para [0056]-[0057], where the error included in the stray light and thus interference vector is caused by the indirect measurement and dependent on a comparison with the indirect (measured) amplitude and the direct (predicted) amplitude),
However, Schweizer does not teach generating a final depth map by applying a confidence map to a depth map and an unwrapped map,
wherein the depth map is generated by calculating variables that minimize a cost function comprising a data term and a normalization term, based on a reflection modulation signal that comprises a direct reflection component and an indirect reflection component and that is generated when a modulation signal that is emitted from a light source to a subject returns to the ToF camera, and applying the variables to the depth image comprising the direct reflection component and the indirect reflection component,
wherein the unwrapped map comprises the direct reflection component and the indirect reflection component,
wherein the confidence map is calculated as a reciprocal number of distortion image data,
On the other hand Ott teaches the difference between amplitude and offset to generate distortion data by taking the difference of amplitudes with respect to the offset for detecting distortion. In addition, Ott teaches using a reciprocal of the scaling factor which is a part of the distortion detection as confidence information (Ott, Para [0109] Fig 3 and Fig 23, where the reciprocal of the scaling factor is used as a confidence indicator).
Accordingly, it would have been obvious of one of ordinary skill in the art, before the effective filing date of the invention to have modified the method of detecting a multi-path interference component of Schweizer in view of Ott by using multi-phase demodulation signals to improve saturation detection via distortion determination (Ott, Para [022] and [0033]).
However, Schweizer in view of Ott still does not teach
wherein the depth map is generated by calculating variables that minimize a cost function comprising a data term and a normalization term, based on a reflection modulation signal that comprises a direct reflection component and an indirect reflection component and that is generated when a modulation signal that is emitted from a light source to a subject returns to the ToF camera, and applying the variables to the depth image comprising the direct reflection component and the indirect reflection component,
wherein the unwrapped map comprises the direct reflection component and the indirect reflection component,
On the other hand, Schafer teaches an unwrapped map which when combined with Schweizer is with direct and indirect components (Schafer, Para [0085], Fig 9, where modulation frequencies are used to obtain unwrapped depth, which when combined with Schweizer obtains unwrapped depth with direct and indirect frequencies),
Accordingly, it would have been obvious of one of ordinary skill in the art, before the effective filing date of the invention to have further modified the method of detecting a multi-path interference component of Schweizer in view of Ott and Schafer by unwrapping modulation frequencies to achieve performance comparable with single frequency ToF for depth estimation (Schafer, Para [0085]).
However, Schweizer in view of Ott and Schafer still does not teach wherein the depth map is generated by calculating variables that minimize a cost function comprising a data term and a normalization term, based on a reflection modulation signal that comprises a direct reflection component and an indirect reflection component and that is generated when a modulation signal that is emitted from a light source to a subject returns to the ToF camera, and applying the variables to the depth image comprising the direct reflection component and the indirect reflection component,
On the other hand, Hitomi teaches a multitude of demodulation signals in a ToF system at different phases and frames and therefore different times (Hitomi, Pp. 13 lin. 15- 26, Fig 4, where modulation signal LMS consists of four demodulation signals in an indirect ToF system, each captured with four different frames and therefore a plurality of different times)
Accordingly, it would have been obvious of one of ordinary skill in the art, before the effective filing date of the invention to have further modified the method of detecting a multi-path interference component of Schweizer in view of Ott, Schafer and Hitomi, by using demodulating signals that are shifted to obtain signal depth information (Hitomi, Pp.4 lin. 15-24).
However, Schweizer in view of Ott, Schafer and Hitomi still does not teach wherein the depth map is generated by calculating variables that minimize a cost function comprising a data term and a normalization term, based on a reflection modulation signal that comprises a direct reflection component and an indirect reflection component and that is generated when a modulation signal that is emitted from a light source to a subject returns to the ToF camera, and applying the variables to the depth image comprising the direct reflection component and the indirect reflection component,
On the other hand Freedman teaches a dual path model using normalized measurements and backscattering values (Freedman, Para [0040], Fig 4, where measurement values 404 in a mapping process 402 are mapped to normalized measurements (normalization term) which computes backscattering values 416 (data term) . The values are composed of amplitude and phase values) to optimize and therefore minimize the cost to the model (Freedman, Para [0041], Fig 4, where the multipath computation 414 a plurality of measurement values are searched in an optimization process), to generate a depth map (Freedman, Para [0036], Fig 3-4, where the resulting depth map 314 is a corrected depth map with corrected depth values as disclosed in Para [0027]) with indexed measurement values to tell the difference between direct and indirect frame data (Freedman, Para [0040], Fig 4, where a single index in a look up table is a measurement value in a time of flight camera. By separating the indices to discrete 4D measurement values, indirect and direct frame data are differentiated).
Accordingly, it would have been obvious of one of ordinary skill in the art, before the effective filing date of the invention to have modified the method of detecting a multi-path interference component of Schweizer in view of Ott, Schafer, Hitomi and Freedman by using an optimization process to generate more accurate depth values (Freedman, Para [0036]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZAKI HAWKINS whose telephone number is (571)272-6595. The examiner can normally be reached Monday-Friday 7:30am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, YUQING XIAO can be reached at (571) 270-3603. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZAKI KEHINDE HAWKINS/ Examiner, Art Unit 3645
/YUQING XIAO/ Supervisory Patent Examiner, Art Unit 3645
Prosecution Insights