DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment and Argument
Applicant’s amendment and argument with respect to pending claims 1-20 filed on 04/21/2026 have been fully considered but the argument regarding the rejection under 35 USC § 102/103 of the claims has been rendered moot in view of a new ground(s) of rejection necessitated by the amendment of the pending claims. Examiners response to the applicant’s argument that are not rendered moot follows below.
Specification
In view of the amendment of the title of the current application, the objection to the specification of the current application is withdrawn.
Claim Rejections - 35 USC § 112
In view of the amendment of the dependent claims 4, 5, 8 and 13, the rejection under 35 USC § 112(b) of the claims is withdrawn.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-6, 8 and 10-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over KWON et al. (US 20180068424 A1) in view of Schmidt (US 20180211398 A1).
Regarding claim 1, KWON et al. (US 20180068424 A1) teaches the claim as follows: An electronic device comprising circuitry configured obtain a reflectance image according to an indirect Time-of-Flight (iToF) principle, the reflectance image (¶0033-0036: the ToF measuring apparatus 1 generates a depth image by calculating a phase of measured light reflected by the subject 2) comprising reflectance values each determined based on a depth value (¶0040: the processor 10 generates an IR image that corresponds to each of the phases based on sensed intensities of IR rays, calculates ToFs by using the IR images of each of the phases, and generates a depth image with depth values based on the calculated ToFs); and apply a reflectance sharpening filter to the reflectance image to obtain a filtered reflectance value for a pixel of the reflectance image (¶0064: the image sensor 50 of the ToF measuring apparatus 1 acquires IR images of different phases of the scene captured for generating a depth image. The corrector 130 of the processor 10 corrects (i.e., adjusts) pixel values of the IR images by using the previously acquired (stored) sharpening filter. At this time, the corrector 130 may perform correction (or an adjustment) by performing a convolution operation on the pixel values of the IR images using the sharpening filter. T0065: the corrected (i.e., adjusted) pixel values).
Kwon does not disclose reflectance values each determined based on…a confidence value of a respective pixel.
However, Schmidt teaches reflectance values each determined based on…a confidence value of a respective pixel (¶0021: One or more raw depth images are captured using a time-of-flight camera of an electronic device…A confidence value representing a corresponding validity of a depth represented by a first pixel of the depth frame is determined based on a comparison of the amplitude energy value of a corresponding first pixel of the amplitude frame to the first threshold. The confidence value, which represents whether the pixel from the depth frame/amplitude frame is to be used to determine a depth of a corresponding spatial feature/can be used to filter/screen pixels of the amplitude frame).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified Kwon’s tof measuring apparatus/method by incorporating the teaching of Schmidt as noted above, in order to suppress stray light artifacts (Schmidt: ¶0017-0020).
Regarding claim 2, KWON in view of Schmidt teaches the electronic device of claim 1. Schmidt further teaches wherein the circuitry is configured to decide based on the filtered reflectance value of the pixel whether a depth measurement of the pixel is false or not (¶0016: dynamically parameterizing and applying a filter for invalidating depth data affected by light distortion for image-based identification of objects in a local environment of an electronic device. ¶0069: If the amplitude energy value of the first pixel of the amplitude frame is less than the threshold value, the corresponding pixel in the confidence array is identified with a value indicating a lack of confidence in the depth measurement represented by the depth value of the corresponding first pixel of the depth frame …For instance, a pixel may be identified as invalid when the amplitude energy value of the first pixel of the amplitude frame is less than the first energy threshold). The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 3, KWON in view of Schmidt teaches the electronic device of claim 2. Schmidt further teaches wherein the circuitry is configured to decide that a depth measurement of the pixel is false if the filtered reflectance value of the pixel is below zero (¶0016: dynamically parameterizing and applying a filter for invalidating depth data affected by light distortion for image-based identification of objects in a local environment of an electronic device. ¶0069: If the amplitude energy value of the first pixel of the amplitude frame is less than the threshold value, the corresponding pixel in the confidence array is identified with a value indicating a lack of confidence in the depth measurement represented by the depth value of the corresponding first pixel of the depth frame …For instance, a pixel may be identified as invalid when the amplitude energy value of the first pixel of the amplitude frame is less than the first energy threshold). The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 4, KWON in view of Schmidt teaches the electronic device of claim 1. Schmidt further teaches wherein the circuitry is configured to determine a confidence value for the pixel, and to decide whether a depth measurement of the pixel is false or not based on the filtered reflectance value of the pixel and based on the confidence value of the pixel (¶0016: dynamically parameterizing and applying a filter for invalidating depth data affected by light distortion for image-based identification of objects in a local environment of an electronic device. ¶0069: If the amplitude energy value of the first pixel of the amplitude frame is less than the threshold value, the corresponding pixel in the confidence array is identified with a value indicating a lack of confidence in the depth measurement represented by the depth value of the corresponding first pixel of the depth frame …For instance, a pixel may be identified as invalid when the amplitude energy value of the first pixel of the amplitude frame is less than the first energy threshold). The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 5, KWON in view of Schmidt teaches the electronic device of claim 4. Schmidt further teaches wherein the circuitry is configured to decide that a depth measurement of the pixel is false if the confidence value of the pixel is below a predetermined threshold and if the filtered reflectance value of the pixel is below zero (¶0016: dynamically parameterizing and applying a filter for invalidating depth data affected by light distortion for image-based identification of objects in a local environment of an electronic device. ¶0069: If the amplitude energy value of the first pixel of the amplitude frame is less than the threshold value, the corresponding pixel in the confidence array is identified with a value indicating a lack of confidence in the depth measurement represented by the depth value of the corresponding first pixel of the depth frame …For instance, a pixel may be identified as invalid when the amplitude energy value of the first pixel of the amplitude frame is less than the first energy threshold). The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 6, KWON in view of Schmidt teaches the electronic device of claim 1. Schmidt further teaches wherein the circuitry is configured to invalidate a depth measurement of the pixel based on the filtered reflectance value of the pixel (¶0016: dynamically parameterizing and applying a filter for invalidating depth data affected by light distortion for image-based identification of objects in a local environment of an electronic device. ¶0069: If the amplitude energy value of the first pixel of the amplitude frame is less than the threshold value, the corresponding pixel in the confidence array is identified with a value indicating a lack of confidence in the depth measurement represented by the depth value of the corresponding first pixel of the depth frame …For instance, a pixel may be identified as invalid when the amplitude energy value of the first pixel of the amplitude frame is less than the first energy threshold). The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 8, KWON in view of Schmidt teaches the electronic device of claim 1. Schmidt further teaches wherein applying the sharpening filter to the reflectance image comprises determining a mean reflectance value of pixels of the reflectance image in the neighborhood of the pixel (¶0064-0066: After the amplitude frame and depth frame are determined, the 2D processor 502 determines whether to update a threshold used to characterize the confidence level for the depth data filter at block 606…When the threshold is to be updated, at block 608 the 2D processor 502 establishes an energy threshold based on the amplitude energy values of the amplitude frame…the estimated pixel energy value is an average amplitude energy for the sum of amplitude energy values of the pixels of the amplitude frame. Figs. 7-10 illustrating a pixel array of the amplitude frame). The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 10, KWON teaches the electronic device of claim 1, wherein the circuitry is further configured to identify spots captured by an iToF sensor, and wherein each pixel of the reflectance image is associated with a respective spot of the spots captured by the iToF sensor (¶0064-0066: The ToF calculator 140 acquires depth values of the scene by calculating ToFs of respective pixels based on the corrected (i.e., adjusted) pixel values). Note that this interpretation is consistent with the disclosure of the current application. See the current application page 5.
Regarding claim 11, KWON teaches the electronic device of claim 1, wherein the circuitry is further configured to identify spots captured by an iToF sensor (¶0064-0066: The ToF calculator 140 acquires depth values of the scene by calculating ToFs of respective pixels based on the corrected (i.e., adjusted) pixel values). Note that this interpretation is consistent with the disclosure of the current application. See the current application page 5, wherein the pixel is a spot peak pixel of a respective spot of the spots captured by the iToF sensor (¶0072: an intensity distribution of IR light sensed by the pixel array of the image sensor 50 may be as shown in FIG. 7. A region 700 showing the highest intensity of IR light may be pixels of columns disposed between pixels of about a 200th column and pixels of about a 300th column on the x axis), and wherein the kernel of the reflectance sharpening filter comprises a predetermined number of spots wherein each spot corresponds to a spot peak pixel (¶0084: FIG. 13, a value of each pixel of an IR image 1300 corresponds to an intensity of reflected light sensed by each pixel of the image sensor 50. The corrector 130 corrects (i.e., adjusts) pixel values of the IR image 1300 by performing a convolution operation on the pixel values of the IR image 1300 by using a sharpening filter 1310. Although it is assumed in FIG. 13 that the sharpening filter 1310 is implemented as a 3×3 matrix for convenience of description...).
Regarding claim 12, KWON in view of Schmidt teaches the electronic device of claim 10. Schmidt further teaches wherein the circuitry is configured to invalidate all depth measurements related to a spot of the spots captured by an iToF sensor based on the filtered reflectance value of the pixel (¶0064-0066: After the amplitude frame and depth frame are determined, the 2D processor 502 determines whether to update a threshold used to characterize the confidence level for the depth data filter at block 606…When the threshold is to be updated, at block 608 the 2D processor 502 establishes an energy threshold based on the amplitude energy values of the amplitude frame…the estimated pixel energy value is an average amplitude energy for the sum of amplitude energy values of the pixels of the amplitude frame. Figs. 7-10 illustrating a pixel array of the amplitude frame). The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 13, KWON in view of Schmidt teaches the electronic device of claim 11. Schmidt further teaches wherein the circuitry is configured to determine a confidence value for the spot peak pixel, and to decide whether a depth measurement of the spot peak pixel is false or not if the confidence value is below a predetermined threshold and the filtered reflectance value of the spot peak pixel is below zero (¶0064-0066: After the amplitude frame and depth frame are determined, the 2D processor 502 determines whether to update a threshold used to characterize the confidence level for the depth data filter at block 606…When the threshold is to be updated, at block 608 the 2D processor 502 establishes an energy threshold based on the amplitude energy values of the amplitude frame…the estimated pixel energy value is an average amplitude energy for the sum of amplitude energy values of the pixels of the amplitude frame. Figs. 7-10 illustrating a pixel array of the amplitude frame). The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 14, KWON teaches the electronic device of claim 1, which further comprises an image sensor (Figs. 1: image sensor 50)
Regarding claim 15, KWON teaches the electronic device of claim 1, which further comprises a spot illuminator (Figs. 1: a light source 20).
Claim 16 drawn to a method claim and recites the limitation analogous to claim 1, and is rejected due to the same reason set forth above with respect to claim 1.
Claim(s) 7 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over KWON et al. (US 20180068424 A1) in view of Schmidt (US 20180211398 A1) as applied to claim 1, and further in view of Vianello et al. "Depth images super-resolution: An iterative approach," 2014 IEEE International Conference on Image Processing (ICIP), Paris, France, 2014, pp. 3778-3782.
Regarding claim 7, KWON teaches the electronic device of claim 1, wherein the filtered reflectance value of the pixel is determined based on the reflectance values of pixels in the reflectance image (¶0064: The corrector 130 of the processor 10 corrects (i.e., adjusts) pixel values of the IR images by using the previously acquired (stored) sharpening filter. At this time, the corrector 130 may perform correction (or an adjustment) by performing a convolution operation on the pixel values of the IR images using the sharpening filter. ¶0065: …the corrected (i.e., adjusted) pixel values).
KWON in view of Schmidt does not explicitly disclose the filtered reflectance value of the pixel is determined based on … a predetermined sharpening factor.
However, Vianello teaches the filtered reflectance value of the pixel is determined based on … a predetermined sharpening factor (abstract, Section 5: weighted joint bilateral filter (WJBF) calculated using weighing factor α).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify KWON by incorporating the teaching of Vianello as noted above, in order to obtain a high resolution depth image (Vianello: Section 9).
Regarding claim 18, Kwon teaches the electronic device of claim 1, wherein the filtered reflectance value is determined based on reflectance values of pixels in the neighborhood of the pixel (¶0060: the sharpening filter processor 120 negates intensity values of the focus pixel and the nearby pixels defined by the point spread function…and acquires a sharpening filter based on the replacement value of the focus pixel and the negated intensity values of the nearby pixels. ¶0061: The sharpening filter is intended to compensate the focus pixel for intensities of IR light reflected to nearby pixels, wherein the intensities are defined by the point spread function).
KWON in view of Schmidt does not explicitly disclose the filtered reflectance value is determined based on…a predetermined sharpening factor.
However, Vianello teaches the filtered reflectance value is determined based on…a predetermined sharpening factor (abstract, Section 5: weighted joint bilateral filter (WJBF) calculated using weighing factor α).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify KWON by incorporating the teaching of Vianello as noted above, in order to obtain a high resolution depth image (Vianello: Section 9).
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over KWON et al. (US 20180068424 A1) in view of Schmidt (US 20180211398 A1) as applied to claim 1, and further in view of Haker et al. "Geometric Invariants for Facial Feature Tracking with 3D TOF Cameras," 2007 International Symposium on Signals, Circuits and Systems, Iasi, Romania, 2007, pp. 1-4.
Regarding claim 19, KWON in view of Schmidt teaches the electronic device of claim 1, KWON teaches wherein each reflectance value of the reflectance image is determined based on a depth value (¶0040: the processor 10 generates an IR image that corresponds to each of the phases based on sensed intensities of IR rays, calculates ToFs by using the IR images of each of the phases, and generates a depth image with depth values based on the calculated ToFs).
Furthermore, Schmidt teaches wherein each reflectance value of the reflectance image is determined based on… a confidence value of a spot peak pixel of a respective spot (¶0021: One or more raw depth images are captured using a time-of-flight camera of an electronic device…A confidence value representing a corresponding validity of a depth represented by a first pixel of the depth frame is determined based on a comparison of the amplitude energy value of a corresponding first pixel of the amplitude frame to the first threshold. The confidence value, which represents whether the pixel from the depth frame/amplitude frame is to be used to determine a depth of a corresponding spatial feature/can be used to filter/screen pixels of the amplitude frame).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified Kwon’s tof measuring apparatus/method by incorporating the teaching of Schmidt as noted above, in order to suppress stray light artifacts (Schmidt: ¶0017-0020).
KWON in view of Schmidt does not teach wherein the circuitry is further configured to identify spots captured by an iToF sensor by applying a local maximum filter to an amplitude image.
However, Haker teaches wherein the circuitry is further configured to identify spots captured by an iToF sensor by applying a local maximum filter to an amplitude image (Page 2, Section II. Conversely, since the nose tends to be a local maximum in the intensity data, we would expect to find the corresponding pixels in the region labeled peak).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified KWON in view of Schmidt by utilizing the teaching of Haker as noted above, in order to accurately detect peak pixels in the image data (Haker: abstract, Section IV).
Allowable Subject Matter
Claims 9 and 17 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Claim 20 is allowed.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NATHNAEL AYNALEM whose telephone number is (571)270-1482. The examiner can normally be reached M-F 9AM-5:30 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, SATH PERUNGAVOOR can be reached at 571-272-7455. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NATHNAEL AYNALEM/Primary Examiner, Art Unit 2488
Prosecution Insights