Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is the first office action on the merits and is responsive to the papers filed 02/26/2024. Claims 1-20 are currently pending and examined below.
Drawings
The drawings are objected to because in Fig. 2 Step S240 says “generating a depth data,” while the claims and specification use “depth map.”
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
[0017], lines 15-16, “receives a plurality of reflect optical signals … from the through the camera 130.” Should be “receives a plurality of reflect optical signals … through the camera 130.”
[0020], lines 21-22, “the processor 110 photographs the plurality of dot patterns … through the camera 130” should be “the camera 130 photographs the plurality of dot patterns … and the processor 110 generates the plurality of dot images.”
[0030], lines 11-14, “the corresponding depth range is the second distance range P3’ and later says “preset value 3000 corresponds to the second distance range P2” should be “the corresponding depth range is the third distance range P3,” and “preset value 3000 corresponding to the third distance range P3.”
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 5, 7, 11, 12, 15, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Fenton et al. (US 20190079192 A1, “Fenton”) in view of Michael Hall (US 20210383560 A1, “Hall”).
Regarding claim 1, Fenton teaches a depth sensing device (Fig. 1, [0038]), comprising:
a processor (Fenton teaches that the Fig. 1 components may be implemented using a processing unit with associated memory executing computer-executable instructions [0038].);
a dot projector, coupled to the processor, and projects a plurality of amplitude modulated optical signals to a sensing target (Fenton teaches a modulated structured light source 110 that projects structured light onto a scene, where the projected light is also modulated. Fenton teaches the source emits transmitted light modulated at a modulation frequency and that the amount of illumination changes periodically. Fenton further teaches the modulated light is arranged as units, and the unit may be a dot ([0038]- [0039]). Fenton also teaches the structured light source projects a plurality of units, e.g., dots, arranged to form a pattern [0061]); and
a camera, coupled to the processor, and photographs a plurality of reflected light signals from the sensing target to obtain (a plurality of dot) images (Fenton teaches image sensor 120 captures the projected structured light image and that dot detection component 135 detects dots in the captured image [0040]- [0041]. Fenton also teaches reflected structured-light images including rows of reflected dots. Figs. 4-5, [0052]- [0057].).
Fenton fails to explicitly teach
wherein the processor generates a first reference depth map of indirect time-of-flight ranging and a second reference depth map of structured light ranging according to the plurality of dot images, and the processor generates a depth map corresponding to the sensing target according to the first reference depth map and the second reference depth map.
Fenton teaches that the structured light is arranged as units/dots, the reflected structured light image is captured by image sensor 120, a TOF depth map calculation component calculates TOF z-depth for reflected dots/units using modulation/phase information, and a structured light depth map calculation component calculates structured light z depth using triangulation. See Fenton [0038], [0061], [0065]- [0067], and [0072]-[0073].
Fenton does not explicitly teach that a plurality of dot images supplies both the first indirect time-of-flight reference depth map and the second structured light reference depth map, nor does Fenton clearly teach generating the final depth map according to both reference depth maps. Hall teaches capturing images of reflected structured/pulsed light, determining TOF phase shifts using one or more captured images, determining a first set of estimated radial distances from the TOF phase shifts, determining a second estimated radial distance based on structured light encoding and at least one captured image, and selecting the correct distance from the first set based on the second distance. See Hall [0005], [0042]- [0043], and claim 1.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify Fenton’s modulated structured light depth sensing system to use Hall’s structured light assisted TOF disambiguation because Fenton already obtains both TOF depth and structured light depth from reflected modulated structured light, while Hall teaches using structured light encoding from captured images to distinguish between possible TOF depth estimates and select the correct distance. The combination would improve depth determination by resolving TOF phase ambiguity while using structured light information, and Hall teaches that combining structured light encoding with TOF allows quick depth determination, reduced code search complexity, and a smaller baseline without sacrificing sensing range. See Hall [0021], [0042]- [0043], and [0047].
Regarding claim 2, Fenton, in view of Hall, teaches the depth sensing device according to claim 1, wherein the dot projector projects the plurality of amplitude modulated optical signals according to a single modulation frequency.
Fenton teaches that the modulated structured light source outputs structured light that is also modulated light, wherein the transmitted light is modulated with a signal at a modulation frequency and the amount of illumination changes periodically. See Fenton [0039] and [0064]. Hall teaches that a single temporal pulsing frequency may produce multiple TOF depth estimates due to phase ambiguity, and that structured light information may be used to disambiguate the TOF estimate without requiring additional output light frequencies. See Hall [0042] and [0047].
It would have been obvious to configure Fenton’s modulated structured-light source to operate at a single modulation frequency and use Hall’s structured light assisted TOF disambiguation because Hall teaches that structured light encoding can resolve the ambiguity of single frequency TOF measurements without requiring additional output light frequencies, thereby reducing capture complexity while maintaining depth determination accuracy.
Regarding claim 5, Fenton, in view of Hall, teaches the depth sensing device according to claim 1, wherein the plurality of amplitude modulated optical signals form a plurality of dot patterns on the sensing target (Fenton teaches that the modulated structured light source outputs structured light that is also modulated light, and that the modulated light is arranged as units, where the unit in Fig. 1 is described as a dot. See Fenton [0039]. Fenton further teaches that the structured light source projects a plurality of units, e.g., dots, arranged to form a pattern or structure on the scene, and the projected structured light image is captured by a corresponding camera. See Fenton [0061]. Fenton Figs. 4 and 5 further illustrate reflected structured light dot patterns from the scene. See Fenton [0052]- [0057].).
Regarding claim 7, Fenton, in view of Hall, teaches the depth sensing device according to claim 1, wherein the processor executes a phase map generation module and a first reference depth generation module, wherein the phase map generation module generates a phase map according to the plurality of dot images, and the first reference depth generation module generates the first reference depth map of indirect time-of-flight ranging according to the phase map.
Fenton teaches that the processor executes depth processing components, including a TOF depth map calculation component 125, and that the TOF z-depth for reflected dots/units is calculated using modulation/phase information. See Fenton [0038] and [0067]. Hall discloses a structured TOF depth sensor in which sensor 404 captures reflected structured light and controller 412 performs TOF phase-shift calculations. See Hall [0038]- [0040]. Hall further teaches a phase map 500 produced from the structured TOF depth sensor, where the phase map represents detected ranges/estimated radial distances produced from the TOF calculation. See Hall [0044]- [0046]. Hall also teaches that multiple raw intensity images are acquired at different phase offsets and are used to determine aggregate phase. See Hall [0051]- [0053].
It would have been obvious to implement Hall’s phase map generation and phase based TOF depth estimation in Fenton’s TOF depth map calculation component because Fenton already calculates TOF depth from modulated structured light dot reflections using phase/modulation information, and Hall teaches using captured structured light images to generate phase information and corresponding TOF radial distance estimates. The combination would predictably improve Fenton’s TOF depth generation by using Hall’s known phase map/phase shift processing for structured TOF depth sensing.
Claims 11-12, 15, 17 are method claims corresponding to device claims 1-2, 5, 7. They are rejected for the same reasons.
Claims 3-4, 6, 13-14, 16 are rejected under 35 U.S.C. 103 as being unpatentable over Fenton in view of Hall and Bloom et al. (WO 2011088254 A2, “Bloom”).
Regarding claim 3, Fenton, in view of Hall, fails to explicitly teach the depth sensing device according to claim 1, wherein any adjacent two of the plurality of amplitude modulated optical signals have a fixed phase difference.
However, Bloom teaches a structured light system in which a pattern generator generates patterns at different temporal phases. In particular, Bloom [0037] teaches that the temporal phase axis represents brightness variation versus time and that the pattern generator generates four phases in sequence: 0°, 90°, 180°, and 270°. Bloom Fig. 8 illustrates the same 0°, 90°, 180°, and 270° temporal phase relationship. Therefore, Bloom teaches adjacent sequential projected optical patterns/signals having a fixed phase difference of 90 degrees.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify Fenton’s modulated structured light projection system to use Bloom’s fixed phase shifted sequential projected patterns/signals because Bloom teaches that generating structured light patterns at known temporal phase offsets enables recovery of phase information from captured camera images and improves structured light depth detection. Applying known fixed phase offsets to Fenton’s modulated structured light projection would predictably provide phase resolved image data for depth calculation in the combined ToF/structured light system.
Regarding claim 4, Fenton, in view of Hall and Bloom, teaches the depth sensing device according to claim 3, wherein the fixed phase difference is 90 degrees.
Bloom teaches a structured light system in which a pattern generator generates multiple temporal phases in sequence. In particular, Bloom [0037] teaches that the pattern generator generates four phases in sequence and identifies the phases as 0°, 90°, 180°, and 270°. Bloom Fig. 8 likewise illustrates the four temporal phases as 0°, 90°, 180°, and 270°. Thus, Bloom teaches adjacent sequential projected optical signals/patterns having a fixed phase difference of 90 degrees.
It would have been obvious to use Bloom’s 90-degree sequential phase shifted structured light projection with Fenton’s modulated structured light depth sensing device because Bloom teaches that generating known phase shifted patterns permits phase detection from captured camera images and improves structured light depth detection, while Fenton already uses modulated structured light and captured reflected images for ToF and structured light depth determination.
Regarding claim 6, Fenton, in view of Hall, fails to explicitly teach the depth sensing device according to claim 5, wherein the plurality of dot patterns are four dot patterns, and the four dot patterns are sequentially projected on the sensing target.
Fenton discloses that the modulated structured light source outputs structured light that is also modulated light, and the modulated light is arranged as units/dots. See Fenton [0039]. Fenton further teaches projecting a plurality of units, e.g., dots, arranged to form a pattern or structure onto a scene. See Fenton [0061].
Fenton, in view of Hall, fails to explicitly teach that the plurality of dot patterns is four dot patterns sequentially projected onto the sensing target.
However, Bloom [0037] teaches that the pattern generator generates four phases in sequence, specifically 0°, 90°, 180°, and 270°, and Bloom Fig. 8 illustrates the same four temporal phases. Thus, Bloom teaches sequentially generating four structured light patterns/signals.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify Fenton’s modulated dot pattern structured light system to sequentially project four phase shifted dot patterns as taught by Bloom because Bloom teaches that generating four known temporal phases allows phase information to be recovered from captured camera images for structured-light depth detection. Applying Bloom’s known four-phase sequential projection to Fenton’s modulated dot-pattern projector would predictably provide multiple phase-shifted dot-pattern images for improved phase/depth determination.
Claims 13-14, 16 are method claims corresponding to device claims 3-4, 6. They are rejected for the same reasons.
Claims 8, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Fenton in view of Hall and Chan et al. (US 20210209778 A1, “Chan”).
Regarding claim 8, Fenton, in view of Hall, fails to explicitly teach the depth sensing device according to claim 1, wherein the processor executes a dot confidence map generation module and a second reference depth generation module, wherein the dot confidence map generation module generates a dot confidence map according to the plurality of dot images, and the second reference depth generation module generates the second reference depth map of structured light ranging according to the dot confidence map.
Fenton teaches that the processor executes depth processing components, including a TOF depth map calculation component 125, a structured light depth map calculation component 130, a dot identification component 140, and a depth confidence factor component 150. Fenton further teaches that these components may be implemented by a processing unit with memory executing computer-executable instructions. See Fenton [0038]. Fenton also teaches generating structured light z-depth using triangulation based on the reflected structured-light dot/unit and using the structured-light z-depth to build a depth image. See Fenton [0072]- [0073]. Hall teaches capturing multiple raw intensity images/raw data at different phase offsets and using the captured image data for structured TOF depth processing. See Hall [0051]- [0053].
However, Fenton, in view of Hall, does not explicitly teach that the processor executes a dot confidence map generation module that generates a dot confidence map according to the captured dot images, nor does it explicitly teach that the structured light reference depth map is generated according to the dot confidence map.
Chan teaches that the system blocks may be implemented by computer software or an instruction controlled computer and teaches a dot ToF depth generator 14 that generates a dot depth map and an associated dot confidence map according to a captured image. See Chan [0011], [0014]. Chan further teaches a structured light depth decoder 21 configured to generate an SL depth map according to the dot confidence map, wherein a known pattern is projected and a reflected image is captured, and the SL depth map is generated based on geometric triangulation. See Chan [0018]. Therefore, it would have been obvious to include Chan’s dot confidence map generation and SL depth decoder processing in the depth sensing device of Fenton, such that the processor generates a dot confidence map from the captured dot image information and generates the second/structured light reference depth map according to the dot confidence map.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify Fenton’s structured-light/TOF depth-sensing device, as modified by Hall to use multiple captured phase images, to further include Chan’s dot confidence map and SL depth decoder because Fenton already teaches using TOF depth, structured light depth, and confidence information in the same depth-sensing system, while Chan teaches using dot confidence information to generate an SL depth map. The combination would predictably improve the reliability of the structured-light depth map by using confidence information associated with the dot depth data, as taught by Chan.
Claim 18 is method claim corresponding to device claim 8. It is rejected for the same reasons.
Claims 9-10, 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Fenton in view of Hall and Wilhelmus Hendrikus Alfonsus Bruls (US 20140285623 A1, “Bruls”).
Regarding claim 9, Fenton, in view of Hall, fails to explicitly teach the depth sensing device according to claim 1, wherein the processor executes a computing module and a conversion module, wherein the computing module subtracts the first reference depth map and the second reference depth map to generate a depth difference map, and the conversion module converts the depth difference map to the depth map.
Fenton also teaches comparing TOF depth and structured-light depth and using a difference between the two estimates. See Fenton [0048]. However, Fenton and Hall do not explicitly teach a computing module that subtracts the first reference depth map and the second reference depth map to generate a depth difference map, and a conversion module that converts the depth difference map to the final depth map.
Bruls teaches processing first and second depth maps by determining differences between the depth maps and generating a resulting depth map in dependence on the determined depth difference. See Bruls [0008]- [0010], [0053]. Bruls further teaches, in the cascaded embodiment of Fig. 4, subtracting a first/original depth map Z1 from a second/stabilized depth map Za using subtractor 26 to generate a difference depth map Zb, processing the difference depth map to generate stabilized difference depth map Zc, and using the processed difference depth map in adder 28 to generate a final depth map Zd. See Bruls [0076], Fig. 4.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Fenton’s depth-sensing system to incorporate the depth-map difference processing taught by Bruls. Fenton already generates respective time-of-flight and structured-light depth information for determining scene depth, while Bruls teaches subtracting one depth map from another to generate a difference depth map and using the processed difference depth map in generating a final depth map. See Bruls [0076]- [0077], Fig. 4. Applying Bruls’s known difference-map processing to Fenton’s respective depth maps would have amounted to applying a known depth-processing technique to a similar depth-sensing system to obtain the predictable result of identifying discrepancies between the independently determined depth values and using those discrepancies to improve the final depth map.
Regarding claim 10, Fenton, in view of Hall and Bruls, teaches the depth sensing device according to claim 9, wherein the conversion module determines that a plurality of depth difference value of the depth difference map respectively corresponds to a plurality of corresponding depth ranges, and calculates the depth map according to the first reference depth map and plurality of corresponding depth ranges.
Bruls teaches determining differences between corresponding values of first and second depth maps and generating a difference depth map that is used in producing a final depth map. See Bruls [0053]- [0069], [0076]- [0077], Fig. 4.
Hall teaches that TOF phase measurements produce a plurality of possible depth estimates corresponding to respective radial-depth regions because of phase ambiguity, and structured-light information is used to select the correct radial distance from those possible ranges. See Hall [0042]- [0043]. Hall further teaches detected ranges 508a-508d, each corresponding to a phase-wrapped TOF solution, and selecting one of those ranges based on structured-light information, including through a LUT or threshold comparison with a triangulation depth estimate. See Hall [0046]-[0047], Fig. 5. Hall further teaches that the selected radial distance constitutes a final depth estimate that is used to construct a depth map. See Hall [0071].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Fenton’s depth-sensing system to incorporate the depth-map difference processing taught by Bruls. Fenton already generates both time-of-flight depth information and structured-light depth information for determining scene depth, while Bruls teaches subtracting one depth map from another to generate a difference depth map and using the processed difference depth map in generating a final depth map. See Bruls [0076]- [0077], Fig. 4. Applying Bruls’s known depth-map difference technique to Fenton’s respective depth maps would have been the use of a known technique to improve a similar depth-processing system in the same way, with the predictable result of identifying and compensating differences between the depth estimates and improving the resulting depth map.
Claims 19-20 are method claims corresponding to device claims 9-10. They are rejected for the same reasons.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Yosub et al. (US 20220364849 A1), teaches Multi-sensor Depth Mapping
Masuda et al. (US 20080205709 A1), teaches ranging apparatus and ranging method
Barnes et al. (US 11092678 B2), teaches measuring and removing the corruption of time-of-flight depth images due to internal scattering
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEMPSON NOEL whose telephone number is (571) 272-3376. The examiner can normally be reached on Monday-Friday 9:30-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Yuqing Xiao can be reached on (571) 270-3603. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JEMPSON NOEL/Examiner, Art Unit 3645
Prosecution Insights