Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
The following is a non-final, first office action in response to the communication filed 01/03/2024. Claims 1-9 are currently pending and have been examined.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 04/03/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 10, 13, and 16 are rejected under 35 U.S.C. 102(a) as being anticipated by Chugunov et al. (US-20230118593-A1; hereinafter, Chugunov).
Regarding claim 1, Chugunov discloses An apparatus for refining depth values, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: (see at least [Page 10, Lines 1-5]; “FIG. 9 illustrates a computer-implemented method 900 for depth determination according to an example embodiment of the present invention. The computer-implemented method can be implemented using one or more memory devices (e.g., a non-transitory memory), one or more processing devices (e.g., a CPU, GPU, etc.), and/or one or more communication channels to transmit one or more instructions.”) obtain a depth representation of a scene, the depth representation comprising depth values based on a time-of-flight (ToF) signal; obtain amplitude values corresponding to the depth values of the depth representation, wherein the amplitude values are based on the ToF signal; (see at least [Page 2, Lines 13-21]; "Fig. 2 illustrates the use of multiple correlation images to calculate depth according to the related art. As illustrated in Fig. 2, by collecting multiple (e.g., four) correlation images 211A, 211B, 211C, 211D with varying phase offsets (e.g., 0, π, π/2, 3π/2, respectively), the phase of the reflected light can be determined... Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230." and see at least [Page 7, Lines 23-26]; "As there are no available datasets, the set of light field data 510 of correlation image 511 with depth map 530 are used to determine ToF amplitude measurements. In some embodiments, the time of flight measurements are decoded or otherwise extracted from the set of light field data 510 to determine initial depth estimate for depth map 530.") generate a peak map based on the amplitude values and the depth values; generate a depth-based mask based on the depth values; and generate refined depth values based on the depth values, the peak map, and the depth-based mask. (see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 8, Lines 7-9]; "For a given masked correlation image, each generated masked pixel 660 (e.g., generated using is masking process as illustrated in Fig. 6) is processed by a convolution refinement network 770 to output a decoded and refined depth reconstruction map 780." and see at least [Page 9, Lines 8-11]; "Thus, the encoder-decoder model of convolution refinement network 770 is configured to aggregate the spatial information and utilize mask structural cues to produce refined depth estimates. The errors between initial depth estimates and refined depth estimates can be used to improve mask patterns.").
Regarding claim 10, Chugunov discloses The apparatus of claim 1, wherein, to generate the depth-based mask, the at least one processor is configured to: determine a plurality of depth-consistency criteria, wherein the plurality of depth-consistency criteria comprises a respective depth-consistency criterion for each depth value based on the depth value; and generate the depth-based mask based on the depth values and the plurality of depth-consistency criteria. (see at least [Page 10, Lines 1-5]; "FIG. 9 illustrates a computer-implemented method 900 for depth determination according to an example embodiment of the present invention. The computer-implemented method can be implemented using one or more memory devices (e.g., a non-transitory memory), one or more processing devices (e.g., a CPU, GPU, etc.), and/or one or more communication channels to transmit one or more instructions." and see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 7, Lines 23-26]; "As there are no available datasets, the set of light field data 510 of correlation image 511 with depth map 530 are used to determine ToF amplitude measurements. In some embodiments, the time of flight measurements are decoded or otherwise extracted from the set of light field data 510 to determine initial depth estimate for depth map 530." and see at least [Page 9, Lines 1-4]; "As a result, convolution refinement network 770 quickly determines high level depth and mask features, as well as determines other image information where raw phase data might significantly differ from a training set." and see at least [Page 4, Lines 15-22]; "To achieve these and other advantages and in accordance with the purpose of the present invention, as embodied and broadly described, the microlens amplitude masks for flying pixel removal in time-of-flight imaging includes systems, devices, methods, and instructions for image depth determination, including receiving an image, adding noise to the image, determining a set of correlation images, each correlation image having a varying phase offset, for each pixel of the image, generating a masked pixel by applying a mask array, and for each masked pixel, determining the depth of the masked pixel to generate a depth map for the image on a per pixel basis.").
Regarding claim 13, Chugunov discloses The apparatus of claim 1, wherein, to generate the refined depth values, the at least one processor is configured to: generating a combined mask based on the peak map and the depth-based mask; and applying the combined mask to the depth values to generate the refined depth values. (see at least [Page 10, Lines 1-5]; "FIG. 9 illustrates a computer-implemented method 900 for depth determination according to an example embodiment of the present invention. The computer-implemented method can be implemented using one or more memory devices (e.g., a non-transitory memory), one or more processing devices (e.g., a CPU, GPU, etc.), and/or one or more communication channels to transmit one or more instructions." and see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 7, Lines 23-26]; "As there are no available datasets, the set of light field data 510 of correlation image 511 with depth map 530 are used to determine ToF amplitude measurements. In some embodiments, the time of flight measurements are decoded or otherwise extracted from the set of light field data 510 to determine initial depth estimate for depth map 530." and see at least [Page 9, Lines 1-4]; "As a result, convolution refinement network 770 quickly determines high level depth and mask features, as well as determines other image information where raw phase data might significantly differ from a training set." and see at least [Page 9, Lines 8-11]; "Thus, the encoder-decoder model of convolution refinement network 770 is configured to aggregate the spatial information and utilize mask structural cues to produce refined depth estimates. The errors between initial depth estimates and refined depth estimates can be used to improve mask patterns." and see at least [Page 10, Lines 16-22]; "At 940, for each pixel of the image, method 900 generates a masked pixel by applying a mask array. As discussed in connection with Fig. 6, by multiplying a set of sub-aperture pixels (such as sub-aperture pixels 641-649) with a mask array (such as mask array 650, including a set of micro-lens masks 651-659) and summing the results on a per pixel basis, a masked pixel (such as masked pixel 660) is produced. Here, sub-aperture pixels 640 are weighted according to a mask array 650. The weighted sub-aperture pixels are combined with the simulated noise to produce an initial depth estimate on a per pixel basis.").
Regarding claim 16, Chugunov discloses A method for refining depth values, the method comprising: obtaining a depth representation of a scene, the depth representation comprising depth values based on a time-of-flight (ToF) signal; (see at least [Page 2, Lines 13-21]; "Fig. 2 illustrates the use of multiple correlation images to calculate depth according to the related art. As illustrated in Fig. 2, by collecting multiple (e.g., four) correlation images 211A, 211B, 211C, 211D with varying phase offsets (e.g., 0, π, π/2, 3π/2, respectively), the phase of the reflected light can be determined... Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230." and see at least [Page 7, Lines 23-26]; "As there are no available datasets, the set of light field data 510 of correlation image 511 with depth map 530 are used to determine ToF amplitude measurements. In some embodiments, the time of flight measurements are decoded or otherwise extracted from the set of light field data 510 to determine initial depth estimate for depth map 530.") obtaining amplitude values corresponding to the depth values of the depth representation, wherein the amplitude values are based on the ToF signal; (see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 7, Lines 23-24]; "As there are no available datasets, the set of light field data 510 of correlation image 511 with depth map 530 are used to determine ToF amplitude measurements.") generating a peak map based on the amplitude values and the depth values; generating a depth-based mask based on the depth values; and generating refined depth values based on the depth values, the peak map, and the depth-based mask. (see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 8, Lines 7-9]; "For a given masked correlation image, each generated masked pixel 660 (e.g., generated using is masking process as illustrated in Fig. 6) is processed by a convolution refinement network 770 to output a decoded and refined depth reconstruction map 780." and see at least [Page 9, Lines 8-11]; "Thus, the encoder-decoder model of convolution refinement network 770 is configured to aggregate the spatial information and utilize mask structural cues to produce refined depth estimates. The errors between initial depth estimates and refined depth estimates can be used to improve mask patterns.").
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Chugunov and in view of Friedman et al. (US-20220292705-A1; hereinafter Friedman).
Regarding claim 15, Chugunov discloses [Note: what Chugunov fails to disclose is strike-through] The apparatus of claim 1, wherein the at least one processor is further configured to "analyze data," (see at least [Page 10, Lines 1-5]; "FIG. 9 illustrates a computer-implemented method 900 for depth determination according to an example embodiment of the present invention. The computer-implemented method can be implemented using one or more memory devices (e.g., a non-transitory memory), one or more processing devices (e.g., a CPU, GPU, etc.), and/or one or more communication channels to transmit one or more instructions." and see at least [Page 6, Lines 22-25]; "In addition, use of microlens mask 411, with its learned mask pattern (as described below), further enables measurements from neighboring pixels with different effective apertures to provide additional data to accurately identify and rectify flying pixels.").
However, Chugunov does not explicitly teach patterns of spots nor those patterns with less spots than depth representation. Instead, Chugunov teaches a processor and analyzed data.
Chugunov discloses a method to accurately identify data and Friedman is directed at projecting spots and patterns of spots. Friedman teaches:
Reflected pattern of spots (see at least Figure 3A and see at least [00128]; "The depth map module 214 retrieves the captured 3D image of the illuminated objects from the image 3D data store 242 and generates a depth map representation of the objects from the captured image (e.g. pattern image) of the illuminated object. As described above, a depth map representation of an object refers to an image containing information about distances of different parts of the surface of the object and/or the scene from a designated viewpoint. The designated viewpoint can be the position of a sensor that captures the image of the object. In an embodiment, the depth maps representations are stored at the Depth Maps representation data store 236 as more fully described below. An example depth map representation is further described below with reference to FIG. 3A." and see at least [0131]; "FIG. 3A shows an example of captured image 335 including reflected light pattern spots, in accordance with embodiments. For illustration matters, each spot of the reflected light pattern spots is colored using gray scale color where each color represents the distance of the spot from a reference point (e.g. camera). For example, the scale 282 includes a grayscale color for a distance of around 40 cm from the camera and continuously the color representation changes to black scale for a distance around 140 cm from the camera and so on the color scale varies according to the distance. Accordingly, the multiple patterns spot 281 of the captured image 285 shown in FIG. 3B are analyzed to yield a depth representation image 287 of the scene as illustrated in FIG. 3A. For example, the cluster of reflected dots on the driver's legs (presented by ellipse 345) are typically around 20-50 cm from the camera while the center mass of the driver (presented be ellipse 355) is more remote from the camera (around 50-80 cm).").
Both Chugunov and Friedman can analyze data. It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method used in Chugunov to include hardware to their design as taught by Friedman. One of ordinary skill would be motivated to include a camera to the sensors of Chugunov along with the programming needed to project patterns of spots to the taken image area. Therefore, the claimed invention is reproduced by combining elements from Chugunov and Friedman.
Claims 3 - 5, 9, 11, 12, and 18 - 20 are rejected under 35 U.S.C. 103 as being unpatentable over Chugunov and in view of Zhang, Feng (US-20230204736-A1; hereinafter Zhang).
Regarding claim 3, Chugunov discloses [Note: what Chugunov fails to disclose is strike-through] The apparatus of claim 1, wherein, to generating the peak map, the at least one processor is configured to: determine a plurality of peaks (see at least [Page 10, Lines 1-5]; "FIG. 9 illustrates a computer-implemented method 900 for depth determination according to an example embodiment of the present invention. The computer-implemented method can be implemented using one or more memory devices (e.g., a non-transitory memory), one or more processing devices (e.g., a CPU, GPU, etc.), and/or one or more communication channels to transmit one or more instructions." and see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 7, Lines 23-26]; "As there are no available datasets, the set of light field data 510 of correlation image 511 with depth map 530 are used to determine ToF amplitude measurements. In some embodiments, the time of flight measurements are decoded or otherwise extracted from the set of light field data 510 to determine initial depth estimate for depth map 530." and see at least [Page 9, Lines 1-4]; "As a result, convolution refinement network 770 quickly determines high level depth and mask features, as well as determines other image information where raw phase data might significantly differ from a training set.").
However, Chugunov does not explicitly teach a window/(time segments) or a plurality of windows. Instead, Chugunov teaches generating a peak map and a processor.
Chugunov discloses a method to analyze amplitude data and Zhang is directed at analyzing amplitudes that are measured in time segments dependent on specific thresholds. Zhang teaches:
Window/(time segment) and plurality of windows (see at least [0094]; "As shown in FIG. 8, the echo signal with the echo superposition is characterized in that amplitude of the echo signal exceeds a first threshold but does not exceed a fourth threshold, or the amplitude exceeds the fourth threshold but does not exceed the second threshold. That is, the receiving unit 101 detects that receiving time of a first digital signal segment with a value of 1 is overlapped with receiving time of a fourth digital signal segment with a value of 0, and the overlapped duration exceeds a third duration threshold; or that receiving time of a fourth digital signal segment with a value of 1 is overlapped with receiving time of a second digital signal segment with a value of 0, and the overlapped duration exceeds a fourth duration threshold. For example, based on the foregoing determining criteria, an echo signal corresponding to the first digital signal segment with a value of 1 shown in FIG. 8 is the echo signal with echo superimposition.").
Both Chugunov and Zhang can analyze data. It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method used in Chugunov to include the programming for windows/(time segments) as taught by Zhang. One of ordinary skill would be motivated to include the time segment programming of Zhang to the programming of Chugunov as both analyze the amplitude data and this would add more information to the datasets generated by Chugunov. Therefore, the claimed invention is reproduced by adding the programming from Zhang to Chugunov to create a more robust programming structure capable of analyzing amplitudes more with windows/(time segments).
Regarding claim 4, Chugunov discloses [Note: what Chugunov fails to disclose is strike-through] The apparatus of claim 1, wherein, to generate the peak map, the at least one processor is configured to: determine whether respective amplitude values (see at least [Page 10, Lines 1-5]; "FIG. 9 illustrates a computer-implemented method 900 for depth determination according to an example embodiment of the present invention. The computer-implemented method can be implemented using one or more memory devices (e.g., a non-transitory memory), one or more processing devices (e.g., a CPU, GPU, etc.), and/or one or more communication channels to transmit one or more instructions." and see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 7, Lines 23-26]; "As there are no available datasets, the set of light field data 510 of correlation image 511 with depth map 530 are used to determine ToF amplitude measurements. In some embodiments, the time of flight measurements are decoded or otherwise extracted from the set of light field data 510 to determine initial depth estimate for depth map 530." and see at least [Page 9, Lines 1-4]; "As a result, convolution refinement network 770 quickly determines high level depth and mask features, as well as determines other image information where raw phase data might significantly differ from a training set.").
The remainder of claim 4 contains analogous limitations to claim 3 and is rejected for similar reasons.
Regarding claim 5, Chugunov discloses [Note: what Chugunov fails to disclose is strike-through] The apparatus of claim 1, wherein, to generate the peak map, the at least one processor is configured to: determine whether respective amplitude values the "data." (see at least [Page 10, Lines 1-5]; "FIG. 9 illustrates a computer-implemented method 900 for depth determination according to an example embodiment of the present invention. The computer-implemented method can be implemented using one or more memory devices (e.g., a non-transitory memory), one or more processing devices (e.g., a CPU, GPU, etc.), and/or one or more communication channels to transmit one or more instructions." and see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 7, Lines 23-26]; "As there are no available datasets, the set of light field data 510 of correlation image 511 with depth map 530 are used to determine ToF amplitude measurements. In some embodiments, the time of flight measurements are decoded or otherwise extracted from the set of light field data 510 to determine initial depth estimate for depth map 530." and see at least [Page 9, Lines 1-4]; "As a result, convolution refinement network 770 quickly determines high level depth and mask features, as well as determines other image information where raw phase data might significantly differ from a training set.").
The remainder of claim 5 contains analogous limitations to claim 3 and is rejected for similar reasons.
Regarding claim 9, Chugunov discloses [Note: what Chugunov fails to disclose is strike-through] The apparatus of claim 1, wherein, to generate the peak map, the at least one processor is configured to cause each (see at least [Page 10, Lines 1-5]; "FIG. 9 illustrates a computer-implemented method 900 for depth determination according to an example embodiment of the present invention. The computer-implemented method can be implemented using one or more memory devices (e.g., a non-transitory memory), one or more processing devices (e.g., a CPU, GPU, etc.), and/or one or more communication channels to transmit one or more instructions." and see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 7, Lines 23-26]; "As there are no available datasets, the set of light field data 510 of correlation image 511 with depth map 530 are used to determine ToF amplitude measurements. In some embodiments, the time of flight measurements are decoded or otherwise extracted from the set of light field data 510 to determine initial depth estimate for depth map 530." and see at least [Page 9, Lines 1-4]; "As a result, convolution refinement network 770 quickly determines high level depth and mask features, as well as determines other image information where raw phase data might significantly differ from a training set.").
The remainder of claim 9 contains analogous limitations to claim 3 and is rejected for similar reasons.
Regarding claim 11, Chugunov discloses [Note: what Chugunov fails to disclose is strike-through] The apparatus of claim 1, wherein, to generate the depth-based mask, the at least one processor is configured to: identify a candidate depth value based on a relationship between the candidate depth value and depth values (see at least [Page 10, Lines 1-5]; "FIG. 9 illustrates a computer-implemented method 900 for depth determination according to an example embodiment of the present invention. The computer-implemented method can be implemented using one or more memory devices (e.g., a non-transitory memory), one or more processing devices (e.g., a CPU, GPU, etc.), and/or one or more communication channels to transmit one or more instructions." and see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 7, Lines 23-26]; "As there are no available datasets, the set of light field data 510 of correlation image 511 with depth map 530 are used to determine ToF amplitude measurements. In some embodiments, the time of flight measurements are decoded or otherwise extracted from the set of light field data 510 to determine initial depth estimate for depth map 530." and see at least [Page 9, Lines 1-4]; "As a result, convolution refinement network 770 quickly determines high level depth and mask features, as well as determines other image information where raw phase data might significantly differ from a training set.").
The remainder of claim 11 contains analogous limitations to claim 3 and is rejected for similar reasons.
Regarding claim 12, Chugunov discloses [Note: what Chugunov fails to disclose is strike-through] The apparatus of claim 1, wherein, to generate the depth-based mask, the at least one processor is configured to: identify a candidate depth value based on a count of depth values (see at least [Page 10, Lines 1-5]; "FIG. 9 illustrates a computer-implemented method 900 for depth determination according to an example embodiment of the present invention. The computer-implemented method can be implemented using one or more memory devices (e.g., a non-transitory memory), one or more processing devices (e.g., a CPU, GPU, etc.), and/or one or more communication channels to transmit one or more instructions." and see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 7, Lines 23-26]; "As there are no available datasets, the set of light field data 510 of correlation image 511 with depth map 530 are used to determine ToF amplitude measurements. In some embodiments, the time of flight measurements are decoded or otherwise extracted from the set of light field data 510 to determine initial depth estimate for depth map 530." and see at least [Page 9, Lines 1-4]; "As a result, convolution refinement network 770 quickly determines high level depth and mask features, as well as determines other image information where raw phase data might significantly differ from a training set.").
The remainder of claim 12 contains analogous limitations to claim 3 and is rejected for similar reasons.
Regarding claim 18, Chugunov discloses [Note: what Chugunov fails to disclose is strike-through] The method of claim 16, wherein generating the peak map comprises: determining a plurality of peaks (see at least [Page 10, Lines 1-5]; "FIG. 9 illustrates a computer-implemented method 900 for depth determination according to an example embodiment of the present invention. The computer-implemented method can be implemented using one or more memory devices (e.g., a non-transitory memory), one or more processing devices (e.g., a CPU, GPU, etc.), and/or one or more communication channels to transmit one or more instructions." and see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 7, Lines 23-26]; "As there are no available datasets, the set of light field data 510 of correlation image 511 with depth map 530 are used to determine ToF amplitude measurements. In some embodiments, the time of flight measurements are decoded or otherwise extracted from the set of light field data 510 to determine initial depth estimate for depth map 530." and see at least [Page 9, Lines 1-4]; "As a result, convolution refinement network 770 quickly determines high level depth and mask features, as well as determines other image information where raw phase data might significantly differ from a training set.").
The remainder of claim 18 contains analogous limitations to claim 3 and is rejected for similar reasons.
Regarding claim 19, Chugunov discloses [Note: what Chugunov fails to disclose is strike-through] The method of claim 16, wherein generating the peak map comprises: determining whether respective amplitude values (see at least [Page 10, Lines 1-5]; "FIG. 9 illustrates a computer-implemented method 900 for depth determination according to an example embodiment of the present invention. The computer-implemented method can be implemented using one or more memory devices (e.g., a non-transitory memory), one or more processing devices (e.g., a CPU, GPU, etc.), and/or one or more communication channels to transmit one or more instructions." and see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 7, Lines 23-26]; "As there are no available datasets, the set of light field data 510 of correlation image 511 with depth map 530 are used to determine ToF amplitude measurements. In some embodiments, the time of flight measurements are decoded or otherwise extracted from the set of light field data 510 to determine initial depth estimate for depth map 530." and see at least [Page 9, Lines 1-4]; "As a result, convolution refinement network 770 quickly determines high level depth and mask features, as well as determines other image information where raw phase data might significantly differ from a training set.").
The remainder of claim 19 contains analogous limitations to claim 3 and is rejected for similar reasons.
Regarding claim 20, Chugunov discloses [Note: what Chugunov fails to disclose is strike-through] The method of claim 16, wherein generating the peak map comprises: determining whether respective amplitude values amplitude value (see at least [Page 10, Lines 1-5]; "FIG. 9 illustrates a computer-implemented method 900 for depth determination according to an example embodiment of the present invention. The computer-implemented method can be implemented using one or more memory devices (e.g., a non-transitory memory), one or more processing devices (e.g., a CPU, GPU, etc.), and/or one or more communication channels to transmit one or more instructions." and see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 7, Lines 23-26]; "As there are no available datasets, the set of light field data 510 of correlation image 511 with depth map 530 are used to determine ToF amplitude measurements. In some embodiments, the time of flight measurements are decoded or otherwise extracted from the set of light field data 510 to determine initial depth estimate for depth map 530." and see at least [Page 9, Lines 1-4]; "As a result, convolution refinement network 770 quickly determines high level depth and mask features, as well as determines other image information where raw phase data might significantly differ from a training set.").
The remainder of claim 20 contains analogous limitations to claim 3 and is rejected for similar reasons.
Claims 2, 7, 8, 14, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Chugunov and in view of Krishnan et al. (US-12596197-B1; hereinafter Krishnan).
Regarding claim 2, Chugunov discloses [Note: what Chugunov fails to disclose is strike-through] The apparatus of claim 1, wherein, to generate the peak map, the at least one processor is configured to: apply (see at least [Page 10, Lines 1-5]; "FIG. 9 illustrates a computer-implemented method 900 for depth determination according to an example embodiment of the present invention. The computer-implemented method can be implemented using one or more memory devices (e.g., a non-transitory memory), one or more processing devices (e.g., a CPU, GPU, etc.), and/or one or more communication channels to transmit one or more instructions." and see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 7, Lines 23-26]; "As there are no available datasets, the set of light field data 510 of correlation image 511 with depth map 530 are used to determine ToF amplitude measurements. In some embodiments, the time of flight measurements are decoded or otherwise extracted from the set of light field data 510 to determine initial depth estimate for depth map 530." and see at least [Page 9, Lines 1-4]; "As a result, convolution refinement network 770 quickly determines high level depth and mask features, as well as determines other image information where raw phase data might significantly differ from a training set.").
However, Chugunov does not explicitly teach applying filters or generating filtered amplitude values. Instead, Chugunov teaches generating a peak map.
Chugunov discloses a method to analyze amplitude data and Krishnan is directed at utilizing an amplitude filter. Krishnan teaches:
Amplitude filter (see at least [0017]; "The frame of raw depth image data that has been determined to potentially have veiling glare present is then filtered at a per-pixel level. Pixels are filtered using a pixel amplitude ratio threshold that is retrieved from previously determined pixel amplitude ratio threshold data. The pixel amplitude ratio threshold data provides information about amplitude thresholds associated with different observed distances and given a specified distance error value that indicates a maximum amount of error in the distances that are determined that is to be tolerated. This results in a dynamic filter in which different pixels may be assessed with different threshold values.").
Both Chugunov and Krishnan can analyze amplitude data. It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method used in Chugunov to include filter programming as taught by Krishnan. One of ordinary skill would be motivated to include the programming from Krishnan regarding filtering amplitudes using pixels into the teachings of Chugunov.
Therefore, the claimed invention is reproduced by using the filter programming from Krishnan in the programming of Chugunov together.
Regarding claim 7, Chugunov discloses [Note: what Chugunov fails to disclose is strike-through] The apparatus of claim 1, wherein, to generate the peak map, the at least one processor is configured to: determine a plurality of amplitude-based depth "data" (see at least [Page 10, Lines 1-5]; "FIG. 9 illustrates a computer-implemented method 900 for depth determination according to an example embodiment of the present invention. The computer-implemented method can be implemented using one or more memory devices (e.g., a non-transitory memory), one or more processing devices (e.g., a CPU, GPU, etc.), and/or one or more communication channels to transmit one or more instructions." and see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 7, Lines 23-26]; "As there are no available datasets, the set of light field data 510 of correlation image 511 with depth map 530 are used to determine ToF amplitude measurements. In some embodiments, the time of flight measurements are decoded or otherwise extracted from the set of light field data 510 to determine initial depth estimate for depth map 530." and see at least [Page 9, Lines 1-4]; "As a result, convolution refinement network 770 quickly determines high level depth and mask features, as well as determines other image information where raw phase data might significantly differ from a training set.").
However, Chugunov does not explicitly teach applying filters or generating filtered amplitude values. Instead, Chugunov teaches generating a peak map.
Chugunov discloses a method to analyze amplitude data and Krishnan is directed at utilizing an amplitude threshold. Krishnan teaches:
Amplitude threshold (see at least [0017]; "The frame of raw depth image data that has been determined to potentially have veiling glare present is then filtered at a per-pixel level. Pixels are filtered using a pixel amplitude ratio threshold that is retrieved from previously determined pixel amplitude ratio threshold data. The pixel amplitude ratio threshold data provides information about amplitude thresholds associated with different observed distances and given a specified distance error value that indicates a maximum amount of error in the distances that are determined that is to be tolerated. This results in a dynamic filter in which different pixels may be assessed with different threshold values.").
Both Chugunov and Krishnan can analyze amplitude data. It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method used in Chugunov to include threshold programming as taught by Krishnan. One of ordinary skill would be motivated to include the programming from Krishnan regarding filtering amplitudes via a threshold using pixels into the teachings of Chugunov.
Therefore, the claimed invention is reproduced by using the threshold programming from Krishnan in the programming of Chugunov together.
Regarding claim 8, Chugunov discloses [Note: what Chugunov fails to disclose is strike-through] The apparatus of claim 1, wherein, to generate the peak map, the at least one processor is configured to: generate an amplitude-based map based on the amplitude values; generate depth-(see at least [Page 10, Lines 1-5]; "FIG. 9 illustrates a computer-implemented method 900 for depth determination according to an example embodiment of the present invention. The computer-implemented method can be implemented using one or more memory devices (e.g., a non-transitory memory), one or more processing devices (e.g., a CPU, GPU, etc.), and/or one or more communication channels to transmit one or more instructions." and see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 7, Lines 23-26]; "As there are no available datasets, the set of light field data 510 of correlation image 511 with depth map 530 are used to determine ToF amplitude measurements. In some embodiments, the time of flight measurements are decoded or otherwise extracted from the set of light field data 510 to determine initial depth estimate for depth map 530." and see at least [Page 9, Lines 1-4]; "As a result, convolution refinement network 770 quickly determines high level depth and mask features, as well as determines other image information where raw phase data might significantly differ from a training set.").
The remainder of claim 8 contains analogous limitations to claim 7 and is rejected for similar reasons.
Regarding claim 14, Chugunov discloses [Note: what Chugunov fails to disclose is strike-through] (see at least [Page 6, Lines 16-21]; "Microlens mask 411, selected from a plurality microlens mask patterns 410, is disposed between sensor (e.g., CMOS sensor pixel 420) and microlens 430. The aperture of microlens mask 411 is configured to selectively block incident light paths to enable a custom aperture for each pixel. This modulates the selection of foreground and background light mixtures on a per-pixel basis and further encodes scene geometric information directly into the ToF measurements.").
However, Chugunov does not explicitly teach depth representation based on tof projection. Instead, Chugunov teaches a tof sensor.
Chugunov discloses a method to use tof sensors and Krishnan is directed at depth representation through a tof sensor. Krishnan teaches:
Tof projection and depth representation (see at least [0015]; "The iTOF sensor provides raw depth image data comprising a plurality of pixels, with each pixel having an observed amplitude and an observed distance." and see at least Figure 1 and see at least [00155]; "In some implementations, the AMD 104 may be equipped with a projector 988. The projector 988 may be able to project an image on a surface, such as the floor, wall, ceiling, and so forth.")
Both Chugunov and Krishnan can utilize tof sensors. It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method used in Chugunov to include hardware as taught by Krishnan. One of ordinary skill would be motivated to include an projector as taught by Krishnan with the sensors in Chugunov such as the CMOS sensor pixel and then the combination will be able to project a sparse tof measurements to generate a depth representation.
Therefore, the claimed invention is reproduced by including hardware taught by Krishnan into the sensor suite of Chugunov.
Regarding claim 17, Chugunov discloses [Note: what Chugunov fails to disclose is strike-through] The method of claim 16, wherein generating the peak map comprises: applying a (see at least [Page 10, Lines 1-5]; "FIG. 9 illustrates a computer-implemented method 900 for depth determination according to an example embodiment of the present invention. The computer-implemented method can be implemented using one or more memory devices (e.g., a non-transitory memory), one or more processing devices (e.g., a CPU, GPU, etc.), and/or one or more communication channels to transmit one or more instructions." and see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 7, Lines 23-26]; "As there are no available datasets, the set of light field data 510 of correlation image 511 with depth map 530 are used to determine ToF amplitude measurements. In some embodiments, the time of flight measurements are decoded or otherwise extracted from the set of light field data 510 to determine initial depth estimate for depth map 530." and see at least [Page 9, Lines 1-4]; "As a result, convolution refinement network 770 quickly determines high level depth and mask features, as well as determines other image information where raw phase data might significantly differ from a training set.").
The remainder of claim 17 contains analogous limitations to claim 2 and is rejected for similar reasons.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Chugunov and in view of Herman et al. (US-20200398797-A1; hereinafter Herman).
Regarding claim 6, Chugunov discloses [Note: what Chugunov fails to disclose is strike-through] The apparatus of claim 1, wherein, to generate the peak map, the at least one processor is configured to: generate a depth-mask. (see at least [Page 2, Lines 19-24]; "For each pixel, the phase ϕ is calculated. Subsequently, a phase map 220 of the correlation images can be converted into a depth map 230. For each pixel, depth z is calculated according to: (z = ϕc/4πω, Eq. (2)) where c is the speed of light and ω is a modulation frequency of the amplitude modulated light that is used for illumination (depicted as illumination 111 in Fig. 1)." and see at least [Page 8, Lines 7-9]; "For a given masked correlation image, each generated masked pixel 660 (e.g., generated using is masking process as illustrated in Fig. 6) is processed by a convolution refinement network 770 to output a decoded and refined depth reconstruction map 780." and see at least [Page 9, Lines 8-11]; "Thus, the encoder-decoder model of convolution refinement network 770 is configured to aggregate the spatial information and utilize mask structural cues to produce refined depth estimates. The errors between initial depth estimates and refined depth estimates can be used to improve mask patterns.").
However, Chugunov does not explicitly teach depth thresholds or a depth-threshold-based mask. Instead, Chugunov teaches generating a peak map and a processor.
Chugunov discloses a method to analyze depth data and Herman is directed at utilizing a depth threshold. Herman teaches:
Depth-threshold-based mask (see at least [0016]; "The instructions can further include instructions to mask a plurality of pixels of the virtual map, each masked pixel being a pixel of the first sensor data having a respective light reflectivity or depth exceeding a light reflectivity threshold or a depth threshold and the corresponding pixel of the second sensor data having a respective light reflectivity or depth below a second light reflectivity threshold or a second depth threshold.").
Both Chugunov and Herman can analyze depth data. It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method used in Chugunov to include threshold programming as taught by Krishnan. One of ordinary skill would be motivated to include the programming from Krishnan regarding depth thresholds using pixels into the teachings of Chugunov to produce depth-threshold-based masks.
Therefore, the claimed invention is reproduced by using the threshold programming from Herman in the programming of Chugunov together.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Mark A Flores whose telephone number is (571)272-9693. The examiner can normally be reached Mon-Thurs 8am - 6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vladimir Magloire can be reached at (571) 270-5144. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MARK ANTHONY FLORES/Examiner, Art Unit 3648
/VLADIMIR MAGLOIRE/Supervisory Patent Examiner, Art Unit 3648
Prosecution Insights