Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/27/2023 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claim 16 is objected to because of the following informalities:
“ The method of claim 1, further comprising: capturing the ToF image of a scene using an ToF sensor; and capturing the left image and the right image of the scene using a color image sensor;” should read “. The method of claim 1, further comprising: capturing the ToF image of a scene using an ToF sensor; and capturing the left image and the right image of the scene using a color image sensor.” The punctuation at the end of the sentence should be a period.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-16 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation ""generating, based on the first reliabilities, a depth map of a scene based on a left image and a right image and selectively based on the ToF image"" in the last two lines of the claim. This statement for generating a depth map is contradictory. The generating step based on the first reliabilities indicates that the ToF images are always being considered for the depth map. But the depth map is generated from left image and right image and also selectively based on the ToF image which is counter to “always there”. For the record, we will interpret that the depth map is based on the left and right image and from the ToF image. Clarification is required.
Dependent claims 2-16 are rejected for inheriting the same deficiency of independent claim 1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-6, 8-13, 15-19, 21-22 are rejected under 35 U.S.C. 103 as being obvious over Boisson et. al. (United States Patent Application Publication US 2015/0178936 A1) in view of Jung et. al. (United States Patent Application Publication US 2019/0080462 A1).
Regarding claim 1, Boisson et. al. discloses a processor-implemented method of estimating depth (Boisson et. al. abstract, method of performing depth estimation, an active depth map associated with an active depth estimation technique; [0003] For active methods Time-of-Flight (ToF) or structured light devices may be used), the method comprising: and generating, based on the first reliabilities, a depth map of a scene based on a left image and a right image and selectively based on the ToF image (Boisson et. al. Fig. 5, a right and left image is used as input for stereo matching; [0015]-[0020] the disparity for the global disparity map is determined based on the minimization of the cost function, which includes variables such as the matching reliability parameters; [0022]-[0023] at each of a plurality of spatial resolution levels, a set of stereo images of the scene corresponding to a passive depth estimation technique, and an active depth map corresponding to an active depth estimation technique is obtained). However, Boisson et. al. fails to disclose calculating a first reliability of each of a plurality of time of flight (ToF) pixels of a ToF image.
PNG
media_image1.png
744
892
media_image1.png
Greyscale
Jung et. al. teaches calculating a first reliability of each of a plurality of time of flight (ToF) pixels of a ToF image (Fig. 1, [0009] The calculating of the first reliabilities may include calculating the first reliabilities of the semantic segments based on whether the object included in the input image is a moving object. [0017] an apparatus for calculating a depth map includes a camera configured to acquire an input image; and a processor configured to divide the input image into segments, calculated reliabilities of the segments, select as least one of the segments based on the reliabilities, estimate pose information of the camera…). The first reliability calculations are important to the claimed invention because it quantifies the probability that a system or component will perform its intended function without failure over a specified period. Thus, it would have been obvious for one skilled in the art prior to the effective filing date of the claimed invention to have combined the teachings of Boisson et. al. and Jung et. al. so that the input image of Jung et. al. is the Time-of-Flight (ToF) images of Boisson et. al.
PNG
media_image2.png
880
684
media_image2.png
Greyscale
Regarding claim 17, which is an electronic device comprising: one or more processors configured to execute instructions; and one or more memories storing the instructions, wherein the execution of the instructions configures the one or more processors to perform the method of claim 1, which the rejection analysis is incorporated herein.
Regarding claim 2 and claim 18, Boisson et. al. and Jung et. al. disclose the method of claim 1 and the electronic device of claim 17, wherein the calculating of the first reliabilities comprises: projecting each of the plurality of ToF pixels onto the left image and the right image (Boisson et. al. Fig. 5, right and left image are inputted for stereo matching, along with calculations of depth reliability). However, Boisson et. al. fails to disclose calculating a second reliability of a respective second ToF pixel, of the plurality of ToF pixels, corresponding to each second ToF projection point on a second corresponding scan line in a first direction; and calculating, based on the calculating of the second reliability, the first reliability of a respective first ToF pixel, of the plurality of ToF pixels, corresponding to each first ToF projection point on a first corresponding scan line in a second direction that is opposite to the first direction.
Jung et. al. teaches calculating a second reliability of a respective second ToF pixel, of the plurality of ToF pixels, corresponding to each second ToF projection point on a second corresponding scan line in a first direction; and calculating, based on the calculating of the second reliability, the first reliability of a respective first ToF pixel, of the plurality of ToF pixels, corresponding to each first ToF projection point on a first corresponding scan line in a second direction that is opposite to the first direction (Jung et. al. Fig. 1, calculate reliabilities of segments, [0009]-[0011] Calculating of the reliabilities may further include fusing the first reliabilities and the second reliabilities; and determining the fused reliabilities to be the reliabilities of both the semantic segments and the depth segments).
The first and second reliability calculations are important to the claimed invention because it quantifies the probability that a system or component will perform its intended function without failure over a specified period. Thus, it would have been obvious for one skilled in the art prior to the effective filing date of the claimed invention to have combined the teachings of Boisson et. al. and Jung et. al. so that the input image of Jung et. al. is the Time-of-Flight (ToF) images of Boisson et. al.
Regarding claim 3, Boisson et. al. and Jung et. al. disclose the method of claim 2, and Boisson et. al. further discloses wherein the calculating of the second reliabilities and the calculating of the first reliabilities are based on an image feature difference of each second ToF projection point of the left image and the right image and a third reliability of a ToF pixel corresponding to a ToF projection point determined similar to an image feature in which a distance between each second ToF projection point is in a preset range on the second corresponding scan line (Boisson et. al. Fig. 5, right and left pictures are used for stereo-matching and a disparity map is created from these inputs, [0015]-[0020], [0079] Reliability weighting Rsensork presents three levels of reliability corresponding to a re-projected Kinect depth sample.).
Regarding claim 4 and claim 19, Boisson et. al. and Jung et. al. disclose the method of claim 1 and electronic device of claim 1, and Jung et. al. further discloses wherein the generating of the depth map comprises: determine a first quantity of first reliability ToF pixels, of the plurality of ToF pixels, that have respective first reliabilities that satisfy a predetermined requirement; selecting, in response to the first quantity satisfying a first threshold requirement, to generate the depth map based on the ToF image (Jung et. al. [0017], Fig. 1, Fig. 2, the reliabilities are first calculated from which a depth map is generated); and selecting, in response to the first quantity not satisfying the first threshold requirement, to generate the depth map without consideration of the ToF image (Jung et. al. [0090] the mapper calculates a depth map by calculating a depth of a captured object. The mapper calculates the depth map of the input image based on the pose information of the camera estimated from the pixels of the selected segment.).
Regarding claim 5, Boisson et. al. and Jung et. al. disclose the method of claim 4, and Boisson et. al. further discloses wherein the generating of the depth map based on the ToF image comprises: performing a first stereo matching of the left image and the right image, including a determination of first matched ToF pixels; predicting, using a first neural network, a fourth reliability of each of the plurality of ToF pixels based on the ToF image and a result of the first stereo matching; and generating a first depth map of the scene by performing a second stereo matching of the left image and the right image based on the ToF image and the fourth reliabilities (Boisson et. al. Fig .4, right and left images are input for stereo-matching, [0058] a process of stereo matching is performed; [0082] At each resolution level the global reliability map can be used for multi-lateral filtering of the depth map fused together with a colour image of the corresponding set of stereo images. In practice a right depth map is filtered with a corresponding right stereo image and a left depth map is filtered with a corresponding left stereo image).
PNG
media_image3.png
481
520
media_image3.png
Greyscale
Regarding claim 6, Boisson et. al. and Jung et. al. discloses the method of claim 5, and Boisson et. al. further discloses wherein the selecting, to generate the depth map based on the ToF image, is based on the first reliabilities and the ToF image in response to a second quantity of second ToF pixels, of the plurality of ToF pixels, that have respective first reliabilities that satisfy the first threshold requirement and a second threshold requirement; or wherein the selecting, to generate the depth map without the consideration of the ToF image, is based on a third quantity of third ToF pixels, of the plurality of ToF pixels that have respective first reliabilities that satisfy the first threshold requirement and do not satisfy the second threshold requirement (Boisson et. al. Fig .4, right and left images are input for stereo-matching, [0058] a process of stereo matching is performed; [0082] At each resolution level the global reliability map can be used for multi-lateral filtering of the depth map fused together with a colour image of the corresponding set of stereo images. In practice a right depth map is filtered with a corresponding right stereo image and a left depth map is filtered with a corresponding left stereo image).
Regarding claim 8, Boisson et. al. and Jung et. al. discloses the method of claim 5, and Boisson et. al. further discloses wherein the generating of the first depth map comprises: calculating a respective matching cost of a candidate disparity corresponding to each of the plurality of ToF pixels during the second stereo matching based on a respective value of each of the plurality of ToF pixels and the predicted fourth reliabilities of each of the plurality of ToF pixels; determining a respective disparity value corresponding to each of the plurality of ToF pixels based on the respective matching cost; and estimating the first depth map using the determined respective disparity value (Boisson et. al. [0074] The consistency term is used to constrain the disparity estimation. The active/passive consistency cost term is representative of the depth consistency between the stereo matching and the active depth measurement, and the aim is to minimize the consistency cost function).
PNG
media_image4.png
548
530
media_image4.png
Greyscale
Regarding claim 9, Boisson et. al. and Jung et. al. discloses the method of claim 5, and Boisson et. al. further discloses wherein the generating of the depth map comprises: projecting the ToF image onto the left image and the right image and generating a second depth map by performing an interpolation (Boisson et. al. [0102] for example depth map interpolation is consistent with disparity estimations), based on corresponding image features of the left image and the right image, on a ToF projection point area that satisfies a preset density; generating a third depth map by performing an interpolation on the first depth map based on image features of the left image and the right image; and generating a fourth depth map of the scene based on the second depth map and the third depth map (Boisson et. al. [0074] The consistency term is used to constrain the disparity estimation. The active/passive consistency cost term is representative of the depth consistency between the stereo matching and the active depth measurement, and the aim is to minimize the consistency cost function. Fig .4, right and left images are input for stereo-matching, [0058] a process of stereo matching is performed; [0082] At each resolution level the global reliability map can be used for multi-lateral filtering of the depth map fused together with a colour image of the corresponding set of stereo images. In practice a right depth map is filtered with a corresponding right stereo image and a left depth map is filtered with a corresponding left stereo image).
Regarding claim 10, Boisson et. al. and Jung et. al. discloses the method of claim 9, and Boisson et. al. further discloses wherein the generating of the second depth map comprises: generating interpolated ToF projection points by respectively performing an interpolation, based on a corresponding image feature of the left image and the right image, on adjacent ToF projection points spaced apart in a preset distance on a corresponding scan line of each ToF projection point; determining a regular grid of a ToF projection point by sampling the interpolated ToF projection points; and generating the second depth map by respectively performing an interpolation, based on a respective image feature of the left image and the right image, on each ToF projection point on each determined regular grid (Boisson et. al. [0074] The consistency term is used to constrain the disparity estimation. The active/passive consistency cost term is representative of the depth consistency between the stereo matching and the active depth measurement, and the aim is to minimize the consistency cost function. Fig .4, right and left images are input for stereo-matching, [0058] a process of stereo matching is performed; [0082] At each resolution level the global reliability map can be used for multi-lateral filtering of the depth map fused together with a colour image of the corresponding set of stereo images. In practice a right depth map is filtered with a corresponding right stereo image and a left depth map is filtered with a corresponding left stereo image).
Regarding claim 11, Boisson et. al. and Jung et. al. discloses the method of claim 9, and Jung et. al. further discloses wherein the performing of the interpolation of the first depth map comprises determining a depth value of a point to be interpolated based on a spatial distance and an image feature difference between a point to be interpolated and adjacent reference points (Jung et. al. [0091] The mapper generates a new key frame or refines a current key frame based on the tracked frames. For example, in a case in which an input image does not include objects captured in a previous frame since the camera used to capture the input image has moved a far distance, the calculation apparatus generates a new key frame from last tracked frames. When the new key frame is generated, a depth map of the corresponding key frame is initialized by projecting a point from the previous key frame onto the new key frame. A frame not corresponding to the new key frame, among the tracked frames, is used to redefine the current key frame).
Regarding claim 12, Boisson et. al. and Jung et. al. discloses the method of claim 4, and Boisson et. al. further discloses wherein the generating of the depth map without consideration of the ToF image comprises generating a fifth depth map of the scene through a stereo matching of the left image and the right image (Boisson et. al. Fig. 4, right and left images are input for stereo-matching, [0058] a process of stereo matching is performed).
Regarding claim 13, Boisson et. al. and Jung et. al. discloses the method of claim 12, and Boisson et. al. further discloses further comprising: updating a depth value of an unreliable depth value point of the generated depth map, wherein the generated depth map comprises the fifth depth map (Boisson et. al. [0082] At each resolution level the global reliability map can be used for multi-lateral filtering of the depth map fused together with a colour image of corresponding set of stereo images. In this way unreliable depth values can be corrected by more reliable neighbors).
Regarding claim 15, Boisson et. al. and Jung et. al. discloses the method of claim 14, and Boisson et. al. further discloses wherein the determining of the reliable depth value point and the unreliable depth value point comprises determining a regular grid of a depth value point based on the updated depth map, and determining the reliable depth value point and the unreliable depth value point on the regular grid (Boisson et. al. [0082] At each resolution level the global reliability map can be used for multi-lateral filtering of the depth map fused together with a colour image of corresponding set of stereo images. In this way unreliable depth values can be corrected by more reliable neighbors.).
Regarding claim 16, Boisson et. al. and Jung et. al. disclose the method of claim 1, and Jung et. al. further discloses further comprising: capturing the ToF image of a scene using an ToF sensor; and capturing the left image and the right image of the scene using a color image sensor (Jung et. al. [0095] The camera is, for example, a red, green, and blue (RBG) camera, or a red, green, and blue-depth (RGB-D) camera).
Regarding claim 21, Boisson et. al. and Jung et. al. discloses the electronic device of claim 17, and Boisson et. al. further discloses wherein, for the performance of the generation of the depth map without consideration of the ToF image, the one or more processors are configured to perform a third stereo matching of the left image and the right image, and generate the depth map dependent on the performed third stereo matching (Boisson et. al. Fig. 1, [0048]-[0049], left and right ToF images are captured using active type camera to estimate depth information, Fig. 4, right and left images are input for stereo-matching, [0058] a process of stereo matching is performed)
Regarding claim 22, Boisson et. al. and Jung et. al. discloses the electronic device of claim 17, and Boisson et. al. further discloses further comprising: a first sensor configured to capture the ToF image of a scene; and a second sensor configured to capture the left image and the right image of the scene (Boisson et. al. Fig. 1, [0048]-[0049], left and right ToF images are captured suing active type camera to estimate depth information).
PNG
media_image5.png
742
510
media_image5.png
Greyscale
Claim(s) 7, 14, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Boisson et. al. (United States Patent Application Publication US 2015/0178936 A1) in view of Jung et. al. (United States Patent Application Publication US 2019/0080462 A1) as applied to claim 19 above, and further in view of Liu et. al. (United States Patent Application Publication US 2016/0239725 A1).
Regarding claim 7, Boisson et. al. and Jung et. al. discloses the method of claim 5, and Boisson et. al. further discloses wherein the predicting of the fourth reliabilities comprises predicting the fourth reliabilities using at least one piece of information among first information, second information, wherein the first information is a difference between a disparity value corresponding to each of the plurality of ToF pixels and a disparity value of each of first matched ToF pixels, wherein the second information is an image feature difference of each of the plurality of ToF pixels of a corresponding projection point of the left image and the right image, and wherein the third information is a difference of depth values between the corresponding projection points and at least one ToF projection point having a determined similar feature in a corresponding projection point area (Boisson et. al. Fig .4, right and left images are input for stereo-matching, [0058] a process of stereo matching is performed; [0074] The consistency term is used to constrain the disparity estimation. The active/passive consistency cost term is representative of the depth consistency between the stereo matching and the active depth measurement, and the aim is to minimize the consistency cost function. [0082] At each resolution level the global reliability map can be used for multi-lateral filtering of the depth map fused together with a colour image of the corresponding set of stereo images. In practice a right depth map is filtered with a corresponding right stereo image and a left depth map is filtered with a corresponding left stereo image). However, Boisson et. al. fails to disclose third information as an input to the first neural network.
Liu et. al. teaches third information as an input to the first neural network (Liu et. al. [0023] A first neural network determines locations of edges. The second neural network determines the confidence values for the pixels. The edge locations are used to selected neighboring pixels in terms of the geodesic distance, and the confidence values are used to determine the reliability (weighting factor) of the neighboring pixels). The neural network is important to the claimed invention because it performs denoising of the ToF acquired images. Thus, it would have been obvious to one skilled in the art prior to the effective filing date of the claimed invention to have combined the teachings of Boisson et. al., Jung et. al., and Liu et. al. so that the neural network is incorporated in the solution of the claimed invention.
Regarding claim 14, Boisson et. al. and Jung et. al. disclose the method of claim 13, and Boisson et. al. further discloses wherein the updating of the depth value comprises: determining a reliable depth value point and the unreliable depth value point of the generated depth map; and generating an updated depth map by performing an interpolation (Boisson et. al. [0102] for example depth map interpolation is consistent with disparity estimations), based on corresponding image features of the left image and the right image, on an area around the unreliable depth value point (Boisson et. al. [0082] At each resolution level the global reliability map can be used for multi-lateral filtering of the depth map fused together with a colour image of corresponding set of stereo images. In this way unreliable depth values can be corrected by more reliable neighbors. In practice a single depth map (right or left) is obtained and fused with the corresponding right or left stereo image).
However, Boisson et. al. fails to disclose predicting, using a second neural network, the depth value of the unreliable depth value point based on a feature of the reliable depth value point and the unreliable depth value point.
Liu et. al. teaches predicting, using a second neural network, the depth value of the unreliable depth value point based on a feature of the reliable depth value point and the unreliable depth value point (Liu et. al. [0023] A first neural network determines locations of edges. The second neural network determines the confidence values for the pixels. The edge locations are used to selected neighboring pixels in terms of the geodesic distance, and the confidence values are used to determine the reliability (weighting factor) of the neighboring pixels).
The neural network is important to the claimed invention because it performs denoising of the ToF acquired images. Thus, it would have been obvious to one skilled in the art prior to the effective filing date of the claimed invention to have combined the teachings of Boisson et. al., Jung et. al., and Liu et. al. so that the neural network is incorporated in the solution of the claimed invention.
Regarding claim 20, Boisson et. al. and Jung et. al. discloses the electronic device of claim 19, and Boisson et. al. further discloses wherein, for the performance of the generation of the depth map based on the ToF image, the one or more processors are configured to: perform a first stereo matching of the left image and the right image; perform a second stereo matching of the left image and the right image based on the ToF image and the other reliabilities; and generate the depth map dependent on the performed second stereo matching (Boisson et. al. Fig. 4, right and left images are input for stereo-matching, [0058] a process of stereo matching is performed. [0082] At each resolution level the global reliability map can be used for multi-lateral filtering of the depth map fused together with a colour image of the corresponding set of stereo images. In practice a right depth map is filtered with a corresponding right stereo image and a left depth map is filtered with a corresponding left stereo image). Boisson et. al. fails to disclose predict, using a first neural network, another reliability of each of the plurality of ToF pixels based on the ToF image and a result of the first stereo matching.
Liu et. al. teaches predict, using a first neural network, another reliability of each of the plurality of ToF pixels based on the ToF image and a result of the first stereo matching (Liu et. al. [0023] A first neural network determines locations of edges. The second neural network determines the confidence values for the pixels. The edge locations are used to selected neighboring pixels in terms of the geodesic distance, and the confidence values are used to determine the reliability (weighting factor) of the neighboring pixels).
The neural network is important to the claimed invention because it performs denoising of the ToF acquired images. Thus, it would have been obvious to one skilled in the art prior to the effective filing date of the claimed invention to have combined the teachings of Boisson et. al., Jung et. al., and Liu et. al. so that the neural network is incorporated in the solution of the claimed invention.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JESSICA YIFANG LIN whose telephone number is (571)272-6435. The examiner can normally be reached M-F 7:00am-6:15pm, with optional day off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be reached at 571-272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JESSICA YIFANG LIN/Examiner, Art Unit 2668 May 20, 2026
/VU LE/Supervisory Patent Examiner, Art Unit 2668
Prosecution Insights