Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 5/19/26 has been entered.
Notice to Applicants
Claims 1-4, 7-9, 11-14, 17-19, 21-24 and 27-29 are pending. Claims 5-6, 10, 15, 16, 20, 25 and 26 have been cancelled.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-4, 7-9, 11-14, 17-19, 21-24 and 27-29 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1-4, 8, 9, 11-14, 18, 19, 21-24, 27 and 28 are rejected under 35 U.S.C. 103 as being unpatentable over Kar et al. (U.S. Publication No. 2020/335638) (hereafter, "Kar") in view of KIM et al. (U.S. Publication No. 2022/0036573) (hereafter, "KIM").
Regarding claim 1, Kar teaches an apparatus for scaling a depth prediction ([0026] The depth estimation system may provide a solution to scale/shift ambiguity (or generally referred to as affine ambiguity) in a monocular depth neural network; [0027] the depth estimation system may reduce placement latency by predicting the scale of placed object/planar surface depth using the neural network), the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to ([0035] The depth estimation system 100 includes one or more processors 140 ... The depth estimation system 100 can also include one or more memory devices 142; FIG. 1A): determine, using a trained machine learning system, a predicted depth map for an image ([0025] a depth estimation system that includes a sensor system that obtains image data, and a neural network configured to generate a depth map based on an image frame of the image data (e.g., using a single image frame to generate a depth map); [0032] the neural network is considered a monocular depth neural network because the neural network predicts a depth map based on a single image frame), the predicted depth map including a plurality of predicted depth values ([0032] the neural network predicts a depth map based on a single image frame; [0040] a dense depth map (e.g., depth map 120) provides depth values (e.g., non-metric depth values) for a large number of pixels in the image or all of the pixels in the image)), each predicted depth value corresponding to a pixel of the image ([0032] the neural network includes a U-net architecture configured to predict pixel-wise depth from a red-green-blue (RGB) image; [0036] A depth map 120 may refer to an image where each pixel represents the depth value according to a non-metric scale (e.g., 0 to 1) for the corresponding pixel in the image; [0037] the neural network 118 predicts a depth map 120 based on a single image frame 104a. The neural network 118 is configured to predict pixel-wise depth from the image frame 104a); obtain a plurality of depth values for the image ([0040] The depth estimation system 100 includes a depth estimate generator 106 that obtains depth estimates 108 (e.g., metric depth estimates) associated with the image data 104. The depth estimates 108 may include depth values in a metric scale for some of the pixels in the image data 104) from a tracker ([0040] a depth estimate generator 106; [0042] Referring to FIG. 1B, the depth estimate generator 106 may include a visual inertial motion tracker 160; [0041] the depth estimate generator 106 also obtains pose data 110) configured to determine the plurality of depth values based on one or more feature points between frames ([0043] The visual inertial motion tracker 160 is configured to generate visual feature points 162 that represent the image data 104. The visual feature points 162 are associated with depth estimates 108; [0046] a single camera may be moved around a scene 125 to capture multiple images), the plurality of depth values including depth values for less than all pixels of the image; and ([0040] The depth estimates 108 obtained by the depth estimate generator 106 may be considered sparse depth estimates (e.g., depth estimates for some of the pixels in the image data but not all of them); [0025] The metric depth estimates obtained by the depth estimate generator may be considered sparse depth estimates (e.g., depth estimates for some of the pixels in the image data but not all of them)) scale the predicted depth map for the image ([0033] The depth estimation system 100 generates a depth map 138 based on depth estimates 108 (obtained from one or more sources) and a depth map 120 … The depth estimation system 100 is configured to convert the depth map 120 having the first scale to the depth map 138 having the second scale; [0047] a depth map transformer 126 configured to transform the depth map 120 generated by the neural network 118 to a depth map 138 using the depth estimates 108; [0048] The depth map transformer 126 is configured to estimate affine parameters 132 based on the depth map 120 generated by the neural network 118 and the depth estimates 108; [0031] the parameter estimator solver is configured to solve a least squares parameter estimation problem within a RANSAC loop to estimate the affine parameters for the depth map to transform it to the second scale (e.g., the metric scale)) using a first … value computed based on the plurality of depth values and a second … value computed based on the plurality of predicted depth values ([0048] The affine parameters 132 include scale 134 and shift 136 of the depth map 120. The scale 134 includes a scale value that indicates the amount of resizing of the depth map 120 ... The depth map transformer 126 is configured to transform the depth map 120 to the depth map 138 using the affine parameters 132 ... the scale 134 and the shift 136 include two numbers (e.g., s=scale, t=shift) which when multiplied and added to the value in each pixel at depth map 120 produce depth map 138 (e.g., D138 (x, y)=s*D120 (x, y)+t), where D120(x, y) is the value in depth map 120 at the pixel location (x, y)); [0051] the parameter estimation solver 178 determines scale 134 and shift 136 based on depth offsets between the depth estimates 108 and the depth map 120 …
PNG
media_image1.png
50
170
media_image1.png
Greyscale
; [0052] The parameter k indicates the scale 134 … li and lj may represent the metric depth values for two points (e.g., two pixels) in the depth estimates 108, and di and dj may represent the non-metric depth values for the corresponding two points (e.g., two pixels) in the depth map 120; The equation (1) clearly shows scale k is calculated using a value computed based on the two depth values (li, lj) and a value computed based on the two predicted depth values (di, dj)).
Kar does not expressly teach … representative … and … representative.
However, KIM teaches correct (scale) the predicted depth map for the image ([0035] estimate depth information of each of the plural pixels in the input image based on the first statistical value and the second statistical value; [0093] the depth information estimated corresponding to the predetermined pixel may be excluded or corrected, for estimation of a depth of an image; [0065] the depth of the image may be represented as a depth map including values of depths for each of the pixels in the image. The depth map may be a single channel of an image) using a first representative value computed based on the plurality of depth values ([0036] Each first statistical value may include a mean of depth values of a corresponding each of the plural pixels in the input image, based on probabilities of the depth values) and a second representative value computed based on the plurality of predicted depth values ([0036] each second statistical value may include a variance or a standard deviation of the depth values of the corresponding each of the plural pixels in the input image, based on the probabilities of the depth values).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the method and device of Kar to incorporate the step/system of using a mean value computed based on depth values and variance based on predicted depth values to output an updated depth map taught by KIM.
The suggestion/motivation for doing so would have been to improve the accuracy of depth estimation by using multiple statistical values ([0106] it is possible to obtain an effect of enhancing the accuracy of depth estimation as well as a training efficiency by using two or more statistical values of the neural network in depth estimation of an image using an estimation model and during the training process of the neural network in the estimation model). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results. Therefore, it would have been obvious to combine Kar and KIM to obtain the invention as specified in claim 1.
Regarding claim 2, the combination of Kar and KIM teaches all the limitations of claim 1 above. Kar teaches wherein the tracker is a six-degree-of-freedom (6DOF) tracker ([0041] the depth estimate generator 106 also obtains pose data 110 ... The pose data 110 may identify a pose (e.g., position and orientation) of a device that executes the depth estimation system 100 (e.g., a smartphone that has the depth estimation system 100). In some examples, the pose data 110 includes a five degree-of-freedom (DoF) position of the device. In some examples, the pose data 110 includes a six DoF position of the device; [0042] Referring to FIG. 1B, the depth estimate generator 106 may include a visual inertial motion tracker 160; [0045] the visual inertial motion tracker 160 is configured to execute a SLAM algorithm).
Regarding claim 3, the combination of Kar and KIM teaches all the limitations of claim 2 above. Kar teaches wherein the 6DOF tracker is configured to use a 6DOF tracking algorithm ([0045] the visual inertial motion tracker 160 is configured to execute a SLAM algorithm which is a tracking algorithm that can estimate the movement of a device (e.g., the smartphone) in space by using the camera 107) to generate the depth values based on matching identified salient feature values across multiple frames and solving for camera motion ([0045] the SLAM algorithm iteratively calculate the position and the orientation (e.g., pose data 110) of the device by analyzing the key points (e.g., visual feature points 162) and descriptors of each image and tracking these descriptors from frame to frame, which can allow for a 3D reconstruction of the environment; [0043] The visual inertial motion tracker 160 is configured to generate visual feature points 162 that represent the image data 104. The visual feature points 162 are associated with depth estimates 108; [0044] The visual feature points 162 are a plurality of points (e.g., interesting points) in 3D space that represent the user's environment ... the user may move her mobile phone's camera around a scene 125 during an AR session 174, where the visual inertial motion tracker 160 may generate visual feature points 162 that represent the scene 125. In some examples, the visual feature points 162 include simultaneous localization and mapping (SLAM) points).
Regarding claim 4, the combination of Kar and KIM teaches all the limitations of claim 1 above. Kar teaches wherein the frames comprise one or more pairs of stereo images ([0046] two slightly different views of the scene are obtained, and these different views are used by the dual-pixel depth estimator 166 to generate the depth estimates 108 … a single camera may be moved around a scene 125 to capture multiple images; [0036] the neural network 118 generates a depth map 120 using two or more image frames 104a).
Regarding claim 8, the combination of Kar and KIM teaches all the limitations of claim 1 above. Kar teaches wherein, to scale the predicted depth map, the at least one processor is configured to ([0033] The depth estimation system 100 generates a depth map 138 based on depth estimates 108 (obtained from one or more sources) and a depth map 120 generated by a neural network 118 … The depth estimation system 100 is configured to convert the depth map 120 having the first scale to the depth map 138 having the second scale): determine a final depth map based on multiplying the predicted depth map with a scale factor ([0048] The scale 134 includes a scale value that indicates the amount of resizing of the depth map 120 ... the scale 134 and the shift 136 include two numbers (e.g., s=scale, t=shift) which when multiplied and added to the value in each pixel at depth map 120 produce depth map 138 (e.g., D138 (x, y) = s*D120 (x, y) + t), where D120(x, y) is the value in depth map 120 at the pixel location (x, y)); [0051] the parameter estimation solver 178 determines scale 134 …
PNG
media_image1.png
50
170
media_image1.png
Greyscale
; [0052] The parameter k indicates the scale 134).
Regarding claim 9, the combination of Kar and KIM teaches all the limitations of claim 8 above. Kar teaches wherein the scale factor includes a relationship between the first … value and the second … value ([0048] The affine parameters 132 include scale 134 and shift 136 of the depth map 120. The scale 134 includes a scale value that indicates the amount of resizing of the depth map 120 ... The depth map transformer 126 is configured to transform the depth map 120 to the depth map 138 using the affine parameters 132 ... the scale 134 and the shift 136 include two numbers (e.g., s=scale, t=shift) which when multiplied and added to the value in each pixel at depth map 120 produce depth map 138 (e.g., D138 (x, y)=s*D120 (x, y)+t), where D120(x, y) is the value in depth map 120 at the pixel location (x, y)); [0051] the parameter estimation solver 178 determines scale 134 and shift 136 based on depth offsets between the depth estimates 108 and the depth map 120 …
PNG
media_image1.png
50
170
media_image1.png
Greyscale
; [0052] The parameter k indicates the scale 134 … li and lj may represent the metric depth values for two points (e.g., two pixels) in the depth estimates 108, and di and dj may represent the non-metric depth values for the corresponding two points (e.g., two pixels) in the depth map 120; The equation (1) clearly shows scale k is calculated using a value computed based on the two depth values (li, lj) and a value computed based on the two predicted depth values (di, dj)).
Kar does not expressly teach … representative … and … representative.
However, KIM teaches a relationship between the first representative value and the second representative value ([0081] a Gaussian probability distribution based on a mean μ and a standard deviation±σ of depth values of a pixel (x, y) included in the input image. The mean μ and the standard deviation a of the depth values of the pixel (x, y) may be statistical values output by any of the neural network; [0036] Each first statistical value may include a mean of depth values of a corresponding each of the plural pixels in the input image, based on probabilities of the depth values, and each second statistical value may include a variance or a standard deviation of the depth values of the corresponding each of the plural pixels in the input image, based on the probabilities of the depth values; [0028] synthesizing an output image based on the probability distribution and a reference image corresponding to the input image).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the method and device of Kar to incorporate the step/system of using a relationship (probability distribution) between a mean value computed based on depth values and variance based on predicted depth values taught by KIM.
Motivation for this combination has been stated in claim 1.
With respect to claim 11, arguments analogous to those presented for claim 1, are applicable.
With respect to claim 12, arguments analogous to those presented for claim 2, are applicable.
With respect to claim 13, arguments analogous to those presented for claim 3, are applicable.
With respect to claim 14, arguments analogous to those presented for claim 4, are applicable.
With respect to claim 18, arguments analogous to those presented for claim 8, are applicable.
With respect to claim 19, arguments analogous to those presented for claim 9, are applicable.
With respect to claim 21, arguments analogous to those presented for claim 1, are applicable.
With respect to claim 22, arguments analogous to those presented for claim 2, are applicable.
With respect to claim 23, arguments analogous to those presented for claim 3, are applicable.
With respect to claim 24, arguments analogous to those presented for claim 4, are applicable.
With respect to claim 27, arguments analogous to those presented for claim 8, are applicable.
With respect to claim 28, arguments analogous to those presented for claim 9, are applicable.
Claim 7, 17 and 29 are rejected under 35 U.S.C. 103 as being unpatentable over Kar et al. (U.S Publication No. 2022/0335638) (hereafter, "Kar") in view of KIM et al. (U.S. Publication No. 2022/0036573) (hereafter, "KIM") and further in view of ZHENG (U.S Publication No. 2018/0302649).
Regarding claim 7, the combination of Kar and KIM teaches all the limitations of claim 1 above.
Kim teaches wherein the first representative value includes a first statistical measure of the plurality of depth values or ([0036] Each first statistical value may include a mean of depth values of a corresponding each of the plural pixels in the input image, based on probabilities of the depth values) … wherein the second representative value includes a first statistical measure of the plurality of predicted depth values of the predicted depth map or ([0036] each second statistical value may include a variance or a standard deviation of the depth values of the corresponding each of the plural pixels in the input image, based on the probabilities of the depth values; [0065] the depth of the image may be represented as a depth map including values of depths for each of the pixels in the image).
KIM does not expressly teach … a second statistical measure value of the plurality of depth values, and … a second statistical measure of the plurality of predicted depth values of the predicted depth map.
However, ZHENG teaches a second statistical measure value of the plurality of depth values, and ([0152] S303. Obtain data of the current image block of the depth map; [0163] S304 ... calculate a second average value of the data of the current image block according to the pixel value of the data of the current image block) … a second statistical measure of the plurality of predicted depth values of the predicted depth map (S303 ... obtain prediction data corresponding to the current image block of the depth map, and obtain the predicted pixel value from the prediction data; [0163] S304. Calculate a first average value of the prediction data according to the predicted pixel value).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the method and device of Kar and KIM to incorporate the step/system of using the average value of depth data of the depth map and the average value of predicted data for coding a depth map taught by ZHENG.
The suggestion/motivation for doing so would have been to improve the efficiency of coding a depth map by reducing a calculation amount ([0006] methods and apparatuses for coding and decoding a depth map, improving coding and decoding efficiency; [0115] a calculation amount is remarkably reduced when the first average value of the prediction data of the current image block is acquired by means of calculation, improving coding efficiency). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results. Therefore, it would have been obvious to combine Kar and KIM with ZHENG to obtain the invention as specified in claim 7.
With respect to claim 17, arguments analogous to those presented for claim 7, are applicable.
With respect to claim 29, arguments analogous to those presented for claim 7, are applicable.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL C. CHANG whose telephone number is (571)270-1277. The examiner can normally be reached Monday-Thursday and Alternate Fridays 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan S. Park can be reached at (571) 272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL C CHANG/ Examiner, Art Unit 2669
/CHAN S PARK/ Supervisory Patent Examiner, Art Unit 2669
Prosecution Insights