DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 11/24/2025 have been fully considered but they are not persuasive.
Regarding rejection on claim 1, applicant argued that Jiang fails to teaching a grayscale image. Applicant then argued that technical approaches employed in Das and Jiang for performing depth completion are fundamentally different, which would not have motivation to combine the teachings of Das and Jiang to arrive at the claimed invention. Last, applicant argued that the single capturing device as described in paragraph 0034 of instant application adopts the grayscale image (rather than RGB/color image) for feeding into the rear-stage neural network model, such that the rear-stage neural network model can be realized on resource-limited and low-power devices, which is unexpected in view of Jiang and/or Das.
However, examiner respectfully disagrees. In response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In this case, both Jiang and Das teach using graph-based neural network to process depth completion. Despite of applicant’s argument, Jiang does not exclude using grayscale image from being input. It would not be unexpected for one of ordinary skill in the art to input a grayscale/monochrome image instead of a RGB/color image into a neural network model of Jiang, especially in view of Das’ teaching on having input image options of a RGB image, a grayscale image, etc. (paragraphs 0034, 0036, 0060, 0062, 0066). In response to applicant's argument that adopting the grayscale image for feeding into the rear-stage neural network model, such that the rear-stage neural network model can be realized on resource-limited and low-power devices, a recitation of the intended use of the claimed invention must result in a structural difference between the claimed invention and the prior art in order to patentably distinguish the claimed invention from the prior art. If the prior art structure is capable of performing the intended use, then it meets the claim.
Thus, rejection is proper and maintained.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jiang et al. (“A Low Memory Footprint Quantized Neural Network for Depth Completion of Very Sparse Time-of-Flight Depth Maps”) in view of Das et al. (US2024/0273742).
To claim 1, Jiang teach a depth completion method of sparse depth map, comprising:
acquiring an image and a sparse depth map corresponding to the image (Fig. 2a, input RGB image and corresponding sparse depth map);
obtaining a nearest neighbor interpolation (NNI) image and a Euclidean distance transform (EDT) image based on the sparse depth map (Fig. 2a, NNI and EDT from sparse depth map);
inputting the image, the NNI image, and the EDT image into a neural network model, thereby outputting a predicted residual map (Fig. 2a; section 4.2, computes a residual w.r.t. the initial guess); and
generating a predicted dense depth map according to the predicted residual map and the NNI image (Fig. 2a, interpolated depth map; section 4.2).
But, Jiang do not expressly said image being a grayscale image.
However, input image being RGB image or grayscale image would have been an obvious design preference.
Das teach depth completion can be performed to infer the dense depth map of a three-dimensional scene given an input image (e.g., RGB image, grayscale image, etc.) and a corresponding sparse reconstruction of the input image (e.g., in the form of a sparse depth map, obtained from computational techniques or active sensors such as lidar, structured light sensors, ToF sensors, etc.) (paragraph 0034).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate teaching of Das into the method of Jiang, in order to apply grayscale image by design preference.
To claim 2, Jiang and Das teach claim 1.
Jiang and Das teach wherein the predicted dense depth map is generated by adopting a pixel-level addition method according to the predicted residual map and the NNI image, wherein the predicted residual map includes residual information of the NNI image (Jiang, section 4.2).
To claim 3, Jiang and Das teach claim 1.
Jiang and Das teach wherein the grayscale image and the sparse depth map are acquired by using a time-of-flight (ToF) sensor (Jiang, Introduction, section 4).
To claim 4, Jiang and Das teach claim 1.
Though Jiang and Das do not expressly disclose further comprising: performing a down-sampling process on the grayscale image, the NNI image, and the EDT image before the grayscale image, the NNI image, and the EDT image are inputted into the neural network model; and performing an up-sampling process on the predicted dense depth map; wherein the down-sampling process and the up-sampling process are performed by bilinear interpolation with antialiasing, claimed features on down-sampling input images and up-sampling output images are well-known techniques in the art to improve computational efficiency, which would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate, hence Official Notice is taken.
To claim 5, Jiang and Das teach claim 1.
Jiang and Das teach wherein the neural network model extracts features of the grayscale image, the NNI image, and the EDT image by adopting an encoder-decoder fashion based on a UNet network architecture (Jiang, Fig. 2b).
To claim 6, Jiang and Das teach claim 1.
Though Jiang and Das do not expressly disclose further comprising: performing a model pruning operation on the neural network model to compress the neural network model, pruning is well-known technique in the art for neural network compression, which would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate, hence Official Notice is taken.
To claim 7, Jiang and Das teach claim 6.
Though Jiang and Das do not expressly disclose wherein the model pruning operation is merely performed on plural target layers of the neural network model, wherein a number of weights of each of the target layers is larger than a threshold, claimed feature of pruning operation is well-known feature in the art for pruning, hence Official Notice is taken.
To claim 8, Jiang and Das teach claim 6.
Though Jiang and Das do not expressly disclose further comprising: performing a model clustering operation on the neural network model to further compress the neural network model after the model pruning operation is performed, clustering after pruning operation is well-known technique in the art for compression, hence Official Notice is taken.
To claim 9, Jiang and Das teach claim 8.
Though Jiang and Das do not expressly disclose wherein the model clustering operation is merely performed on plural target layers of the neural network model, wherein a number of weights of each of the target layers is larger than a threshold, claimed feature of clustering operation is well-known feature in the art for clustering, hence Official Notice is taken.
To claim 10, Jiang and Das teach claim 1.
Jiang and Das teach further comprising: quantizing the neural network model from a floating-point number model to an integer model (Jiang, section 2.2).
To claim 11, Jiang and Das teach a system for depth completion of sparse depth map (as explained in response to claims 1 and 3 above).
To claim 12, Jiang and Das teach claim 11.
Jiang and Das teach wherein the processor generates the predicted dense depth map according to the predicted residual map and the NNI image by adopting a pixel-level addition method, wherein the predicted residual map includes residual information of the NNI image (as explained in response to claim 2 above).
To claim 13, Jiang and Das teach claim 12.
Jiang and Das teach wherein the processor is further configured to: perform a down-sampling process on the grayscale image, the NNI image, and the EDT image before the grayscale image, the NNI image, and the EDT image are inputted into the neural network model; and perform an up-sampling process on the predicted dense depth map; wherein the down-sampling process and the up-sampling process are performed by bilinear interpolation with antialiasing (as explained in response to claim 4 above).
To claim 14, Jiang and Das teach claim 11.
Jiang and Das teach wherein the neural network model extracts features of the grayscale image, the NNI image, and the EDT image by adopting an encoder-decoder fashion based on a UNet network architecture (as explained in response to claim 5 above).
To claim 15, Jiang and Das teach claim 11.
Jiang and Das teach wherein the processor is further configured to: perform a model pruning operation on the neural network model to compress the neural network model (as explained in response to claim 6 above).
To claim 16, Jiang and Das teach claim 15.
Jiang and Das teach wherein the model pruning operation is merely performed on plural target layers of the neural network model, wherein a number of weights of each of the target layers is larger than a threshold (as explained in response to claim 7 above).
To claim 17, Jiang and Das teach claim 16.
Jiang and Das teach wherein the processor is further configured to: perform a model clustering operation on the neural network model to further compress the neural network model after the model pruning operation is performed (as explained in response to claim 8 above).
To claim 18, Jiang and Das teach claim 17.
Jiang and Das teach wherein the model clustering operation is merely performed on plural target layers of the neural network model, wherein a number of weights of each of the target layers is larger than a threshold (as explained in response to claim 9 above).
To claim 19, Jiang and Das teach claim 11.
Jiang and Das teach wherein the processor is further configured to: quantize the neural network model from a floating-point number model to an integer model (as explained in response to claim 10 above).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZHIYU LU whose telephone number is (571)272-2837. The examiner can normally be reached Weekdays: 8:30AM - 5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen R Koziol can be reached at (408) 918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
ZHIYU . LU
Primary Examiner
Art Unit 2669
/ZHIYU LU/Primary Examiner, Art Unit 2665 January 3, 2026