DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/07/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 4-5, 9 and 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Luo et al. US PG-Pub(US 20260038134 A1) in view of Ran et al. ("Few-Shot Depth Completion Using Denoising Diffusion Probabilistic Model").
Regarding Claim 1, Luo teaches a depth map generating method(¶[0045], discloses “FIG. 2 is a flowchart of a depth map completion method”), comprising: acquiring an RGB color image(¶[0047] discloses “the scene image is a red green blue (RGB) three-channel color image.”) through a monocular camera provided in a robot system(¶[0048] disclose “, the scene image is an image obtained by an image acquisition device (such as camera) by photographing.”, the image is acquired by an image acquisition device such as a camera and ¶[0039] discloses performing monocular depth estimation which means the camera system could be a monocular camera. ; acquiring a 3D point cloud through a light detection and ranging (LiDAR) sensor provided in the robot system(¶[0052], “a depth map acquired by a depth perception device (such as laser radar, depth cameras).“, a depth map/3d point cloud is acquired from a laser radar/lidar and ¶[0239] further discloses the laser radar is coupled to an autonomous driving system to capture the driving scene.); generating a sparse depth map including only depth information for some points in a given space from the 3D point cloud (¶[0052], “the sparse depth map is a depth map acquired by a depth perception device (such as laser radar, depth cameras).” ¶[0053], “the sparse depth map may also be a depth map with some missing depth information obtained through preprocessing or various computations.” ); and generating a dense depth map including depth information for all points in the given space.(¶[0069]-¶[0072] disclose the idea of performing image restoration using a neural network based on depth completion to generate a dense depth map.)
Luo does not explicitly teach inputting the RGB color image and the sparse depth map into a pre-trained diffusion model;
Ran teaches inputting the RGB color image and the sparse depth map into a pre-trained diffusion model;(Page 6560, Left Col, Paragraph 4, “we first introduced DDPM into the depth completion tasks and proposed a few-shot learning paradigm for depth completion by pre-training a diffusion model on a mass of RGB data without corresponding depth annotations. Then only a limited number of samples with corresponding depth maps is needed to fine-tune a fusion module and achieve good results.”, discloses a pre-trained diffusion model that takes RGB and depth maps as inputs to generate a dense depth map.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Luo with Ran in order to input the RGB images and depth map into a diffusion model. One skilled in the art would have been motivated to modify Luo in this manner in order to propose a few shot learning paradigm for depth completion based on pre trained denoising diffusion probabilistic model. (Ran, Abstract)
Regarding Claim 2, the combination of Luo and Ran teach the depth map generating method of claim 1, where Ran further teaches further comprising: training the pre-trained diffusion model by using the sparse depth map as training data according to a predetermined setting. (Fig 1. “Our model takes an RGB image and a sparse depth map as input and predicts a dense depth map.” and discloses the model is trained on a smaller dataset with a frozen backbone.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Luo with Ran in order to train the diffusion model using the sparse map as input. One skilled in the art would have been motivated to modify Luo in this manner in order to apply the kernels to fuse multi-scale features for depth completion. (Ran, Page 6561, Left Col, Paragraph 1)
Regarding Claim 4, the combination of Luo and Ran teach the depth map generating method of claim 2, where Ran further teaches wherein training the pre-trained diffusion model includes: reading the predetermined setting; determining a condition to be given along with noise, as an input to the pre-trained diffusion model, when it is determined that the predetermined setting includes a second setting; concatenating the determined condition with the noise(Page 6561, Right Col, Paragraph 1, “A diffusion model takes a single RGB image as input and adds Gaussian noise into it step by step until it becomes an entire isotropic Gaussian noise image. Then it removes noise gradually and recovers the original image. They are called the forward process and the reverse process”, discloses adding noise until the original image becomes an entire isotropic Gaussian noise image and then performing a reverse process to remove the noise during the training.); and training the pre-trained diffusion model based on the condition concatenated with the noise(Page 6562, Self-Supervised pre-training, Paragraph 1,“The forward process of the diffusion model is adding Gaussian noise to the original image during the time step T until it becomes an isotropic Gaussian distribution. This can be seen as a Markov process because the status under time step It only depends on the previous status It−1”, discloses training the diffusion model with noise added to the original image).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Luo with Ran in order to train the diffusion model by concatenating noise to the original image as a training condition. One skilled in the art would have been motivated to modify Luo in this manner in order to apply the kernels to fuse multi-scale features for depth completion. (Ran, Page 6561, Left Col, Paragraph 1)
Regarding Claim 5, the combination of Luo and Ran teach the depth map generating method of claim 4, where Ran further teaches wherein: the condition includes any one of a first condition, a second condition, a third condition, a fourth condition, or a fifth condition, the first condition includes the sparse depth map, the second condition includes the RGB color image and the sparse depth map, the third condition includes the RGB color image, an edge image, and the sparse depth map, the fourth condition includes a gray image and the sparse depth map, and the fifth condition includes the gray image, the edge image, and the sparse depth map. . (Fig 1. “Our model takes an RGB image and a sparse depth map as input and predicts a dense depth map.” and discloses the model is trained on a smaller dataset with a frozen backbone. The Claim recites any one of the condition so under the BRI of the claim language the Examiner is providing claim mapping to only one of the conditions required in the claim limitation)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Luo with Ran in order to train the diffusion model using the sparse map as input. One skilled in the art would have been motivated to modify Luo in this manner in order to apply the kernels to fuse multi-scale features for depth completion. (Ran, Page 6561, Left Col, Paragraph 1)
Regarding Claim 9, the combination of Luo and Ran teach The depth map generating method of claim 2, where Ran further teaches wherein training the pre-trained diffusion model includes: using the generated dense depth map as a ground truth image. (Fig. 3 shows. From top to bottom: (a) original RGB images; (b) sparse LiDAR data; (c) ground truth. The ground truth depth image is used in the training dataset.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Luo with Ran in order to train the diffusion model using a ground truth depth image. One skilled in the art would have been motivated to modify Luo in this manner in order to propose a few shot learning paradigm for depth completion based on pre trained denoising diffusion probabilistic model. (Ran, Abstract)
Regarding Claim 13, claim 13 is considered an apparatus claim substantially corresponding to claim 1. Please see the discussion of claim 1 above for a discussion of similar limitations. Furthermore, Luo teaches a depth map generating apparatus(See Fig. 3), comprising: at least one memory device configured to store program code(See, ¶[0015]); and at least one processor configured(See, ¶[0015), by executing the program code stored in the at least one memory device(See, ¶[0015)
Regarding claim 14, it is substantially similar to claim 2 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Luo et al. US PG-Pub(US 20260038134 A1) in view of Ran et al. ("Few-Shot Depth Completion Using Denoising Diffusion Probabilistic Model") in view of Ke et al. ("Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation").
Regarding Claim 3, while the combination of Luo and Ran teach the depth map generating method of claim 2, they do not explicitly teach wherein training the pre-trained diffusion model includes: reading the predetermined setting; normalizing a depth value of the sparse depth map used as the training data to a value in a range of -1 to 1, when it is determined that the predetermined setting includes a first setting; and training the pre-trained diffusion model based on the sparse depth map on which the normalization has been performed.
Ke teaches wherein training the pre-trained diffusion model includes: reading the predetermined setting; normalizing a depth value of the sparse depth map used as the training data to a value in a range of -1 to 1 (Page 4, Section 3.3, Paragraph 1, “For the ground truth depth maps d, we implement a linear normalization such that the depth primarily falls in the value range [−1,1], to match the designed input value range of the VAE.”, discloses performing a normalization such that the value of the depth maps fall within a -1 to 1 range.)
when it is determined that the predetermined setting includes a first setting; and training the pre-trained diffusion model based on the sparse depth map on which the normalization has been performed. (Section 3.3, training on synthetic data discloses once normalization is performed the data is used to train the model.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Luo and Ran with Ke in order to normalize the depth maps before training. One skilled in the art would have been motivated to modify Luo and Ran in this manner in order to improve the prediction details and smoothness. (Ke, Page 3, Left Col, Paragraph 1)
Claim 6 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Luo et al. US PG-Pub(US 20260038134 A1) in view of Ran et al. ("Few-Shot Depth Completion Using Denoising Diffusion Probabilistic Model") in view of Zhang et al. ("Adding Conditional Control to Text-to-Image Diffusion Models").
Regarding Claim 6, while the combination of Luo and Ran teach the depth map generating method of claim 2, they do not explicitly teach wherein training the pre-trained diffusion model includes: reading the predetermined setting; giving the sparse depth map as a condition to each of internal layers constituting the pre-trained diffusion model, when it is determined that the predetermined setting includes a third setting; and training the pre-trained diffusion model to which the condition is given, based on the sparse depth map.
Zhang teaches wherein training the pre-trained diffusion model includes: reading the predetermined setting; giving the sparse depth map as a condition to each of internal layers constituting the pre-trained diffusion model(Page 3815, Section 3.1. ControlNet, “ControlNet injects additional conditions into the blocks of a neural network (Figure 2)”), when it is determined that the predetermined setting includes a third setting and training the pre-trained diffusion model to which the condition is given, based on the sparse depth map. (Figure 2 shows the model being trained by the condition given and Section 3.1 further discloses training the diffusion model based on conditional settings)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Luo and Ran with Zhang in order to train the diffusion model by conditions set. One skilled in the art would have been motivated to modify Luo and Ran in this manner in order to facilitate wider applications to control image diffusion models. (Zhang, Abstract).
Regarding claim 15, it is substantially similar to claim 6 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Claims 7 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Luo et al. US PG-Pub(US 20260038134 A1) in view of Ran et al. ("Few-Shot Depth Completion Using Denoising Diffusion Probabilistic Model") in view of Yu et al. US PG-Pub(US 20210150240 A1).
Regarding Claim 7, while the combination of Luo and Ran teach the depth map generating method of claim 2, they do not explicitly teach wherein training the pre-trained diffusion model includes: reading the predetermined setting; and training the pre-trained diffusion model by including a pixel having a value of 0 in a ground truth image in the training data, when it is determined that the predetermined setting includes a fourth setting.
Yu teaches reading the predetermined setting; and training the model by including a pixel having a value of 0 in a ground truth image in the training data when it is determined that the predetermined setting includes a fourth setting. (¶[0035], “the spoofing ground truth relative to depth can be set to zero. During training”, discloses the depth value of the ground truth is set to 0 and included in the training dataset and ¶[0051] further discloses the depth ground truth and material ground truth are prepared offline and loaded in during training.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Luo and Ran with Yu in order to include the depth values of 0 as ground truth data. One skilled in the art would have been motivated to modify Luo and Ran in this manner in order to improve the material recognition robustness. (Yu, ¶[0038])
Regarding claim 16, it is substantially similar to claim 7 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Claims 8 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Luo et al. US PG-Pub(US 20260038134 A1) in view of Ran et al. ("Few-Shot Depth Completion Using Denoising Diffusion Probabilistic Model") in view of Yoo et al. US PG-Pub(US 20230230265 A1).
Regarding Claim 8, the combination of Luo and Ran teach the depth map generating method of claim 2, they do not explicitly teach wherein training the pre-trained diffusion model includes: reading the predetermined setting; and training the pre-trained diffusion model without including a pixel having a value of 0 in a ground truth image in the training data, when it is determined that the predetermined setting includes a fifth setting.
Yoo teaches wherein training the model includes: reading the predetermined setting; and training the model without including a pixel having a value of 0 in a ground truth image in the training data, when it is determined that the predetermined setting includes a fifth setting. ([0051], “Depth loss according to the present embodiment is as follows.
.sub.depth (d.gt)=∥1.sub.{gt>0}⊙ (d−gt)∥.sub.1. [Equation 1]
Where d represents the final depth map output by generating unit 100, gt represents ground truth, and ∥1.sub.{gt>0}∥ represents valid depth pixels of ground truth data. [0052] Adversarial loss is used to train the generating unit 100 and discriminating unit 102.”, discloses training the model to reduce loss by excluding pixels that have a value of 0 as they are considered invalid pixels.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Luo and Ran with Yoo in order to train the model by excluding pixels with values below 0. One skilled in the art would have been motivated to modify Luo and Ran in this manner in order to propose a patch GAN-based depth completion method and apparatus in an autonomous vehicle that can improve performance. (Yoo, ¶[0009])
Regarding claim 17, it is substantially similar to claim 8 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Claims 10-12 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Luo et al. US PG-Pub(US 20260038134 A1) in view of Ran et al. ("Few-Shot Depth Completion Using Denoising Diffusion Probabilistic Model") in view of Yang et al. ("CN 115456909 A").
Regarding Claim 10, while the combination of Luo and Ran teach the depth map generating method of claim 1, they do not explicitly teach further comprising: searching for a pixel having a depth value of 0 among pixels constituting the dense depth map; and calculating a value of the pixel having the depth value of 0 based on pixels located around the searched pixel and having a depth value other than 0 to fill the dense depth map with the calculated value.
Yang teaches searching for a pixel having a depth value of 0 among pixels constituting the dense depth map(Page 3, Paragraph 1, discloses determining abnormal points in a depth map and Page 3 Paragraph 4 discloses “when the depth value of the abnormal point centroid in the abnormal area is non-zero, determining that the abnormal point type is the noise point”, when the value is 0 it indicates an abnormal point.);; and calculating a value of the pixel having the depth value of 0 based on pixels located around the searched pixel and having a depth value other than 0 to fill the dense depth map with the calculated value. (Page 7, Paragraph 8, “the first replacement unit 1031, for if the abnormal point type is a deletion point, then respectively judging whether the depth value of each vertex in the abnormal area is greater than the first threshold value, and the depth value of each vertex is greater than the first threshold value, the depth value of the deletion point is replaced by the average value of the missing point surrounding non-zero depth value;”, discloses once the abnormal point of 0 is detected in the depth map the point is replaced by an average value of the non-zero depth values around the point.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Luo and Ran with Yang in order to fill depth values that are zero. One skilled in the art would have been motivated to modify Luo and Ran in this manner in order to accurately de-noising the depth image. (Yang, Abstract)
Regarding Claim 11, the combination of Luo, Ran and Yang teach the depth map generating method of claim 10, where Yang further teaches wherein filling the dense depth map includes: calculating the value of the pixel having the depth value of 0 through a Gaussian random function(Page 5, Paragraph 1, “obtain the pre processed image, wherein the image filtering mode comprises but not limited to the square filtering, mean filtering and Gaussian filtering and linear filtering and median filtering and bilateral filtering non linear filtering, and if using mean filtering, then the kernel size of the filter can be 3 * 3, the default anchor point (i.e., the output pixel point of the filter) is the centre of the core.” , discloses using gaussian filtering to determine values that are 0 in the depth image.); and filling the dense depth map with the calculated value (Page 7, Paragraph 8, “the first replacement unit 1031, for if the abnormal point type is a deletion point, then respectively judging whether the depth value of each vertex in the abnormal area is greater than the first threshold value, and the depth value of each vertex is greater than the first threshold value, the depth value of the deletion point is replaced by the average value of the missing point surrounding non-zero depth value;”, discloses once the abnormal point of 0 is detected in the depth map the point is replaced by an average value of the non-zero depth values around the point.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Luo and Ran with Yang in order to use Gaussian filtering to determine depth values that are abnormal/zero. One skilled in the art would have been motivated to modify Luo and Ran in this manner in order to accurately de-noising the depth image. (Yang, Abstract)
Regarding Claim 12, the combination of Luo, Ran and Yang teach the depth map generating method of claim 10, where Yang further teaches wherein filling the dense depth map includes: calculating the value of the pixel having the depth value of 0 as an average of values included in a 3×3 filter surrounding the pixel having the depth value of 0(Page 5, Paragraph 1, “obtain the pre processed image, wherein the image filtering mode comprises but not limited to the square filtering, mean filtering and Gaussian filtering and linear filtering and median filtering and bilateral filtering non linear filtering, and if using mean filtering, then the kernel size of the filter can be 3 * 3, the default anchor point (i.e., the output pixel point of the filter) is the centre of the core.”, discloses a 3x3 filter around the point with the value of 0.) and filling the dense depth map with the calculated value. (Page 7, Paragraph 8, “the first replacement unit 1031, for if the abnormal point type is a deletion point, then respectively judging whether the depth value of each vertex in the abnormal area is greater than the first threshold value, and the depth value of each vertex is greater than the first threshold value, the depth value of the deletion point is replaced by the average value of the missing point surrounding non-zero depth value;”, discloses once the abnormal point of 0 is detected in the depth map the point is replaced by an average value of the non-zero depth values around the point.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Luo and Ran with Yang in order to use Gaussian filtering to determine depth values that are abnormal/zero. One skilled in the art would have been motivated to modify Luo and Ran in this manner in order to accurately de-noising the depth image. (Yang, Abstract)
Regarding claim 18, it is substantially similar to claim 10 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Regarding claim 19, it is substantially similar to claim 11 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Regarding claim 20, it is substantially similar to claim 12 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HAN D HOANG whose telephone number is (571)272-4344. The examiner can normally be reached Monday-Friday 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JOHN M VILLECCO can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAN HOANG/Primary Examiner, Art Unit 2661
Prosecution Insights