DETAILED ACTION
This action is in response to the application filed on August 10th, 2023. Claims 1-20 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on August 10th, 2023 is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 9 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. The claim term “reprojecting the reconstructed mesh into the plurality of image frames” is not clear and therefore the metes and bounds of the claim are also unclear. Specifically, it is unclear how a reprojection can be performed when no initial projection has been performed. Does this claim require an initial projection, followed by a reprojection after processing? For examination purposes, this claims is being interpreted as “projecting the reconstructed mesh into the plurality of image frames”.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 5-6, 8, 10-11, 14-15, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over US20210065392 (hereinafter referred to as Bleyer) in view of US20230419586 (herein after referred to as Kim).
In regards to claim 1, Bleyer teaches a method, comprising: receiving a plurality of image frames representing a scene (Bleyer Figure 1 Step 120; Figures 6, 7, and 12 Examiner note: Figure 1 includes capturing depth images of an environment, and figures 6, 7, and 12 are all examples of depth images captured of the environment.); determining a first depth prediction for the scene based on the plurality of image frames (Bleyer Paragraph [0047] “This first depth map can be generated in real-time (e.g., by an HMD) or it could have been generated at an earlier time and retained in storage (e.g., either locally on the HMD or in a cloud environment). As described earlier, a depth map can be generated from any number of depth images of the environment.” Examiner note: The depth images of this reference are analogous to the image frames of the present disclosure, and they all pertain to a single environment.); determining a reconstructed mesh from the plurality of image frames (Bleyer Paragraph [0003] “From the depth maps (and possibly the depth images), a surface reconstruction mesh and/or a three-dimensional (3D) point cloud can be computed to provide a three-dimensional digital representation of the environment”); determining a second depth prediction for the scene Bleyer Paragraph [0097] “Returning to FIG. 1, method 100 describes the above operations in act 125, which occurs after a second depth map (e.g., depth map 1100 of FIG. 11 or depth map 1300 of FIG. 13) is generated based on the second set of one or more depth image(s) (e.g., depth image 1000 of FIG. 10 or depth image 1200 of FIG. 12) that were captured using either the increased exposure time (e.g., increased exposure time 1015, which is used to acquire additional photons for dimly lit areas in the environment) or the decreased exposure time (e.g., decreased exposure time 1215, which is used to acquire fewer photons for brightly lit areas in the environment).”); and determining a third depth prediction based on the first depth prediction and the second depth prediction (Bleyer Figure 14 “Depth Map 1415”; Paragraph [0097] “Specifically, in act 125, selected portions of the second depth map are merged with the first depth map by replacing the invalid-depth pixels of the first depth map with corresponding newly acquired valid-depth pixels of the second depth map. FIG. 14 is representative of the operations described in act 125.”).
Bleyer fails to teach determining a depth prediction for the scene based on the reconstructed mesh.
However, Kim teaches determining a reconstructed mesh from the plurality of image frames (Kim Figure 13 Step S110-S130); determining a depth prediction for the scene based on the reconstructed mesh (Kim Figure 13 Step S140 Paragraph [0078] “step S140 of generating global depth maps for respective input images from the entire mesh in which tile-unit meshes are aggregated.”)
Kim is considered to be analogous to the claimed invention because they are both in the same field of creating a depth map of a scene. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Bleyer to include the teachings of Kim, to provide the advantage of a tile based mesh, which improves performance when compared to generating a single mesh for an entire area (Kim Paragraph [0061]-[0062] “Further, when the entire wide area terrain is generated as a single mesh, it is difficult to handle the mesh. That is, it is difficult to perform rendering and interaction on the single entire mesh at a subsequent application step… For this operation, in order to reconstruct a wide area terrain, a tiling procedure for partitioning the entire terrain into tiles of a suitable size is essentially required.”)
In regards to claim 2, Bleyer in view of Kim teaches the method of claim 1, wherein the third depth prediction is determined based on one or more criteria with respect to first depth values of the first depth prediction and second depth values of the second depth prediction. (Bleyer Figure 14; Paragraph [0097] “Specifically, in act 125, selected portions of the second depth map are merged with the first depth map by replacing the invalid-depth pixels of the first depth map with corresponding newly acquired valid-depth pixels of the second depth map. FIG. 14 is representative of the operations described in act 125.”)
In regards to claim 5, Bleyer in view of Kim teaches the method of claim 2, wherein determining the third depth prediction comprises: determining a fusion mask indicating reflective surfaces in the scene (Bleyer Paragraph [0042] “Instead of referring only to bright or dim areas, some embodiments rely on reflectivity measurements. For instance, the operations discussed herein for “bright” areas can also be performed for highly reflective surfaces and the operations discussed herein for “dim” areas can be performed for low reflective surfaces. Therefore, although the remaining portion of this disclosure focuses on dim and bright areas, the principles are equally applicable to low and high reflective surfaces, where a “high” reflective surface is any surface whose reflectivity satisfies an upper reflectivity threshold and where a “low” reflective surface is any surface whose reflectivity satisfies a lower reflectivity threshold.”; Figure 14; Paragraph [0095] “Depth map 1300 is shown as having accurate depth readings for the bright area(s) 1310, which correspond to the bright area(s) 1205, as illustrated by the valid-depth pixels 1315.” Examiner note: The first paragraph shows that while this reference generally refers to bright or dim areas, the bright or dim areas could be replaced by highly reflective or low reflective surfaces, respectively. The second paragraph teaches that the highly reflective (or bright areas) are used when determining the fusion of the first and second depth map. Figure 14 also shows the fusion of these depth maps.), wherein determining the third depth prediction is based on the fusion mask. (Bleyer Paragraph [0102] “FIG. 14 shows how the data from the valid-depth pixel(s) 1405B and the data from the valid-depth pixel(s) 1410A (i.e. selected portions of the two different subsequently-generated depth maps) can be merged with the first depth map 1400 in order to compensate or correct for the deficiencies (e.g., invalid-depth pixel(s) 1400A and 1400B) in that first depth map 1400.” Examiner note: The valid depth pixels of the bright/highly reflective surface are used to create a valid depth map.)
In regards to claim 6, Bleyer in view of Kim teaches the method of claim 1, wherein the plurality of image frames comprise image frames corresponding to a plurality of camera poses within the scene (Bleyer Figures 6 and 7; Paragraph [0059] “FIG. 6 illustrates a scenario in which multiple depth images are being captured for environment 300 of FIG. 3, as shown by the illustration labeled depth image capture 600.”; Paragraph [0060] “FIG. 7 illustrates how a first/initial depth map 700 can be generated based on the depth information included within any number of depth images (e.g., the depth images 605 from FIG. 6).” Examiner note: The depth images of figure 6 are combined to create the depth map of figure 7. Furthermore, the depth images of figure 6 are captured from 2 cameras in different positions/poses, as can be seen in figure 2 “Stereoscopic Depth Camera System”).
In regards to claim 8, Bleyer in view of Kim teaches the method of claim 1, further comprising determining a second reconstructed mesh based on the third depth prediction. (Bleyer Paragraph [0003] “From the depth maps (and possibly the depth images), a surface reconstruction mesh and/or a three-dimensional (3D) point cloud can be computed to provide a three-dimensional digital representation of the environment”; Paragraph [0102] “FIG. 14 shows how the data from the valid-depth pixel(s) 1405B and the data from the valid-depth pixel(s) 1410A (i.e. selected portions of the two different subsequently-generated depth maps) can be merged with the first depth map 1400 in order to compensate or correct for the deficiencies (e.g., invalid-depth pixel(s) 1400A and 1400B) in that first depth map 1400. As a consequence of the merging operation, a newly merged depth map 1415 is generated and now includes accurate depth readings at locations that were previously invalid.”)
In regards to claim 10, Bleyer in view of Kim teaches a memory storing processor-readable code; and at least one processor coupled to the memory, the at least one processor configured to execute the processor-readable code to cause the at least one processor to perform operations (Bleyer Paragraph [0152] “In its most basic configuration, computer system 2000 includes various different components. FIG. 20 shows that computer system 2000 includes at least one processor(s) 2005 (aka a “hardware processing unit”), input/output (“I/O”) 2010, camera system 2015 (which is representative of the depth sensors described throughout this disclosure and which may include an illuminator 2015A), and storage 2020.”) and renders obvious the remaining claim limitations as in the consideration of claim 1.
In regards to claim 11, Bleyer in view of Kim renders obvious the claim limitations as in the consideration of claims 2 and 10.
In regards to claim 14, Bleyer in view of Kim renders obvious the claim limitations as in the consideration of claims 5 and 11.
In regards to claim 15, Bleyer in view of Kim renders obvious the claim limitations as in the consideration of claims 6 and 10.
In regards to claim 18, Bleyer in view of Kim teaches a non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations (Bleyer Paragraph [0155] “Storage 2020 may be physical system memory, which may be volatile, non-volatile, or some combination of the two. The term “memory” may also be used herein to refer to non-volatile mass storage such as physical storage media”) and renders obvious the remaining claim limitations as in the consideration of claim 1.
In regards to claim 19, Bleyer in view of Kim renders obvious the claim limitations as in the consideration of claims 2 and 18.
Claims 3 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Bleyer in view of Kim as applied to the claims above, and further in view of US11520041 (herein after referred to as Eubank).
In regards to claim 3, Bleyer in view of Kim teaches the method of claim 2, but fails to teach wherein the one or more criteria comprises: for each corresponding first depth value of the first depth values and second depth value of the second depth values for the scene: determining a difference between the first depth value and the second depth value; when the difference meets a threshold value, selecting the second depth value for the third depth prediction; and when the difference fails to meet the threshold value, selecting the first depth value for the third depth prediction.
However, Eubank teaches wherein the one or more criteria comprises: for each corresponding first depth value of the first depth values and second depth value of the second depth values for the scene: determining a difference between the first depth value and the second depth value (Eubank Figure 9B 9-11 “a difference between the first depth estimation and the second depth estimation”); when the difference meets a threshold value, selecting the second depth value for the third depth prediction (Eubank Figure 9B 9-12 “Does a difference between the first depth estimation and the second depth estimation exceed a threshold value? YES”); and when the difference fails to meet the threshold value, selecting the first depth value for the third depth prediction (Eubank Figure 9B 9-13 “Does a difference between the first depth estimation and the second depth estimation exceed a threshold value? NO”).
Eubank is considered to be analogous to the claimed invention because they are both in the same field of using multiple depth predictions to improve depth mapping. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Bleyer in view of Kim to include the teachings of Eubank, to provide the advantage of resolving any errors that appear when performing depth map matching (Eubank Column 16 Line 64 “In some implementations, the device determines a confirmed depth estimation based on resolving any mismatch between the first depth estimation and the second depth estimation.”)
In regards to claim 12, Bleyer in view of Kim and Eubank renders obvious the claim limitations as in the consideration of claims 3 and 11.
Claims 4, 7, are rejected under 35 U.S.C. 103 as being unpatentable over Bleyer in view Kim as applied to the claims above, and further in view of CN118115347 (herein after referred to as Li).
In regards to claim 4, Bleyer in view of Kim teaches the method of claim 2, but fails to teach wherein determining the first depth prediction is based on a depth model, and the method further comprises: training the depth model with the third depth prediction.
However, Li teaches wherein determining the first depth prediction is based on a depth model (Li Page 5 Paragraph 9 “In a fourth aspect, the present invention also provides a depth sensor configured to obtain depth information of a target space and to calibrate the depth information by a machine learning model obtained using a training method as described above.”), and the method further comprises: training the depth model with the third depth prediction (Li Page 5 Paragraph 8 “inputting the corresponding third depth information and the fused depth information into a machine learning model for training to obtain model parameters.”).
Li is considered to be analogous to the claimed invention because they are both in the same field of using multiple depth predictions to improve depth mapping. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Bleyer in view of Kim to include the teachings of Li, to provide the advantage of increase depth data precision (Li Page 9 Paragraph 2 “constructing a machine learning model, taking the depth true value and the target value as a group of training data to input the machine learning model, the trained machine learning model is used for calibrating the depth data collected by the target depth sensor, so as to improve the precision of the depth data.”)
In regards to claim 7, Bleyer in view of Kim teaches the method of claim 6, but fails to teach wherein the first depth prediction is based on a self-supervised model operating on the plurality of image frames.
However, Li teaches wherein the first depth prediction is based on a self-supervised model operating on the plurality of image frames. (Li Page9 Paragraph 2 “the trained machine learning model is used for calibrating the depth data collected by the target depth sensor, so as to improve the precision of the depth data.” Examiner note: The machine learning model of this disclosure is used to improve the depth prediction of the target depth sensor, which is analogous to basing the depth prediction on the model.)
Li is considered to be analogous to the claimed invention because they are both in the same field of using multiple depth predictions to improve depth mapping. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Bleyer in view of Kim to include the teachings of Li, to provide the advantage of increase depth data precision (Li Page 9 Paragraph 2 “constructing a machine learning model, taking the depth true value and the target value as a group of training data to input the machine learning model, the trained machine learning model is used for calibrating the depth data collected by the target depth sensor, so as to improve the precision of the depth data.”)
In regards to claim 13, Bleyer in view of Kim and Li renders obvious the claim limitations as in the consideration of claims 4 and 11.
In regards to claim 16, Bleyer in view of Kim and Li renders obvious the claim limitations as in the consideration of claims 7 and 15.
In regards to claim 20, Bleyer in view of Kim and Li renders obvious the claim limitations as in the consideration of claims 4 and 18.
Allowable Subject Matter
Claims 9 and 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Claim 9: This claim requires a projection of a reconstructed mesh into a plurality of images frames, specifically by using the third depth prediction (which is a combination of the first and second depth prediction) to block reflective artifacts in the first depth prediction. The prior art of record does not include these limitations. The prior art of record does use a third “fused” depth prediction to block reflective artifacts from appearing in depth maps, but does not specifically use the third depth prediction to block reflective artifacts within the first depth prediction. This limitation is quite narrow, as the third depth prediction is based upon the first depth prediction, so this claim requires using the “fused” depth map output to block artifacts within the first “unfused” depth map. The remaining limitation “training a self-supervised depth model with the third depth prediction to mitigate reflective artifacts” is taught by Bleyer in view of Kim and Li, as shown with consideration to claims 4 and 7. A close prior art “Dealing with Laser Scanner Failure: Mirrors and Windows” teaches a method of fusing depth predictions to identify mirrors within a scene, and using the mirrors location to correct the depth predictions. More specifically, this reference stores two individual depth predictions, and fuses them as needed. This prior art does teach using the third depth prediction to block artifacts in the first depth prediction, but does not do this by reprojecting the reconstructed mesh into image frames. Finally, another close prior art, US20250117956, teaches (in figure 3) using a 3D scan to identify reflective surfaces, and then removes those reflective surfaces from the 3D scan. This prior art teaches using a depth prediction to block artifacts, but does not use the third depth prediction to block artifacts in the first depth prediction. For at least the reasons stated above, this claim is not taught by the prior art of record.
Claim 17: This claim is not taught by the prior art for the same reasons discussed with respect to claim 9.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
“Polarization structured light 3D depth image sensor for scenes with reflective surfaces” teaches a method of using multiple depth sensors to identify reflective surfaces and eliminate any incorrect depth predictions. This art does not meet the effective filing date of the disclosed invention
US20190246000 teaches a method of determining multiple models of a scene, and comparing those models to determine if there is an object that easily reflects light.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CALEB LOGAN ESQUINO whose telephone number is (703)756-1462. The examiner can normally be reached M-Th 7:00AM-5:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CALEB L ESQUINO/ Examiner, Art Unit 2677
/ATIBA O FITZPATRICK/ Primary Examiner, Art Unit 2677