DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 5 is objected to because of the following informalities:
Claim 5 is objected to for depending on itself. It appears claim 5 should depend on claim 3.
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitations use a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are: “an acquisition unit configured to estimate an illumination and to acquire a scalar depth value” in claim 14 and “the acquisition unit is configured to estimate the illumination based on an input of a color and a depth stream acquired using an RGB-D camera” in claim 15.
Because these claim limitations are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have these limitations interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitations to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitations recite sufficient structure to perform the claimed function so as to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-20 are rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Zollhofer et. al (“Shading-based Refinement on Volumetric Signed Distance Functions”).
Regarding Claim 1, Zollhofer teaches a three-dimensional (3D) scanning method comprising:
Abstract: “We present a novel method to obtain fine-scale detail in 3D reconstructions generated with low-budget RGB-D cameras or other commodity scanning devices.”
acquiring high-detail geometry by integrating a volumetric fusion and multiview shape-from-shading (SfS).
Abstract: “In our approach, we leverage RGB data to refine these reconstructions through shading cues, as color input is typically of much higher resolution than the depth data. As a result, we obtain reconstructions with high geometric detail, far beyond the depth resolution of the camera itself…we formulate the inverse shading problem on the volumetric distance field, and present a novel objective function which jointly optimizes for fine-scale surface geometry and spatially-varying surface reflectance.”
Implicit Surface Generation: “A TSDF enables the regularization and noise aware integration of noisy input depth data (e.g., obtained from commodity RGB-D sensors), and is a versatile representation that has been used in many scanning pipelines and surface reconstruction approaches…Similar to Kinect Fusion [Newcombe et al. 2011; Izadi et al. 2011], we incrementally align and integrate RGB-D images to get an initial coarse signed distance field D (Section 4.3) which is subsequently geometrically refined (Section 5).
Explanation: Zollhofer teaches 3D reconstruction from RGB-D scanning, refinement to obtain fine geometric detail, volumetric fusion using TSDF, and shading-based refinement from multi-view RGB-D data.
Regarding Claim 2, Zollhofer teaches the 3D scanning method of claim 1, wherein high-resolution geometry and texture are acquired from RGB-D stream data by performing the acquiring in real time.
Abstract: “As color input is typically of much higher resolution than the depth data. As a result, we obtain reconstructions with high geometric detail, far beyond the depth resolution of the camera itself.”
Introduction: “Allow consumers to densely scan objects at real-time frame rates.”
Overview: “We first capture input color and depth using commodity sensors (e.g., Microsoft Kinect). This yields a sequence of depth images Di and color images Ci, which we use to generate an implicit surface representing the scanned scene (Section 4.3).”
Regarding Claim 3, Zollhofer teaches the 3D scanning method of claim 1, wherein the acquiring comprises:
a first stage of estimating an illumination and acquiring a scalar depth value;
Introduction: “From globally-aligned color and depth input, we estimate the incident lighting distribution…”
Overview: “To accommodate general and uncontrolled lighting environments, we continually estimate incident irradiance.”
Integration of Depth Images: “Similar to Kinect Fusion [Newcombe et al. 2011; Izadi et al. 2011], we incrementally compute the fused implicit signed distance field D by integrating depth data from the individual depth images Di.”
a second stage of estimating a photometric normal and a diffuse albedo using the scalar depth value acquired from the estimated illumination;
Abstract: “We formulate the inverse shading problem on the volumetric distance field, and present a novel objective function which jointly optimizes for fine-scale surface geometry and spatially-varying surface reflectance.”
Overview: “We rephrase the inverse shading problem for an implicit TSDF surface, simultaneously optimizing for refined surface geometry and dense, spatially-varying albedo.”
Refinement of Signed Distance Functions: “On a signed distance field, surface normals n ∈ R 3 are given by the gradient operator. In our case, we express the normals by the gradient of the refined signed distance function D…”
and a third stage of integrating the scalar depth value to a volumetric distance field, refining the photometric normal and the diffuse albedo and blending the photometric normal and the diffuse albedo in a texture space.
Integration of Depth Images: “Similar to Kinect Fusion [Newcombe et al. 2011; Izadi et al. 2011], we incrementally compute the fused implicit signed distance field D by integrating depth data from the individual depth images Di.”
Overview & Integration of Depth Images: “These dense albedos, as opposed to coarsely aligned clusters of albedos as in many previous shading-based refinement methods (e.g., [Wu et al. 2011; Wu et al. 2013]), further inform the lighting estimation and enable more accurate shape refinement results…this yields the refined scene model D˜ with fine-scale detail from the RGB data mapped to the shape model… in addition, we integrate color data from input RGB images Ci to obtain the volumetric color function C in the same manner…later, per-voxel color values are used as shading constraints to determine the refined distance field D˜ (see Section 5.2).”
Regarding Claim 4, Zollhofer teaches the 3D scanning method of claim 3, wherein the first stage comprises estimating the illumination based on an input of a color and a depth stream acquired using an RGB-D camera.
Introduction: “From globally-aligned color and depth input, we estimate the incident lighting distribution…”
Overview: “We first capture input color and depth using commodity sensors (e.g., Microsoft Kinect).
Regarding Claim 5, Zollhofer teaches the 3D scanning method of claim 5, wherein the third stage comprises refining the photometric normal and the diffuse albedo in real time through geometry-aware texture warping and blending the photometric normal and the diffuse albedo in the texture space.
Abstract: “A tailored GPU-based Gauss-Newton solver enables us to refine large shape models to previously unseen resolution within only a few seconds.”
Overview: “This yields the refined scene model D˜ with fine-scale detail from the RGB data mapped to the shape model…we rephrase the inverse shading problem for an implicit TSDF surface, simultaneously optimizing for refined surface geometry and dense, spatially-varying albedo.”
Regarding Claim 6, Zollhofer teaches the 3D scanning method of claim 3, wherein the second stage comprises estimating the photometric normal and the diffuse albedo using the scalar depth value acquired from the estimated illumination and the multiview SfS.
Abstract: “Our core contribution is shading-based refinement directly on the implicit surface representation, which is generated from globally-aligned RGB-D images. We formulate the inverse shading problem on the volumetric distance field, and present a novel objective function which jointly optimizes for fine-scale surface geometry and spatially-varying surface reflectance.”
Introduction: “A formulation of the inverse shading problem on a TSDF, allowing for joint optimization of fine-scale geometric detail and dense, spatially-varying albedo (Section 5).”
Refinement of Signed Distance Functions: “On a signed distance field, surface normals n ∈ R 3 are given by the gradient operator. In our case, we express the normals by the gradient of the refined signed distance function D…”
Regarding Claim 7, Zollhofer teaches the 3D scanning method of claim 6, wherein the second stage comprises acquiring a color and a depth stream as an input using an RGB-D camera under the illumination and estimating an approximate value to the illumination using a spherical harmonics coefficient.
Overview: “We first capture input color and depth using commodity sensors (e.g., Microsoft Kinect).
Refinement of Signed Distance Functions: “For Lambertian reflectance, the incident irradiance at a point p is known to be smooth, and can be efficiently represented using spherical harmonics… typically, a good approximation is given by the first nine spherical harmonics basis functions; i.e., up to 2nd order.”
Regarding Claim 8, Zollhofer teaches the 3D scanning method of claim 3, wherein the second stage estimating the photometric normal and the diffuse albedo through iterative optimization.
Overview: “We optimize our new objective function with a custom GPU-based parallel Gauss-Newton optimizer which allows to solve for tens of millions of variables within a few seconds.”
Geometry Refinement and Albedo Estimation on Signed Distance Fields: “This involves solving a non-linear optimization problem to minimize a similarity measure… a better, yet more complex strategy, is to simultaneously optimize for unknown albedos and refined geometry.”
Regarding Claim 9, Zollhofer teaches the 3D scanning method of claim 8, wherein the second stage comprises:
estimating the photometric normal by optimizing the scalar depth value and optimizing the scalar depth value and the diffuse albedo by minimizing an energy function (Fig. 9);
Geometry Refinement and Albedo Estimation on Signed Distance Fields: “This involves solving a non-linear optimization problem to minimize a similarity measure…our method solves for refined geometry D˜ and unknown albedos a in a combined global optimization problem…the unknowns are optimized by solving the following non-linear least squares problem…”
PNG
media_image1.png
204
714
media_image1.png
Greyscale
and minimizing the energy function as follows: E (D, a) = Edata + λdregEdreg + λdsensorEdsensor + λaregEareg + λatempEatemp, where Edata denotes shading data, Edreg denotes a spatial regularization, Edsensor denotes a depth constraint, Adreg and Adsensor denote corresponding weights for depth regularization and constraint, respectively, Eareg and Eatemp denote spatial and temporal regularizers of albedo, respectively, and Aareg and Aatemp denote corresponding weights for albedo regularizers, respectively (Geometry Refinement and Albedo Estimation on Signed Distance Fields).
PNG
media_image2.png
145
359
media_image2.png
Greyscale
Regarding Claim 10, Zollhofer teaches the 3D scanning method of claim 3, wherein the third stage comprises achieving real-time multiview SfS by progressively refining a geometry registration between the texture space and the volumetric distance field using a normal texture.
Fig. 2 Caption: “Afterwards, hierarchical shading-based refinement based on RGB input is used to add fine scale detail to the reconstruction.”
Overview & Integration of Depth Images: “This yields the refined scene model D˜ with fine-scale detail from the RGB data mapped to the shape model.”
Refinement of Signed Distance Functions: “On a signed distance field, surface normals n ∈ R 3 are given by the gradient operator. In our case, we express the normals by the gradient of the refined signed distance function D…”
Regarding Claim 11, Zollhofer teaches the 3D scanning method of claim 10, wherein the third stage comprises optimizing a geometry correspondence between normals in the texture space and geometry in a canonical space of a truncated signed distance function (TSDF).
Implicit Surface Generation: “We follow Curless and Levoy [1996], and represent surface geometry of a scanned scene using a truncated signed distance function (TSDF), denoted as D.”
Refinement of Signed Distance Functions: “In order to refine a surface based on the shading constraint, we need to solve for the per-voxel albedo a, per-scene lighting l, and most importantly, the per-voxel normal n, which is directly coupled to the underlying signed distance function (see next paragraph).
Regarding Claim 12, Zollhofer teaches the limitations of claims 1, 2, 3, and 10 above because claim 12 recites a method that amounts to substantially the same steps.
Regarding Claim 13, Zollhofer teaches the 3D scanning method of claim 12, wherein the integrating comprises integrating the scalar depth value to a canonical space of a truncated signed distance function (TSDF) by refining a depth through inverse rendering.
Introduction: “Core to these approaches is the underlying surface representation of a truncated signed distance field (TSDF) [Curless and Levoy 1996]. This representation stores the distance values to the closest surface point in 3D in a voxel grid.”
Integration of Depth Images: “Similar to Kinect Fusion [Newcombe et al. 2011; Izadi et al. 2011], we incrementally compute the fused implicit signed distance field D by integrating depth data from the individual depth images Di.”
Regarding Claim 14, Zollhofer teaches the limitations of claims 1, 2, and 3 above because claim 14 recites a system that amounts to substantially the same steps.
Regarding Claim 15, Zollhofer teaches the system of claim 14, and additional limitations are met as in the consideration of claim 4 above.
Regarding Claim 16, Zollhofer teaches the system of claim 14, and additional limitations are met as in the consideration of claim 5 above.
Regarding Claim 17, Zollhofer teaches the system of claim 14, and additional limitations are met as in the consideration of claim 6 above.
Regarding Claim 18, Zollhofer teaches the system of claim 14, and additional limitations are met as in the consideration of claim 9 above.
Regarding Claim 19, Zollhofer teaches the system of claim 14, and additional limitations are met as in the consideration of claim 10 above.
Regarding Claim 20, Zollhofer teaches the system of claim 14, and additional limitations are met as in the consideration of claim 11 above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Du (CN111656407A) discloses a real-time RGB-D 3D reconstruction method that updates a dynamic 3D model by fusing depth maps and texture information, refining geometry and appearance through multi-view integration and optimization.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM ADU-JAMFI whose telephone number is (571)272-9298. The examiner can normally be reached M-T 8:00-6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM ADU-JAMFI/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677