DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 05/20/2024 was filed after the mailing date of the application. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 9-12 recite the limitation “the predicting the second surface normal vector”. There is insufficient antecedent basis for this limitation in the claim. The limitation “the predicting the second surface normal vector”, which is critical in interpreting claims 9-12, was missing in independent claim 1. Although claim 1 recites “estimating a second surface normal vector”, the steps of “estimating” and “predicting” are two distinct processes, and cannot be assumed to be the same.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 6-7, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Dave et al. (US. Patent App. Pub. No. 2022/0335682, “Dave”, hereinafter).
As per claim 1, as shown in Fig. 3-5, Dave teaches a method of determining an illumination pattern, the method comprising:
constructing a dataset by estimating a first surface normal vector of a three-dimensional (3D) object (such as 3D material design, ¶ [23]) from a first image obtained by capturing the 3D object of which surface normal information is known, the dataset comprising basis images of the 3D object (¶ [35], “The normal approximation is a material map with approximate normal values representing approximate surface geometry of the object depicted in the captured images. In some embodiments, the normal approximation is saved in a Red-Green-Blue (RGB) format where each pixel value represents a 3D vector indicating the direction in which the surface normal is pointing”. See also ¶ [77] referring to step 604 of Fig. 6);
generating simulation images in which virtual illumination patterns, obtained based on a combination of the basis images, are applied to the 3D object (¶ [80]);
estimating a second surface normal vector of the 3D object, by reconstructing a surface normal using a photometric stereo technique (further addressed below) based on the virtual illumination patterns and simulation images corresponding to the virtual illumination patterns (¶ [78], Fig. 6, step 606, generating a different type of material map, including a normal material map beside the normal map generated in step 604); and
training a neural network to determine an illumination pattern based on a difference between the first surface normal vector and the second surface normal vector (¶ [56] with reference to step 608 of Fig. 6, and ¶ [79-81], generating an updated set of material maps to reduce or eliminate the difference of the input images and the initial rendered images based on the lighting patterns).
Dave does not expressly teach the second normal vector estimation using the photometric stereo, but instead, using a neural network. However, as with the first normal vector estimation addressed above, Dave does teach estimating the first normal vector using a photometric stereo technique. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to reuse the photometric stereo technique for the second normal vector estimation using the simulated light patterns, instead of using a neural network since this is well within the level of ordinary skill in the art and can reduce computational resource.
As per claim 2, Dave also teaches wherein the constructing the dataset comprises: obtaining the first image by capturing the 3D object under a basis illumination (¶ [27]); and
estimating the first surface normal vector from the first image by using a differentiable rendering technique (¶ [3]).
As per claim 6, Dave impliedly teaches wherein the estimating the first surface normal vector comprises:
aligning the first image and a second image, which is rendered by using the differentiable rendering technique, to be the same by optimizing a movement parameter and a rotation parameter of the 3D object in a virtual environment; and estimating the first surface normal vector based on the aligned first image and second image (see ¶ [53-54], i.e., in order to compare the input image and the rendered image using differentiable renderer for the normal material map approximation, the images should be aligned to find the differences, wherein the rendered image using the same camera parameters as the input image, such as lighting conditions, rotational positions, etc.).
As per claim 7, Dave also impliedly teaches wherein the generating the simulation images comprises: corresponding to the basis images, synthesizing the simulation images obtained by simulating, in a differentiable method, images captured by using the virtual illumination patterns (¶ [34], digital maps synthesized by components of digital material generator. The differentiable method is addressed above).
Claim 16, which is similar in scope to claim 1 as addressed above, is thus rejected under the same rationale.
Claim 3 is are rejected under 35 U.S.C. 103 as being unpatentable over Dave et al. (US. Patent App. Pub. No. 2022/0335682) in view of Sato et al. (US. Patent App. Pub. No. 2010/0289878, “Sato”).
As per claim 3, Dave does not explicitly teach wherein the obtaining the first image comprises performing preprocessing of removing a specular reflection component from the first image.
However, in a very similar method of estimating normal vector (see ¶ [4-6], and Fig. 3), Sato teaches this feature, i.e., performing preprocessing of removing a specular reflection component from the first image (Fig. 75, ¶ [548], “Through the process up to this point, a viewpoint conversion has already been performed on the diffuse reflection image obtained by removing the actual specular reflection component from the subject”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method as taught by Sato into the method as taught by Dave as addressed above, the advantage of which is to generate (or estimate) high-precision normal information over a wide area (¶ [67]).
Claims 13, 15, 17, 19, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Dave et al. (US. Patent App. Pub. No. 2022/0335682) in view of Logothetis et al. (US. Patent App. Pub. No. 2022/0051471, “Logothetis”).
As per claim 13, as addressed in claim 1, Dave teaches a method of modeling a three-dimensional (3D) scene, the method comprising:
obtaining illumination patterns, corresponding to a 3D target object, by using a trained neural network (see claim 1);
capturing a target scene comprising the 3D target object by using the illumination patterns (¶ [2]).
Dave does not expressly teach modeling a 3D scene corresponding to the target scene by restoring, based on the illumination patterns, a surface normal of the 3D target object using a photometric stereo technique. However, as addressed in claim 1, Dave does teach using photometric stereo technique to estimate surface normal.
Logothetis teaches a similar method to that of Dave (see Abstract), wherein the method further comprises the above feature, i.e., modeling a 3D scene corresponding to the target scene by restoring, based on the illumination patterns, a surface normal of the 3D target object using a photometric stereo technique (¶ [28-32]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the method as taught by Logothetis and apply to the method as taught by Dave as addressed above, the advantage of which is for using a fast-to-obtain training data while still allowing the network to learn global illumination effects and real-world imperfections (¶ [33]).
As per claim 15, as addressed in claim 1, the combined Dave-Logothetis does teach wherein the neural network is trained by a dataset constructed by estimating a first surface normal vector of a 3D object from a first image, the first image being obtained by capturing the 3D object of which surface normal information is known.
Claim 17 is similar in scope to claim 13 as addressed above, with the exception of a communication interface that is also taught by Dave as shown in Fig. 7, ¶ [26]. Claim 17 is thus rejected under the same rationale.
As per claim 19, the combined Dave-Logothetis also teaches a display configured to display at least one of the illumination patterns or the modeled 3D scene (Dave, ¶ [74]).
As per claim 20, as addressed above, the combined Dave-Logothetis does also teach at least one of a lighting stage, a handheld flash camera, an imaging system comprising a display camera system, a wearable device comprising a smart glass, a head-mounted device (HMD) comprising at least one of an augmented reality (AR) device, a virtual reality (VR) device, or a mixed reality (MR) device; or a user terminal comprising at least one of a television, a smartphone, a personal computer (PC), a tablet, or a laptop (Dave, ¶ 84]).
Claims 14 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Dave et al. (US. Patent App. Pub. No. 2022/0335682) in view of Logothetis et al. (US. Patent App. Pub. No. 2022/0051471) further in view of Sato et al. (US. Patent App. Pub. No. 2010/0289878).
As per claim 14, the combined Dave-Logothetis does not teach wherein the modeling the 3D scene comprises:
obtaining a diffuse reflection image corresponding to one of the illumination patterns by separating a diffuse reflection component and a specular reflection component in each frame of the target scene; and estimating a surface normal vector of the 3D target object by applying the photometric stereo technique to the diffuse reflection image.
However, in a similar method, Sato teaches this feature, i.e., obtaining a diffuse reflection image corresponding to one of the illumination patterns by separating a diffuse reflection component and a specular reflection component in each frame of the target scene; and estimating a surface normal vector of the 3D target object by applying the photometric stereo technique to the diffuse reflection image (¶ [62], “…the viewpoint conversion section viewpoint-converts both a diffuse reflection image obtained by separating a diffuse reflection component and a specular reflection component from an image of the subject and a normal image obtained from the normal information generating section…”), using a photometric stereo method described in ¶ [6]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply the method taught by Sato to the combined Dave-Logothetis, the advantage of which is to generate (or estimate) high-precision normal information over a wide area (¶ [67]).
Claim 18, which is similar in scope to claim 14 as addressed above, is thus rejected under the same rationale.
Allowable Subject Matter
Claims 4-5, 8 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is an examiner' s statement of reasons for allowable subject matter:
The prior art taken singly or in combination does not teach or suggest, a method for determining illumination pattern, among other things, comprising:
…predicting a location of lattice points, displayed on a display device, by using a mirror; and removing the specular reflection component from the first image by adjusting a location of the display device to be in the location of the lattice points (claim 4); or
…synthesizing the simulation images by applying, for each of the virtual illumination patterns, a weighted sum in which a red, green, and blue (RGB) color intensity corresponding to at least a part of each of the basis images is multiplied by a corresponding virtual illumination pattern (claim 8).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Hau H. Nguyen whose telephone number is: 571-272-7787. The examiner can normally be reached on MON-FRI from 8:30-5:30.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard, can be reached on (571) 272-7773.
The fax number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/HAU H NGUYEN/Primary Examiner, Art Unit 2611