DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 5 and 16 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claims 5 and 16 recite “the triangles” in line 3. The claims previously recite “one or more triangles” and “the first and second sets of triangles” in lines 1-2. It is unclear which triangles are referenced in the line 3 recitation of “the triangles.” Therefore, the scope is unclear.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4-8, 12, 13, 15-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Feng et al. (“Learning Disentangled Avatars with Hybrid 3D Representations”; hereinafter “Feng”) in view of Duan et al. (“BakedAvatar: Baking Neural Fields for Real-Time Head Avatar Synthesis”; hereinafter “Duan”).
Regarding claim 1, Feng discloses A method for controlling a device (“3D human reconstruction and reenactment in numerous applications such as virtual and augmented reality, telepresence,” pg. 1, sec. 1, para. 1), the method comprising: receiving, by a processor, an input two-dimensional (2D) image (“Given a monocular video,” pg. 4, sec. 3, para. 1); receiving, by the processor, a hybrid three-dimensional (3D) model (“Hybrid 3D Representations,” title; “DELTA represents the body or face with an explicit mesh-based parametric 3D model and the clothing or hair with an implicit neural radiance field,” abstract) including a first set of triangles forming a triangular mesh (“we choose mesh as the representation for the face and body,” pg. 4, sec. 3.1, para. 1; “triangular meshes,” pg. 1, sec. 1, para. 2), … the vertices of both of the first … sets of triangles including rigging information (“linear blend skinning function (i.e., LBS),” pg. 6, sec. 3.2, para. 1; LBS is an algorithm used to deform a character mesh based on the movements of an underlying skeleton and is part of skeletal rigging); deforming, by the processor, the first … sets of triangles of the hybrid 3D model based on 3D animation parameters and the rigging information, to generate deformed triangles (“deform the head/body to the observation pose using the linear blend skinning function (i.e., LBS),” pg. 6, sec. 3.2, para. 1); rendering, by the processor, the deformed triangles based on rendering the first set of triangles using a texture mapping technique (“images are rendered from the textured mesh,” pg. 8, sec. 5.1, para. 3) …; and displaying, on a display of the device, an animated 3D object based on the rendered triangles and the input 2D image (“DELTA marries the strong controllability of the mesh-based face and the high-fidelity rendering of the NeRF-based hair,” pg. 3, col. 2, para. 1; “3D human reconstruction and reenactment in numerous applications such as virtual and augmented reality, telepresence, games, and movies,” pg. 1, sec. 1, para. 1).
While Feng discloses a hybrid 3D representation including a mesh-based face representation and NeRF-based hair representation, Feng does not disclose that the NeRF-based hair portion of the hybrid 3D representation is associated with an underlying mesh, meaning the NeRF-based hair portion of Feng does not include a second set of triangles with associated alpha map and neural feature maps, the vertices of the second set of triangles including rigging information, deforming the second set of triangles, and rendering the second set of triangles using deferred neural rendering based on the neural feature maps and the alpha map to generate rendered triangles.
In the same art of 3D avatar representations, Duan teaches a NeRF-based representation with underlying mesh that includes a second set of triangles (“deformable multi-layer meshes,” pg. 225:1, col. 2, para. 1) with associated alpha map and neural feature maps (“we represent this radiance field using multiple linear blendable bases, where each basis can be easily baked into a static texture for efficient querying. Concretely, assuming there are 𝑛𝑇 texture bases, the radiance field then predicts bases’ colors {c1, c2, ··· , c𝑛𝑇} and occupancies {𝛼1, 𝛼2, ··· , 𝛼𝑛𝑇}, as well as a shared 𝑑𝑝-dimensional position feature f𝑝 ∈ R𝑑𝑝 in canonical space,” pg. 225:5, col. 1, para. 2), the vertices of the second set of triangles including rigging information, deforming the second set of triangles (“compute deformations with per-vertex blendshape and linear blend skinning (LBS) operations on the extracted meshes,” pg. 225:4, sec. 3.1.2, para. 1; “LBS bone transformations,” pg. 225:6, sec. 3.4, para. 1), and rendering the second set of triangles using deferred neural rendering based on the neural feature maps and the alpha map to generate rendered triangles (“the neural fields associated with a head avatar can be baked into the deformable layered meshes and corresponding textures for real-time rendering,” pg. 225:4, sec. 3, para. 1; “We render all mesh layers from inner levels to outer levels respectively, where the light-weighted MLP is evaluated at all rendered pixels to blend radiance texture bases, and finally combine them with standard alpha composition to produce the final color,” pg. 225:6, sec. 3.3, para. 1).
Before the effective filing date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Duan to Feng, thereby replacing the NeRF of Feng with the NeRF of Duan, which is associated with an underlying, deformable mesh. The motivation would have been that it is “deployable in a standard polygon rasterization pipeline” (Duan, abstract), and that it “generates synthesis results of comparable quality to other state-of-the-art methods while significantly reducing the inference time required” (Duan, abstract).
Regarding claim 2, the combination of Feng and Duan renders obvious wherein the first set of triangles correspond to a 3D surface mesh of a 3D morphable model (“we choose mesh as the representation for the face and body,” Feng, pg. 4, sec. 3.1, para. 1; “SMPL-X is an expressive body model with detailed face shape and expressions,” Feng, pg. 5, col. 1, para. 2), and wherein the second set of triangles correspond to a neural radiance field (NeRF) (“model both hair and clothing with NeRF,” Feng, pg. 5, col. 1, para. 2; “the neural fields associated with a head avatar can be baked into the deformable layered meshes and corresponding textures for real-time rendering,” Duan, pg. 225:4, sec. 3, para. 1; see claim 1 for motivation to combine).
Regarding claim 4, the combination of Feng and Duan renders obvious receiving predetermined weights for a color prediction neural network; and rendering the second set of triangles based on the color prediction neural network and the predetermined weights (“The global MLP takes inputs of global expression and pose conditions and produces the weights of spatial MLP. The spatial MLP takes inputs of position texture, normal, and view direction, which vary spatially in screen space, and produces normalized blending weights. These weights are then combined with multiple radiance texture bases to generate color and alpha values of the current mesh layer,” Duang, Fig. 3 caption; see claim 1 for motivation to combine).
Regarding claim 5, the combination of Feng and Duan renders obvious wherein one or more triangles among the first and second sets of triangles have textures mapped to pre-computed textures, and the one or more triangles deform along with the triangles (“Our final representation for real-time rendering consists of four components: multi-layer meshes with vertex attributes (i.e., UVs, normals, FLAME weights), position and radiance texture atlases for each mesh, vertices and joint regressor of the FLAME template mesh,” Duan, pg. 255:6, sec. 3.4, para. 1; see claim 1 for motivation to combine).
Regarding claim 6, the combination of Feng and Duan renders obvious wherein the first set of triangles correspond to a head or neck region of a 3D head avatar, and wherein the second set of triangles correspond to a hair region of the 3D head avatar (“DELTA represents the body or face with an explicit mesh-based parametric 3D model and the clothing or hair with an implicit neural radiance field,” Feng, abstract; “the neural fields associated with a head avatar can be baked into the deformable layered meshes and corresponding textures for real-time rendering,” Duan; pg. 225:4, sec. 3, para. 1; see claim 1 for motivation to combine).
Regarding claim 7, Feng discloses A method for controlling a device (“3D human reconstruction and reenactment in numerous applications such as virtual and augmented reality, telepresence,” pg. 1, sec. 1, para. 1), the method comprising: receiving, by a processor, a video segment of a subject (“Given a monocular video,” pg. 4, sec. 3, para. 1); fitting, by the processor, a three-dimensional (3D) morphable model to the video segment of the subject to obtain animation parameters for frames of the video segment (“we conducted SMPL-X fitting for all frames during data processing,” pg. 8, sec. 5.2, para. 1); … training, by the processor, … an opacity field neural network, and a color prediction neural network within a corresponding canonical space to generate … a trained opacity field neural network, and a trained color prediction neural network (“we define the avatar exterior (hair or clothing) in the canonical 3D space as an implicit function … the implicit NeRF-based function 𝐹ℎ outputs an emitted RGB color 𝒄nerf and a volume density 𝜎,” pg. 5, col. 2, para. 2; “our model training,” pg. 8, sec. 4, para. 3); and generating, by the processor, a hybrid 3D model including a 3D surface mesh for the 3D morphable model and a neural radiance field (NeRF) (“Hybrid 3D Representations,” title; “DELTA represents the body or face with an explicit mesh-based parametric 3D model and the clothing or hair with an implicit neural radiance field,” abstract) … based on … the trained opacity field neural network, and the trained color prediction neural network (“the implicit NeRF-based function 𝐹ℎ outputs an emitted RGB color 𝒄nerf and a volume density 𝜎,” pg. 5, col. 2, para. 2).
While Feng discloses a hybrid 3D representation including a mesh-based face representation and NeRF-based hair representation, Feng does not disclose that the NeRF-based hair portion of the hybrid 3D representation is associated with an underlying mesh, so Feng does not disclose constructing, by the processor, a prism lattice structure over regions of the 3D morphable model designated for neural radiance field (NeRF) rendering, the prism lattice structure being configured to deform in tandem with the 3D morphable model; training a feature field neural network to generate a trained feature field neural network, the NeRF defined within the prism lattice structure, or the NeRF based on the trained feature field neural network.
In the same art of 3D avatar representations, Duan teaches a NeRF-based representation with underlying mesh that includes constructing, by the processor, a prism lattice structure over regions of the 3D morphable model designated for neural radiance field (NeRF) rendering, the prism lattice structure being configured to deform in tandem with the 3D morphable model (“the neural fields associated with a head avatar can be baked into the deformable layered meshes and corresponding textures for real-time rendering,” pg. 225:4, sec. 3, para. 1); training, by the processor, a feature field neural network, an opacity field neural network, and a color prediction neural network within a corresponding canonical space to generate a trained feature field neural network, a trained opacity field neural network, and a trained color prediction neural network (“the radiance field then predicts bases’ colors {c1, c2, ··· , c𝑛𝑇} and occupancies {𝛼1, 𝛼2, ··· , 𝛼𝑛𝑇}, as well as a shared 𝑑𝑝-dimensional position feature f𝑝 ∈ R𝑑𝑝 in canonical space,” pg. 225:5, col. 1, para. 2), the NeRF defined within the prism lattice structure (“the neural fields associated with a head avatar can be baked into the deformable layered meshes,” pg. 225:4, sec. 3, para. 1), and the NeRF based on the trained feature field neural network (“the radiance field then predicts … 𝑑𝑝-dimensional position feature f𝑝 ∈ R𝑑𝑝 in canonical space,” pg. 225:5, col. 1, para. 2).
Before the effective filing date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Duan to Feng, thereby replacing the NeRF of Feng with the NeRF of Duan, which is associated with an underlying, deformable mesh. The motivation would have been that it is “deployable in a standard polygon rasterization pipeline” (Duan, abstract), and that it “generates synthesis results of comparable quality to other state-of-the-art methods while significantly reducing the inference time required” (Duan, abstract).
Regarding claim 8, the combination of Feng and Duan renders obvious wherein the training (“model training,” Feng, pg. 8, sec. 5, para. 3) the feature field neural network, the opacity field neural network, and the color prediction neural network is based on: rendering images by casting rays, determining ray intersections with the 3D surface mesh (“the camera ray will stop when it intersects with the mesh in the 3D space,” Feng, pg. 6, sec. 3.3, para. 3) and the prism lattice (“the spatial MLP is evaluated at each ray-manifold intersection,” Duan, pg. 225:5, col. 1 para. 4; see claim 7 for motivation to combine), and sampling feature and opacity fields (“sample all texels for baking radiance bases and position features … evaluate the radiance field to obtain a smooth estimation of colors {c1, c2, ··· , c𝑛𝑇} and occupancies {𝛼1, 𝛼2, ··· , 𝛼𝑛𝑇} for 𝑛𝑇 radiance bases, as well as the position feature,” Duan, pg. 225:6, col. 1, para. 1); comparing rendered images to ground truth images; and minimizing a difference between the rendered images and the ground truth images (“During training, we randomly sample a number of rays, and the geometry, deformation, and appearance fields are jointly optimized in an end-to-end manner. This process is supervised by image-based losses,” Duan, pg. 255:5, sec. 3.1.4, para. 1; see claim 7 for motivation to combine).
Regarding claim 12, it is rejected using the same citations and rationales described in the rejection of claim 1, with the additional limitations of A device, comprising: a display configured to display an image; a memory configured to store animation information; and a controller (“The tested devices include a gaming laptop,” Duan, pg. 225:8, Table 3 caption; see claim 1 for motivation to combine).
Regarding claims 13 and 15-17, they are rejected using the same citations and rationales described in the rejections of claims 2 and 4-6, respectively.
Regarding claim 18, the combination of Feng and Duan renders obvious wherein the hybrid three-dimensional 3D model is based on constructing a prism lattice structure over regions of a 3D morphable model designated for neural radiance field (NeRF) rendering, the prism lattice structure being configured to deform in tandem with the 3D morphable model (“the neural fields associated with a head avatar can be baked into the deformable layered meshes and corresponding textures for real-time rendering,” Duan, pg. 225:4, sec. 3, para. 1; see claim 1 for motivation to combine).
Regarding claim 20, the combination of Feng and Duan renders obvious wherein the hybrid three-dimensional 3D model is generated based on outputs of a trained feature field neural network, a trained opacity field neural network, and a trained color prediction neural network (“the radiance field then predicts bases’ colors {c1, c2, ··· , c𝑛𝑇} and occupancies {𝛼1, 𝛼2, ··· , 𝛼𝑛𝑇}, as well as a shared 𝑑𝑝-dimensional position feature f𝑝 ∈ R𝑑𝑝 in canonical space,” Duan, pg. 225:5, col. 1, para. 2; “In the training stage, we first learn all fields as coordinate-based implicit MLPs,” Duan, pg. 225:10, sec. A.1, para. 1; see claim 1 for motivation to combine)
Claims 10, 11, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Feng and Duan, and further in view of Overbeck et al. (US 2021/0368157; hereinafter “Overbeck”)
Regarding claim 10, the combination of Feng and Duan renders obvious creating texture maps for remaining triangles of the prism lattice structure, including an alpha map and two feature maps (“we represent this radiance field using multiple linear blendable bases, where each basis can be easily baked into a static texture for efficient querying. Concretely, assuming there are 𝑛𝑇 texture bases, the radiance field then predicts bases’ colors {c1, c2, ··· , c𝑛𝑇} and occupancies {𝛼1, 𝛼2, ··· , 𝛼𝑛𝑇}, as well as a shared 𝑑𝑝 -dimensional position feature,” Duan, pg. 225:5, col. 1, para. 2); obtaining a rigged triangular mesh; and outputting the rigged triangular mesh and the texture maps (“Our final representation for real-time rendering consists of four components: multi-layer meshes with vertex attributes (i.e., UVs, normals, FLAME weights), position and radiance texture atlases for each mesh, vertices and joint regressor of the FLAME template mesh,” Duan, pg. 225:6, sec. 3.4, para. 1; see claim 1 for motivation to combine).
The combination of Feng and Duan does not disclose pruning triangles from the prism lattice structure.
In the same art of 3D mesh representation, Overbeck teaches pruning triangles from the prism lattice structure (“removing any triangle that does not intersect with at least one opaque texel,” para. 147).
Before the effective filing date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the triangle pruning of Overbeck to the combination of Feng and Duan. The motivation would have been to increase efficiency by reducing extraneous triangles.
Regarding claim 11, the combination of Feng, Duan, and Overbeck renders obvious transmitting an exported hybrid 3D model based on the rigged triangular mesh and the texture maps (“transmitting … mesh and a texture atlas,” Overbeck, para. 30; see claim 10 for motivation to combine; the triangular mesh and texture map transmission of Overbeck is applied to the hybrid 3D model based on the rigged triangular mesh and the texture maps of the combination of Feng and Duan).
Regarding claim 19, the combination of Feng and Duan does not disclose the prism lattice structure is a pruned prism lattice structure including triangles with associated opacity values that are greater than or equal to a predetermined opacity value.
In the same art of 3D mesh representation, Overbeck teaches the prism lattice structure is a pruned prism lattice structure including triangles with associated opacity values that are greater than or equal to a predetermined opacity value (“removing any triangle that does not intersect with at least one opaque texel,” para. 147).
Before the effective filing date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the triangle pruning of Overbeck to the combination of Feng and Duan. The motivation would have been to increase efficiency by reducing extraneous triangles.
Allowable Subject Matter
Claims 3, 9, and 14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 3, the known prior art does not disclose rendering the second set of triangles using the deferred neural rendering based on sampling the neural feature maps and the alpha map, discarding low-opacity pixels, and inputting sampled features and a camera direction to a color prediction neural network to generate a rendered image.
Regarding claim 9, the known prior art does not disclose refining the trained opacity field neural network to favor binary values and associating each ray with a single feature vector from an opaque triangle of the prism lattice structure.
Regarding claim 14, it contains allowable subject matter for the same reasons as claim 3.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ryan McCulley whose telephone number is (571)270-3754. The examiner can normally be reached Monday through Friday, 8:00am - 4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571) 272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN MCCULLEY/Primary Examiner, Art Unit 2611
Prosecution Insights