DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This action is in response to the amendment filed on 20th April, 2026. Claims 1, 11, and 18 have been amended. Claim 5 has been cancelled. Claims 1-4 and 6-20 remain rejected in the application.
Response to Arguments
Applicant's arguments with respect to Claims 1, 11, and 18 filed on 20th April, 2026, with respect to the rejection under 35 U.S.C. § 103, regarding that the prior art does not teach the limitation(s): "transforming a first point of a three-dimensional (3D) model representing an avatar to model an eye as a surface that is translucent, the model being configured to compute specular reflections of light associated with the surface, the computation being explicit" has/have been fully considered, but are moot because of new grounds for rejection. It has now been taught by the combination of Berard, Wood-16, and Wood-15.
Regarding arguments to Claims 2-4, 6-10, 12-17, and 19-20, they directly/indirectly depend on independent Claims 1, 11, and 18 respectively. Applicant does not argue anything other than independent Claims 1, 11, and 18. The limitations in those claims, in conjunction with combination, was previously established as explained.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 6, 8, 11, and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Bêrard et al. (US 20180012418 A1, previously cited), hereinafter referenced as Berard, in view of Wood et al. ("Learning an Appearance-Based Gaze Estimator from One Million Synthesized Images", previously cited), hereinafter referenced as Wood-16, and further in view of Wood et al. ("Rendering of Eyes for Eye-Shape Registration and Gaze Estimation"), hereinafter referenced as Wood-15.
Regarding Claim 1, Berard discloses a method (Berard, [0112]: teaches a method for reconstructing eyes) comprising:
transforming a first point of a three-dimensional (3D) model representing an avatar to model an eye as a surface that is translucent (Berard, [0045]: teaches a limbus opacity mask that defines the transparency transition <read on first point> from the sclera to the cornea, where "the position of the limbus can be extracted from the limbus opacity mask by mapping a percent opacity level to the mesh (e.g., a 50 percent opacity level or other suitable opacity level)"; [0046]: teaches an eyeball model <read on 3D eye model of avatar> including a rigid transformation and a uniform scale factor, which are interpreted to transform and scale surface points <read on transformed first point> on the eyeball model, to create a deformed eyeball model, which include semantic features that are used to establish correspondence, such as the white sclera, the transparent <read on translucent> cornea, and the limbus; Note: it should be noted that the cornea is a clear, dome-shaped outer surface of the eye),
[[the model being configured to compute specular reflections of light associated with the surface,]]
[[the computation being explicit;]]
warping a second point associated with the eye of the 3D model [[to predict a rotation and a translation associated with the second point]] (Berard, [0129]: teaches "fitting the eyeball model to the at least one eye can include identifying a limbus of the at least one eye in the one or more input images, and minimizing a distance between a limbus of the eyeball model and the identified limbus of the at least one eye"; [0130]: teaches "reconstructing the at least one eye using the parametric eye model with the determined parameters," where "the reconstructed eye can be combined with the remaining parts <read on warping second point> of a subject's reconstructed face depicted in the one or more input images");
[[generating an indicator of reflection based on the transformed first point and the warped second point using a first layer of the model;]]
[[generating a spherical indicator based on the transformed first point and the warped second point using a second layer of the model; and]]
[[generating an image including an image point that is generated based on the indicator of reflection and the spherical indicator.]]
However, Berard does not expressly disclose
the model being configured to compute specular reflections of light associated with the surface,
the computation being explicit;
warping a second point associated with the eye of the 3D model to predict a rotation and a translation associated with the second point;
generating an indicator of reflection based on the transformed first point and the warped second point using a first layer of the model;
generating a spherical indicator based on the transformed first point and the warped second point using a second layer of the model; and
generating an image including an image point that is generated based on the indicator of reflection and the spherical indicator.
Wood-16 discloses
the model being configured to compute specular reflections of light associated with the surface (Wood-16, [Section 3 Approximate Eyeball Model]: teaches varying eyeball shapes and textures, where the eyeball is wet, which reflects the environment <read on computed specular reflections> and surrounding eye area using textures of the physically based shader, which "models how light behaves in reality to achieve consistent effects under various lighting conditions"; [Section 3 Approximate Eyeball Model]: further teaches physically based refraction, where each pixel on the surface of the cornea is calculated for light refraction),
the computation being explicit (Wood-16, [Section 3 Approximate Eyeball Model]: teaches calculating for each pixel on the surface of the cornea using the following equation <read on explicit computation>; Note: it should be noted that an explicit computation is a non-iterative calculation, where one parameter determines the result);
PNG
media_image1.png
107
696
media_image1.png
Greyscale
warping a second point associated with the eye of the 3D model to predict a rotation and a translation associated with the second point (Wood-16, [Section 4 Generative Eye Region Model]: teaches procedural methods for eyelid movement associated with gaze estimation <read on predicting rotation and translation>, where general movement is described as a rotation with different parts of the lid having different rotational axes; FIG. 12 teaches a rendered eye region <read on second point>);
PNG
media_image2.png
324
520
media_image2.png
Greyscale
[[generating an indicator of reflection based on the transformed first point and the warped second point using a first layer of the model;]]
[[generating a spherical indicator based on the transformed first point and the warped second point using a second layer of the model; and]]
[[generating an image including an image point that is generated based on the indicator of reflection and the spherical indicator.]]
Wood-16 is analogous art with respect to Berard because they are from the same field of endeavor, namely generating realistic 3D deformable eye models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a generative eye region that includes eye gaze parameters, and camera parameters as taught by Wood-16 into the teaching of Berard. The suggestion for doing so would allow for real-time modifiable editing and adjustments of the 3D eye model, thereby improving the user experience. Therefore, it would have been obvious to combine Wood-16 with Berard.
However, the combination of Berard and Wood-16 does not expressly disclose
generating an indicator of reflection based on the transformed first point and the warped second point using a first layer of the model;
generating a spherical indicator based on the transformed first point and the warped second point using a second layer of the model; and
generating an image including an image point that is generated based on the indicator of reflection and the spherical indicator.
Wood-15 discloses
generating an indicator of reflection based on the transformed first point and the warped second point using a first layer of the model (Wood-15, [Section 3 Dynamic Eye-Region Model]: teaches a simplified eyeball model including two separate mesh parts of the eye, which are joined and smoothed together <read on transformed first point>, and a texture of the eye that consists of composited images in three separate layers <read on first layer of model>; [Section 3 Dynamic Eye-Region Model]: teaches modeling eyelid motion and eyelashes, where compressed or stretched tissue <read on warped second point> around the eye is modeled such that "skin details like wrinkles and folds are either attenuated or exaggerated" as shown in FIG. 5; [Section 4 Training Data Synthesis]: teaches creating realistic illumination by capturing omni-directional light information and storing it in a texture, where it is then projected onto a sphere around the object, such as an eyeball model, such that during rendering, when a ray hits that texture, it takes that texture's pixel value as light intensity <read on reflection indicator> (i.e., how light bounces or reflects off materials));
PNG
media_image3.png
264
828
media_image3.png
Greyscale
generating a spherical indicator based on the transformed first point and the warped second point using a second layer of the model (Wood-15, [Section 3 Dynamic Eye-Region Model]: teaches a simplified eyeball model including two separate mesh parts of the eye, which are joined and smoothed together <read on transformed first point>, and a texture of the eye that consists of composited images in three separate layers <read on second layer of model>; [Section 3 Dynamic Eye-Region Model]: teaches modeling eyelid motion and eyelashes, where compressed or stretched tissue <read on warped second point> around the eye is modeled such that "skin details like wrinkles and folds are either attenuated or exaggerated" as shown in FIG. 5; [Section 4 Training Data Synthesis]: teaches posing the eyeball model, where spherical coordinates <read on spherical indicator> are used to determine gaze vectors, pitch angles, and yaw angles and define anatomical constraints (i.e., valid eyeball rotations)); and
generating an image including an image point that is generated based on the indicator of reflection and the spherical indicator (Wood-15, [Section 4 Training Data Synthesis]: teaches creating realistic illumination by capturing omni-directional light information and storing it in a texture, where it is then projected onto a sphere <read on image point> around the object, such as an eyeball model, such that during rendering, when a ray hits that texture, it takes that texture's pixel value as light intensity <read on reflection indicator> (i.e., how light bounces or reflects off materials); [Section 4 Training Data Synthesis]: teaches rendering images of eyes using a path-tracing engine, where paths of many light rays per pixel are traced; [Section 4 Training Data Synthesis]: teaches posing the eyeball model, where spherical coordinates <read on spherical indicator> are used to determine gaze vectors, pitch angles, and yaw angles and define anatomical constraints (i.e., valid eyeball rotations)).
Wood-15 is analogous art with respect to Berard, in view of Wood-16 because they are from the same field of endeavor, namely modelling realistic 3D eyes and the surrounding skin region. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a dynamic eye-region model that includes texture layers (e.g., sclera, iris, veins) to generate realistic 3D eyes based on eye-shape registrations and gaze estimations as taught by Wood-15 into the teaching of Berard, in view of Wood-16. The suggestion for doing so would allow the system to utilize ray/path tracing to calculate light reflectivity and refractivity, as well as diffuse surface properties of the surrounding skin area, thereby yielding predictable results. Therefore, it would have been obvious to combine Wood-15 with Berard, in view of Wood-16.
Regarding Claim 11, Berard discloses a method (Berard, [0112]: teaches a method for reconstructing eyes) comprising:
selecting a first point from a 3D model representing an avatar (Berard, [0129]: teaches "identifying a limbus of the at least one eye in the one or more input images, and minimizing a distance between a limbus of the eyeball model and the identified limbus <read on first point> of the at least one eye"),
the first point being associated with an eye (Berard, [0129]: teaches "identifying a limbus of the at least one eye in the one or more input images, and minimizing a distance between a limbus of the eyeball model and the identified limbus <read on first point> of the at least one eye");
selecting a second point from the 3D model (Berard, [0126]: teaches a 3D face scan "can include a portion of the face surrounding both eyes <read on second point>, a portion of the face around one eye, the entire face, or any other suitable portion of the face"),
the second point being associated with a periocular region associated with the eye (Berard, [0126]: teaches a 3D face scan "can include a portion of the face surrounding both eyes <read on second point associated with periocular region>, a portion of the face around one eye, the entire face, or any other suitable portion of the face");
[[generating indicator of reflection based on the first point and the second point using a first layer of a model configured to]]
[[compute specular reflections of light associated with the surface,]]
[[the computation being explicit;]]
[[generating a spherical indicator based on the first point and the second point using a second layer of the model,]]
[[the spherical indicator identifying a rotation and a translation; and]]
[[generating an image including an image point that is generated based on the indicator of reflection, and the spherical indicator.]]
However, Berard does not expressly disclose
generating indicator of reflection based on the first point and the second point using a first layer of a model configured to
compute specular reflections of light associated with the surface,
the computation being explicit;
generating a spherical indicator based on the first point and the second point using a second layer of the model,
the spherical indicator identifying a rotation and a translation; and
generating an image including an image point that is generated based on the indicator of reflection, and the spherical indicator.
Wood-16 discloses
[[generating indicator of reflection based on the first point and the second point using a first layer of a model configured to]]
compute specular reflections of light associated with the surface (Wood-16, [Section 3 Approximate Eyeball Model]: teaches varying eyeball shapes and textures, where the eyeball is wet, which reflects the environment <read on computed specular reflections> and surrounding eye area using textures of the physically based shader, which "models how light behaves in reality to achieve consistent effects under various lighting conditions"; [Section 3 Approximate Eyeball Model]: further teaches physically based refraction, where each pixel on the surface of the cornea is calculated for light refraction),
the computation being explicit (Wood-16, [Section 3 Approximate Eyeball Model]: teaches calculating for each pixel on the surface of the cornea using the following equation <read on explicit computation>; Note: it should be noted that an explicit computation is a non-iterative calculation, where one parameter determines the result);
PNG
media_image1.png
107
696
media_image1.png
Greyscale
[[generating a spherical indicator based on the first point and the second point using a second layer of the model,]]
the spherical indicator identifying a rotation and a translation (Wood-16, [Section 4 Generative Eye Region Model]: teaches procedural methods for eyelid movement associated with gaze estimation <read on identifying rotation and translation>, where general movement is described as a rotation with different parts of the lid having different rotational axes; FIG. 12 teaches including eye gaze parameters <read on spherical indicator>); and
[[generating an image including an image point that is generated based on the indicator of reflection, and the spherical indicator.]]
Wood-16 is analogous art with respect to Berard because they are from the same field of endeavor, namely generating realistic 3D deformable eye models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a generative eye region that includes eye gaze parameters, and camera parameters as taught by Wood-16 into the teaching of Berard. The suggestion for doing so would allow for real-time modifiable editing and adjustments of the 3D eye model, thereby improving the user experience. Therefore, it would have been obvious to combine Wood-16 with Berard.
However, the combination of Berard and Wood-16 does not expressly disclose
generating indicator of reflection based on the first point and the second point using a first layer of a model configured to…
generating a spherical indicator based on the first point and the second point using a second layer of the model; and
generating an image including an image point that is generated based on the indicator of reflection, and the spherical indicator.
Wood-15 discloses
generating indicator of reflection based on the first point and the second point using a first layer of a model configured to (Wood-15, [Section 3 Dynamic Eye-Region Model]: teaches a simplified eyeball model including two separate mesh parts of the eye, which are joined and smoothed together <read on first point>, and a texture of the eye that consists of composited images in three separate layers <read on first layer of model>; [Section 3 Dynamic Eye-Region Model]: teaches modeling eyelid motion and eyelashes, where compressed or stretched tissue <read on second point> around the eye is modeled such that "skin details like wrinkles and folds are either attenuated or exaggerated" as shown in FIG. 5; [Section 4 Training Data Synthesis]: teaches creating realistic illumination by capturing omni-directional light information and storing it in a texture, where it is then projected onto a sphere around the object, such as an eyeball model, such that during rendering, when a ray hits that texture, it takes that texture's pixel value as light intensity <read on reflection indicator> (i.e., how light bounces or reflects off materials))…
generating a spherical indicator based on the first point and the second point using a second layer of the model (Wood-15, [Section 3 Dynamic Eye-Region Model]: teaches a simplified eyeball model including two separate mesh parts of the eye, which are joined and smoothed together <read on first point>, and a texture of the eye that consists of composited images in three separate layers <read on second layer of model>; [Section 3 Dynamic Eye-Region Model]: teaches modeling eyelid motion and eyelashes, where compressed or stretched tissue <read on second point> around the eye is modeled such that "skin details like wrinkles and folds are either attenuated or exaggerated" as shown in FIG. 5; [Section 4 Training Data Synthesis]: teaches posing the eyeball model, where spherical coordinates <read on spherical indicator> are used to determine gaze vectors, pitch angles, and yaw angles and define anatomical constraints (i.e., valid eyeball rotations)); and
generating an image including an image point that is generated based on the indicator of reflection, and the spherical indicator (Wood-15, [Section 4 Training Data Synthesis]: teaches creating realistic illumination by capturing omni-directional light information and storing it in a texture, where it is then projected onto a sphere <read on image point> around the object, such as an eyeball model, such that during rendering, when a ray hits that texture, it takes that texture's pixel value as light intensity <read on reflection indicator> (i.e., how light bounces or reflects off materials); [Section 4 Training Data Synthesis]: teaches rendering images of eyes using a path-tracing engine, where paths of many light rays per pixel are traced; [Section 4 Training Data Synthesis]: teaches posing the eyeball model, where spherical coordinates <read on spherical indicator> are used to determine gaze vectors, pitch angles, and yaw angles and define anatomical constraints (i.e., valid eyeball rotations)).
Wood-15 is analogous art with respect to Berard, in view of Wood-16 because they are from the same field of endeavor, namely modelling realistic 3D eyes and the surrounding skin region. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a dynamic eye-region model that includes texture layers (e.g., sclera, iris, veins) to generate realistic 3D eyes based on eye-shape registrations and gaze estimations as taught by Wood-15 into the teaching of Berard, in view of Wood-16. The suggestion for doing so would allow the system to utilize ray/path tracing to calculate light reflectivity and refractivity, as well as diffuse surface properties of the surrounding skin area, thereby yielding predictable results. Therefore, it would have been obvious to combine Wood-15 with Berard, in view of Wood-16.
Regarding Claim 18, it recites the limitations that are similar in scope to Claim 11, but in a non-transitory computer-readable storage medium. As shown in the rejection, the combination of Berard, Wood-16, and Wood-15 discloses the limitations of Claim 11. Additionally, Berard discloses a non-transitory computer-readable storage medium (Berard, [0116]: teaches a non-transitory computer-readable storage medium) comprising
instructions stored thereon that, when executed by at least one processor, are configured to cause a computing system to (Berard, [0116]: teaches the non-transitory computer-readable storage medium including stored code, where one or more computer systems can execute said code as executable instructions by one or more processors):…
Thus, Claim 18 is met by Berard according to the mapping presented in the rejection of Claim 1, given the method corresponds to a non-transitory computer-readable storage medium.
Regarding Claim 6, the combination of Berard, Wood-16, and Wood-15 discloses the method of Claim 1. The combination of Berard and Wood-15 does not expressly disclose the limitations of Claim 6; however, Wood-16 discloses wherein the warping of the second point includes
generating a deformable volumetric reconstruction for a periocular region associated with the eye (Wood-16, FIG. 5 teaches a generated deformable model of the periocular region; Note: it should be noted that the periocular region is the surrounding area of the eye, which can include eyelids, eyebrows, eye bags, etc.).
PNG
media_image4.png
344
527
media_image4.png
Greyscale
Wood-16 is analogous art with respect to Berard, in view of Wood-15 because they are from the same field of endeavor, namely generating realistic 3D deformable eye models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a generative eye region that includes eye gaze parameters, and camera parameters as taught by Wood-16 into the teaching of Berard, in view of Wood-15. The suggestion for doing so would allow for real-time modifiable editing and adjustments of the 3D eye model, thereby improving the user experience. Therefore, it would have been obvious to combine Wood-16 with Berard, in view of Wood-15.
Regarding Claims 8 and 17, the combination of Berard, Wood-16, and Wood-15 discloses the methods of Claims 1 and 11 respectively. Additionally, Berard further discloses wherein the generating of the image point includes
changing a view direction of the eye (Berard, [0106]: teaches multi-pose fitting for the eyeball, where the pupil position affects the view direction as shown in FIG. 8A).
PNG
media_image5.png
300
343
media_image5.png
Greyscale
Claims 2-4, 7, 9-10, 12-16, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Bêrard et al. (US 20180012418 A1, previously cited), hereinafter referenced as Berard, in view of Wood et al. ("Learning an Appearance-Based Gaze Estimator from One Million Synthesized Images", previously cited), hereinafter referenced as Wood-16, and further in view of Wood et al. ("Rendering of Eyes for Eye-Shape Registration and Gaze Estimation"), hereinafter referenced as Wood-15 as applied to Claims 1, 11, and 18 above respectively, and further in view of Zhao et al. (US 20230027890 A1, previously cited), hereinafter referenced as Zhao.
Regarding Claims 2 and 12, the combination of Berard, Wood-16, and Wood-15 discloses the methods of Claims 1 and 11 respectively. The combination of Berard, Wood-16, and Wood-15 does not expressly disclose the limitations of Claims 2 and 12; however, Zhao discloses
storing a plurality of image points using the generated image point (Zhao, [0098]: teaches encoding that are "sent through the first and second neural networks with a plurality of RGB image representations," where "the output length of the second neural network (that is, volumetric light map network) is 12 for each queried 3D location <read on stored image points> while that of the first neural network (that is, material network) is 4 including one channel for skin roughness and 3 channels for skin sub-surface scattering"); and
generating an image representing the avatar based on a plurality of image points (Zhao, [0098]: teaches encoding that are "sent through the first and second neural networks with a plurality of RGB image representations," where "the output length of the second neural network (that is, volumetric light map network) is 12 for each queried 3D location while that of the first neural network (that is, material network) is 4 including one channel for skin roughness and 3 channels for skin sub-surface scattering"; [0098]: further teaches "transport aggregation is performed to obtain a weighted value for each ray based on a density distribution among the sampled points along the ray, where the values of each component on rays are aggregated with introduced values from preprocessed albedo and normal maps, and then visualized as an image with pixels representing their intensities following a rendering equation (1)," where "the rendered RGB values <read on image points> are constrained with ground truth pixel colors").
Zhao is analogous art with respect to the combination of Berard, Wood-16, and Wood-15 because they are from the same field of endeavor, namely utilizing neural networks for lighting 3D face models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a volumetric and material neural networks as taught by Zhao into the combined teaching of Berard, Wood-16, and Wood-15. The suggestion for doing so would allow the neural networks to better predict lighting effects on the face model, including the eyes, thereby resulting in a more natural-looking face model. Therefore, it would have been obvious to combine Zhao with the combination of Berard, Wood-16, and Wood-15.
Regarding Claims 3 and 13, the combination of Berard, Wood-16, Wood-15, and Zhao discloses the methods of Claims 2 and 12 respectively. The combination of Berard, Wood-16, and Wood-15 does not expressly disclose the limitations of Claims 3 and 13; however, Zhao discloses wherein the generating of the image representing the avatar includes
rendering the image representing the avatar using raytracing to compute reflection rays and refraction rays (Zhao, [0099]: teaches "generating a volumetric neural radiance field based on at least the volumetric skin reflectance field (obtained at 306) and the volumetric light map (obtained at 310)," where "based on one or more input rendering assets, including the three-dimensional geometry, the albedo map, and the high-frequency map, a density field is generated based on an absolute depth of a sample to ray-mesh intersections <read on computing reflection and refraction rays>, which is a distance from any sample to an intersecting point on the three-dimensional geometric mesh along a ray direction").
Zhao is analogous art with respect to the combination of Berard, Wood-16, and Wood-15 because they are from the same field of endeavor, namely utilizing neural networks for lighting 3D face models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a volumetric and material neural networks as taught by Zhao into the combined teaching of Berard, Wood-16, and Wood-15. The suggestion for doing so would allow the neural networks to better understand lighting behaviors on the face model, including the eyes, thereby resulting in a more natural-looking face model. Therefore, it would have been obvious to combine Zhao with the combination of Berard, Wood-16, and Wood-15.
Regarding Claim 19, the combination of Berard, Wood-16, and Wood-15 discloses the non-transitory computer-readable storage medium of Claim 18. The combination of Berard, Wood-16, and Wood-15 does not expressly disclose the limitations of Claim 19; however, Zhao discloses wherein the instructions are further configured to cause the computing system to
render a plurality of image points representing the avatar using raytracing to compute reflection rays and refraction rays to generate an image representing the avatar (Zhao, [0099]: teaches "generating a volumetric neural radiance field based on at least the volumetric skin reflectance field (obtained at 306) and the volumetric light map (obtained at 310)," where "based on one or more input rendering assets, including the three-dimensional geometry, the albedo map, and the high-frequency map, a density field is generated based on an absolute depth of a sample to ray-mesh intersections <read on computing reflection and refraction rays>, which is a distance from any sample to an intersecting point on the three-dimensional geometric mesh along a ray direction").
Zhao is analogous art with respect to the combination of Berard, Wood-16, and Wood-15 because they are from the same field of endeavor, namely utilizing neural networks for lighting 3D face models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a volumetric and material neural networks as taught by Zhao into the combined teaching of Berard, Wood-16, and Wood-15. The suggestion for doing so would allow the neural networks to better understand lighting behaviors on the face model, including the eyes, thereby resulting in a more natural-looking face model. Therefore, it would have been obvious to combine Zhao with the combination of Berard, Wood-16, and Wood-15.
Regarding Claim 4, the combination of Berard, Wood-16, and Wood-15 discloses the method of Claim 1. The combination of Berard, Wood-16, and Wood-15 does not expressly disclose the limitations of Claim 4; however, Zhao discloses wherein the spherical indicator include
specular SH coefficients (Zhao, [0132]: teaches a "direct specular component <read on specular SH coefficients> models the light that comes from the light source and is reflected directly on the surface without experiencing energy absorption," where "given all possible incident ray directions
ω
i
at a single point
x
, the direct specular radiance
L
d
s
of outgoing direction
ω
o
is calculated") and
diffuse SH coefficients (Zhao, [0135]: teaches a direct diffuse component <read on diffuse SH coefficients>, which "takes into account the energy absorption between the incident light and particles of the material" and "to measure diffuse reflection of radiance out of the total incident radiance, the direct diffuse radiance Lad in outgoing direction
ω
o
is calculated").
Zhao is analogous art with respect to the combination of Berard, Wood-16, and Wood-15 because they are from the same field of endeavor, namely utilizing neural networks for lighting 3D face models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a volumetric and material neural networks as taught by Zhao into the combined teaching of Berard, Wood-16, and Wood-15. The suggestion for doing so would allow the neural networks to predict how light affects certain surface points of the face model, such as the eyes, resulting in a more accurate and realistic output. Therefore, it would have been obvious to combine Zhao with the combination of Berard, Wood-16, and Wood-15.
Regarding Claims 7 and 16, the combination of Berard, Wood-16, and Wood-15 discloses the methods of Claims 1 and 11 respectively. The combination of Berard, Wood-16, and Wood-15 does not expressly disclose the limitations of Claims 7 and 16; however, Zhao discloses
disentangling a reflectance associated with environmental lighting (Zhao, [0128]: teaches "lighting <read on reflectance associated with environmental lighting> can be disentangled from texture by adjusting weight in density and visibility field while keeping the lightmap fully differentiable"); and
relighting the image point based on an environmental map (Zhao, [0072]: teaches specular reflectance reconstruction <read on relighting image point>, where "with reconstructed assets, camera calibration, and all-white baked environmental texture, ray-based rendering may be performed under all white, matching the environment <read on environmental map> of capture and training on combined color images").
Zhao is analogous art with respect to the combination of Berard, Wood-16, and Wood-15 because they are from the same field of endeavor, namely utilizing neural networks for lighting 3D face models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a volumetric and material neural networks as taught by Zhao into the combined teaching of Berard, Wood-16, and Wood-15. The suggestion for doing so would allow the neural networks to better understand lighting behaviors on the face model, including the eyes, thereby resulting in a more natural-looking face model. Therefore, it would have been obvious to combine Zhao with the combination of Berard, Wood-16, and Wood-15.
Regarding Claims 9, 14, and 20, the combination of Berard, Wood-16, and Wood-15 discloses the methods and the non-transitory computer-readable storage medium of Claims 1, 11, and 18 respectively. The combination of Berard and Wood-16 does not expressly disclose the limitations of Claims 9, 14, and 20; however, Wood-15 discloses wherein the model includes
[[a trained Neural Radiance Fields (NeRF) with Spherical Harmonics Lighting (SHL) model trained to]]
generate the indicator of reflection as an albedo (Wood-15, [Section 3 Dynamic Eye-Region Model]: teaches a simplified eyeball model including two separate mesh parts of the eye, which are joined and smoothed together <read on transformed first point>, and a texture of the eye that consists of composited images in three separate layers <read on first layer of model>; [Section 4 Training Data Synthesis]: teaches creating realistic illumination by capturing omni-directional light information and storing it in a texture, where it is then projected onto a sphere around the object, such as an eyeball model, such that during rendering, when a ray hits that texture, it takes that texture's pixel value <read on albedo> as light intensity <read on reflection indicator> (i.e., how light bounces or reflects off materials)) and trained to
generate the spherical indicator as spherical harmonic coefficients using the transformed first point and the warped second point as inputs to the trained model (Wood-15, [Section 3 Dynamic Eye-Region Model]: teaches a simplified eyeball model including two separate mesh parts of the eye, which are joined and smoothed together <read on transformed first point>, and a texture of the eye that consists of composited images in three separate layers; [Section 3 Dynamic Eye-Region Model]: teaches modeling eyelid motion and eyelashes, where compressed or stretched tissue <read on warped second point> around the eye is modeled such that "skin details like wrinkles and folds are either attenuated or exaggerated" as shown in FIG. 5; [Section 4 Training Data Synthesis]: teaches posing the eyeball model to generate training input data, where spherical coordinates <read on spherical indicator> are used to determine gaze vectors, pitch angles, and yaw angles and define anatomical constraints <read on spherical harmonic coefficients> (i.e., valid eyeball rotations and shapes); Note: it should be noted that spherical harmonic coefficients are interpreted to be coefficients that define the shape of a sphere).
Wood-15 is analogous art with respect to Berard, in view of Wood-16 because they are from the same field of endeavor, namely modelling realistic 3D eyes and the surrounding skin region. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a dynamic eye-region model that includes texture layers (e.g., sclera, iris, veins) to generate realistic 3D eyes based on eye-shape registrations and gaze estimations as taught by Wood-15 into the teaching of Berard, in view of Wood-16. The suggestion for doing so would allow the system to utilize ray/path tracing to calculate light reflectivity and refractivity, as well as diffuse surface properties of the surrounding skin area, thereby yielding predictable results. Therefore, it would have been obvious to combine Wood-15 with Berard, in view of Wood-16.
However, the combination of Berard, Wood-16, and Wood-15 does not expressly disclose
a trained Neural Radiance Fields (NeRF) with Spherical Harmonics Lighting (SHL) model trained to…
Zhao discloses
a trained Neural Radiance Fields (NeRF) with Spherical Harmonics Lighting (SHL) model trained to (Zhao, [0064]: teaches memory 120 storing a volumetric light map generation module, where "the volumetric light map generation module 126 may include a second neural network (also referred to herein as volumetric light map network) for generating a set of representations including spherical harmonic representations <read on SHL model> that model indirect light behavior based on a desired lighting condition"; [0065]: teaches memory 120 storing "a rendering module 128 that includes instructions for generating a volumetric neural radiance field based on one or more of the skin reflectance representations from the first neural network, the volumetric light map from the second neural network <read on trained NeRF>, and the one or more rendering assets")…
Zhao is analogous art with respect to the combination of Berard, Wood-16, and Wood-15 because they are from the same field of endeavor, namely utilizing neural networks for lighting 3D face models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a volumetric and material neural networks as taught by Zhao into the combined teaching of Berard, Wood-16, and Wood-15. The suggestion for doing so would allow the neural networks to better understand lighting behaviors on the face model, including the eyes, thereby resulting in a more natural-looking face model. Therefore, it would have been obvious to combine Zhao with the combination of Berard, Wood-16, and Wood-15.
Regarding Claims 10 and 15, the combination of Berard, Wood-16, Wood-15, and Zhao discloses the methods of Claims 9 and 14 respectively. The combination of Berard, Wood-16, and Wood-15 does not expressly disclose the limitations of Claims 10 and 15; however, Zhao discloses wherein the NeRF with SHL model is trained with a subject at least one of:
following a mobile camera with a gaze of the subject while keeping their head static and forward facing (Zhao, [0059]: teaches the computing system 100 being a mobile phone <read on mobile camera>; [0079]: teaches one or more rendering assets 202 that correspond to one or more digital images including a subject, where one of the views is a front view <read on head being static and forward facing>; [0127]: teaches the visibility at each point is defined to model whether the point is visible from view direction
ω
o
<read on gaze>);
focusing on a first static camera (Zhao, FIG. 21 teaches camera locations 2002 for the left, front, and right cameras, where the cameras are stationary <read on focus on first static camera>) and
PNG
media_image6.png
363
577
media_image6.png
Greyscale
changing the gaze of the subject to a second static camera (Zhao, FIG. 21 teaches camera locations 2002, where the left, front and right cameras have different views of the subject <read on change gaze of subject to second static camera>); and
focusing on the first static camera (Zhao, FIG. 21 teaches camera locations 2002 for the left, front, and right cameras, where the cameras are stationary <read on focus on first static camera>) and
rotating a head of the subject in a pattern with eyes of the subject static (Zhao, [0068]: teaches "the platform may rotate the object <read on head of subject> with various rotation angle steps to allow the object and/or the subject to be seen by the camera(s) 174 at desired angles"; [0067]: teaches "the one or more light source(s) 172 may be configured to illuminate the subject and/or the object with a controllable field of illumination using threshold-order (e.g., up to 5th order) Spherical Harmonics (SH) or Fourier Series (FS) illumination patterns").
Zhao is analogous art with respect to the combination of Berard, Wood-16, and Wood-15 because they are from the same field of endeavor, namely utilizing neural networks for lighting 3D face models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to capture the head of the user with stationary cameras as taught by Zhao into the combined teaching of Berard, Wood-16, and Wood-15. The suggestion for doing so would result in a more accurate representation of the user's head, thereby preparing the model for modification. Therefore, it would have been obvious to combine Zhao with the combination of Berard, Wood-16, and Wood-15.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Bêrard et al. (US 20160143524 A1) discloses reconstructing one or more surfaces of an object including one or more opaque surfaces behind one or more refractive surfaces; and
Rhee et al. (US 20120188228 A1) discloses realistically reproducing an eyeball that verifies and analyzes a material property and deformation property.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KARL TRUONG whose telephone number is (703)756-5915. The examiner can normally be reached 10:30 AM - 7:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.D.T./Examiner, Art Unit 2614
/YuJang Tswei/Primary Examiner, Art Unit 2614
Prosecution Insights