DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 27th October, 2025 has been entered.
Response to Amendment
This action is in response to the amendment filed on 27th October, 2025. Claims 1, 11, and 18 have been amended. Claims 1-20 remain rejected in the application.
Response to Arguments
Applicant's arguments with respect to Claims 1, 11, and 18 filed on 27th October, 2025, with respect to the rejection under 35 U.S.C. § 103, regarding that the prior art does not teach the limitation(s): "transforming the first point to model the identified eye as a surface that is translucent" and "warping the second point to predict a rotation and a translation associated with the second point" have been fully considered, but are moot because of new grounds for rejection. It has now been taught by the combination of Knorr, Berard, and Wood.
Regarding arguments to Claims 2-10, 12-17, and 19-20, they directly/indirectly depend on independent Claims 1, 11, and 18 respectively. Applicant does not argue anything other than independent Claims 1, 11, and 18. The limitations in those claims, in conjunction with combination, was previously established as explained.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5-6, 8, 11, and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Knorr et al. (US 20150279113 A1, previously cited), hereinafter referenced as Knorr, in view of Bêrard et al. (US 20180012418 A1, previously cited), hereinafter referenced as Berard, and further in view of Wood et al. ("Leaning an Appearance-Based Gaze Estimator from One Million Synthesized Images"), hereinafter referenced as Wood.
Regarding Claim 1, Knorr discloses a method (Knorr, [0199]: teaches a light estimation method) comprising:
[[identifying an eye in a three-dimensional (3D) model representing an avatar;]]
[[selecting a first point from the 3D model based on the identified eye;]]
[[selecting a second point from the 3D model based on the identified eye;]]
[[transforming the first point to model the identified eye as a surface that is translucent;]]
[[warping the second point to predict a rotation and a translation associated with the second point;]]
generating an indicator of reflection [[based on the transformed first point and the warped second point]] using a first layer of a model (Knorr, [0128]: teaches a human face specific characteristic being a reflection in the eyes <read on generating indicator of reflection>, which can be used to determine the light incident on the face; [0121]: teaches automatically determining face regions using machine learning <read on first layer of model>; Note: it should be noted that although not expressly stated, all neural networks have layers, such as an input layer, hidden layers, and an output layer);
generating a spherical indicator [[based on the transformed first point and the warped second point]] using a second layer of the model (Knorr, [0129]: teaches a human face specific characteristic being "some pre-modeled radiance transfer for a particular face region like basis images of the face or face region under basis illumination (Spherical Harmonics basis, or point lights) <read on generating spherical indicator> which can be used to find the best fitting linear combination resulting in an estimation for the incident lighting"; [0121]: teaches automatically determining face regions using machine learning <read on second layer of model>);
generating an image point based on the indicator of reflection and the spherical indicator (Knorr, [0190]: teaches a radiance transfer function specifying "how a particular surface point of a face <read on generating image point> responds to incident light from the environment from a particular direction in terms of radiance reflected <read on indicator of reflection> at this point towards the camera," where "the radiance transfer function depends mainly on the surface orientation at the location <read on spherical indicator> and occlusions of parts of the distant environment at the surface location by local geometry, but also includes material properties and interreflections of light between local geometry"); and
generating an image including the image point (Knorr, [0194]: teaches calculating the radiance transfer function for a particular surface point in the image, which is displayed to the user <read on generating image including image point>).
However, Knorr does not expressly disclose
identifying an eye in a three-dimensional (3D) model representing an avatar;
selecting a first point from the 3D model based on the identified eye;
selecting a second point from the 3D model based on the identified eye;
transforming the first point to model the identified eye as a surface that is translucent;
warping the second point to predict a rotation and a translation associated with the second point;
generating an indicator of reflection based on the transformed first point and the warped second point using a first layer of a model; and
generating a spherical indicator based on the transformed first point and the warped second point using a second layer of the model.
Berard discloses
identifying an eye in a three-dimensional (3D) model representing an avatar (Berard, [0129]: teaches identifying a sclera and limbus of at least one eye in the one or more input images (e.g., the three-dimensional scan));
selecting a first point from the 3D model based on the identified eye (Berard, [0129]: teaches "identifying a limbus of the at least one eye in the one or more input images, and minimizing a distance between a limbus of the eyeball model and the identified limbus <read on first point> of the at least one eye");
selecting a second point from the 3D model based on the identified eye (Berard, [0126]: teaches a 3D face scan "can include a portion of the face surrounding both eyes <read on second point>, a portion of the face around one eye, the entire face, or any other suitable portion of the face");
transforming the first point to model the identified eye as a surface that is translucent (Berard, [0046]: teaches "the eyeball model can also include a rigid transformation <read on transforming first point> for the eyeball as well as a uniform scale factor"; [0047]: teaches the deformation eyeball model <read on identified eye> including semantic features that are used to establish correspondence, such as the white sclera, the transparent <read on translucent> cornea, and the limbus; [0048]: teaches using an iterative morphable model technique that alters between establishing correspondences and computing the eyeball model; Note: it should be noted that the cornea is a clear, dome-shaped outer surface of the eye);
warping the second point [[to predict a rotation and a translation associated with the second point]] (Berard, [0129]: teaches "fitting the eyeball model to the at least one eye can include identifying a limbus of the at least one eye in the one or more input images, and minimizing a distance between a limbus of the eyeball model and the identified limbus of the at least one eye"; [0130]: teaches "reconstructing the at least one eye using the parametric eye model with the determined parameters," where "the reconstructed eye can be combined with the remaining parts <read on warping second point> of a subject's reconstructed face depicted in the one or more input images"; [0045]: teaches the limbus opacity mask defining a transparency transition from the sclera to the cornea, where "the position of the limbus can be extracted from the limbus opacity mask by mapping a percent opacity level to the mesh");
generating an indicator of reflection based on the transformed first point and the warped second point using a first layer of a model (Berard, [0046]: teaches "the eyeball model can also include a rigid transformation <read on transformed first point> for the eyeball as well as a uniform scale factor"; [0130]: teaches "reconstructing the at least one eye using the parametric eye model with the determined parameters," where "the reconstructed eye can be combined with the remaining parts <read on warped second point> of a subject's reconstructed face depicted in the one or more input images"); and
generating a spherical indicator based on the transformed first point and the warped second point using a second layer of the model (Berard, [0046]: teaches "the eyeball model can also include a rigid transformation <read on transformed first point> for the eyeball as well as a uniform scale factor"; [0130]: teaches "reconstructing the at least one eye using the parametric eye model with the determined parameters," where "the reconstructed eye can be combined with the remaining parts <read on warped second point> of a subject's reconstructed face depicted in the one or more input images").
Berard is analogous art with respect to Knorr because they are from the same field of endeavor, namely modelling detailed 3D facial representations of a user. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a parametric eyeball model to fit into a non-rigid 3D face as taught by Berard into the teaching of Knorr. The suggestion for doing so would result in a detailed and accurate 3D face that is modifiable by the user, thereby allowing the user to create various and natural facial expressions with the 3D face model. Therefore, it would have been obvious to combine Berard with Knorr.
However, the combination of Knorr and Berard does not expressly disclose
warping the second point to predict a rotation and a translation associated with the second point.
Wood discloses
warping the second point to predict a rotation and a translation associated with the second point (Wood, [Section 4 Generative Eye Region Model]: teaches procedural methods for eyelid movement associated with gaze estimation <read on predicting rotation and translation>, where general movement is described as a rotation with different parts of the lid having different rotational axes; FIG. 12 teaches a rendered eye region <read on second point>).
PNG
media_image1.png
491
789
media_image1.png
Greyscale
Wood is analogous art with respect to Knorr, in view of Berard because they are from the same field of endeavor, namely generating realistic 3D deformable eye models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a generative eye region that includes eye gaze parameters, and camera parameters as taught by Wood into the teaching of Knorr, in view of Berard. The suggestion for doing so would allow for real-time modifiable editing and adjustments of the 3D eye model, thereby improving the user experience. Therefore, it would have been obvious to combine Wood with Knorr, in view of Berard.
Regarding Claim 11, Knorr discloses a method (Knorr, [0199]: teaches a light estimation method) comprising:
[[selecting a first point from a 3D model representing an avatar,]]
[[the first point being associated with an eye;]]
[[selecting a second point from the 3D model,]]
[[the second point being associated with a periocular region associated with the eye;]]
generating indicator of reflection [[based on the first point and the second point]] using a first layer of a model (Knorr, [0128]: teaches a human face specific characteristic being a reflection in the eyes <read on generating indicator of reflection>, which can be used to determine the light incident on the face; [0121]: teaches automatically determining face regions using machine learning <read on first layer of model>; Note: it should be noted that although not expressly stated, all neural networks have layers, such as an input layer, hidden layers, and an output layer);
generating a spherical indicator [[based on the first point and the second point]] using a second layer of the model (Knorr, [0129]: teaches a human face specific characteristic being "some pre-modeled radiance transfer for a particular face region like basis images of the face or face region under basis illumination (Spherical Harmonics basis, or point lights) <read on generating spherical indicator> which can be used to find the best fitting linear combination resulting in an estimation for the incident lighting"; [0121]: teaches automatically determining face regions using machine learning <read on second layer of model>),
[[the spherical indicator identifying a rotation and a translation;]]
generating an image point based on the indicator of reflection, and the spherical indicator (Knorr, [0190]: teaches a radiance transfer function specifying "how a particular surface point of a face <read on generating image point> responds to incident light from the environment from a particular direction in terms of radiance reflected <read on indicator of reflection> at this point towards the camera," where "the radiance transfer function depends mainly on the surface orientation at the location <read on spherical indicator> and occlusions of parts of the distant environment at the surface location by local geometry, but also includes material properties and interreflections of light between local geometry"); and
generating an image including the image point (Knorr, [0194]: teaches calculating the radiance transfer function for a particular surface point in the image, which is displayed to the user <read on generating image including image point>).
However, Knorr does not expressly disclose
selecting a first point from a 3D model representing an avatar,
the first point being associated with an eye;
selecting a second point from the 3D model,
the second point being associated with a periocular region associated with the eye;
generating indicator of reflection based on the first point and the second point using a first layer of a model; and
generating a spherical indicator based on the first point and the second point using a second layer of the model,
the spherical indicator identifying a rotation and a translation.
Berard discloses
selecting a first point from a 3D model representing an avatar (Berard, [0129]: teaches "identifying a limbus of the at least one eye in the one or more input images, and minimizing a distance between a limbus of the eyeball model and the identified limbus <read on first point> of the at least one eye"),
the first point being associated with an eye (Berard, [0129]: teaches "identifying a limbus of the at least one eye in the one or more input images, and minimizing a distance between a limbus of the eyeball model and the identified limbus <read on first point> of the at least one eye");
selecting a second point from the 3D model (Berard, [0126]: teaches a 3D face scan "can include a portion of the face surrounding both eyes <read on second point>, a portion of the face around one eye, the entire face, or any other suitable portion of the face"),
the second point being associated with a periocular region associated with the eye (Berard, [0126]: teaches a 3D face scan "can include a portion of the face surrounding both eyes <read on second point associated with periocular region>, a portion of the face around one eye, the entire face, or any other suitable portion of the face");
generating indicator of reflection based on the first point and the second point using a first layer of a model (Berard, [0046]: teaches "the eyeball model can also include a rigid transformation <read on transformed first point> for the eyeball as well as a uniform scale factor"; [0130]: teaches "reconstructing the at least one eye using the parametric eye model with the determined parameters," where "the reconstructed eye can be combined with the remaining parts <read on warped second point> of a subject's reconstructed face depicted in the one or more input images"); and
generating a spherical indicator based on the first point and the second point using a second layer of the model (Berard, [0046]: teaches "the eyeball model can also include a rigid transformation <read on transformed first point> for the eyeball as well as a uniform scale factor"; [0130]: teaches "reconstructing the at least one eye using the parametric eye model with the determined parameters," where "the reconstructed eye can be combined with the remaining parts <read on warped second point> of a subject's reconstructed face depicted in the one or more input images"),
[[the spherical indicator identifying a rotation and a translation]].
Berard is analogous art with respect to Knorr because they are from the same field of endeavor, namely modelling detailed 3D facial representations of a user. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a parametric eyeball model to fit into a non-rigid 3D face as taught by Berard into the teaching of Knorr. The suggestion for doing so would result in a detailed and accurate 3D face that is modifiable by the user, thereby allowing the user to create various and natural facial expressions with the 3D face model. Therefore, it would have been obvious to combine Berard with Knorr.
However, the combination of Knorr and Berard does not expressly disclose
the spherical indicator identifying a rotation and a translation.
Wood discloses
the spherical indicator identifying a rotation and a translation (Wood, [Section 4 Generative Eye Region Model]: teaches procedural methods for eyelid movement associated with gaze estimation <read on identifying rotation and translation>, where general movement is described as a rotation with different parts of the lid having different rotational axes; FIG. 12 teaches including eye gaze parameters <read on spherical indicator>).
Wood is analogous art with respect to Knorr, in view of Berard because they are from the same field of endeavor, namely generating realistic 3D deformable eye models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a generative eye region that includes eye gaze parameters, and camera parameters as taught by Wood into the teaching of Knorr, in view of Berard. The suggestion for doing so would allow for real-time modifiable editing and adjustments of the 3D eye model, thereby improving the user experience. Therefore, it would have been obvious to combine Wood with Knorr, in view of Berard.
Regarding Claim 18, it recites the limitations that are similar in scope to Claim 11, but in a non-transitory computer-readable storage medium. As shown in the rejection, the combination of Knorr, Berard, and Wood discloses the limitations of Claim 11. Additionally, Knorr discloses a non-transitory computer-readable storage medium (Knorr, [0080]: teaches software code sections being contained on a non-transitory computer readable medium) comprising
instructions stored thereon that, when executed by at least one processor, are configured to cause a computing system to (Knorr, [0080]: teaches software code sections <read on instructions> being contained on a non-transitory computer readable medium; [0102]: teaches mobile device 500 <read on computing system> including a processing device 518, such as a microprocessor <read on processor>):…
Thus, Claim 18 is met by Knorr according to the mapping presented in the rejection of Claim 1, given the method corresponds to non-transitory computer-readable storage medium.
Regarding Claim 5, the combination of Knorr, Berard, and Wood discloses the method of Claim 1. The combination of Knorr and Wood does not expressly disclose the limitations of Claim 5; however, Berard discloses wherein the transforming of the first point includes
explicit modeling of a surface of the eye (Berard, [0111]: teaches "the different geometric surface structure can be seen in the different irises 1002, 1004, 1006, which is inherently linked to the color" as shown in FIG. 10; [0095]: teaches a multi-view stereo (MVS) reconstruction algorithm that "reconstructs the white sclera well since the surface of the sclera is mostly diffuse," where "the white sclera reconstruction may a t lower quality than skin due to strong specular reflections which result in a noisier surface"; [0095]: further teaches a "parametric eye model can serve as a regularizer to get rid of the noise").
PNG
media_image2.png
355
421
media_image2.png
Greyscale
Berard is analogous art with respect to Knorr, in view of Wood because they are from the same field of endeavor, namely modelling detailed 3D facial representations of a user. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a parametric eyeball model to fit into a non-rigid 3D face as taught by Berard into the teaching of Knorr, in view of Wood. The suggestion for doing so would result in a detailed and accurate 3D face that is modifiable by the user, thereby allowing the user to create various and natural facial expressions with the 3D face model. Therefore, it would have been obvious to combine Berard with Knorr, in view of Wood.
Regarding Claim 6, the combination of Knorr, Berard, and Wood discloses the method of Claim 1. The combination of Knorr and Berard does not expressly disclose the limitations of Claim 6; however, Wood discloses wherein the warping of the second point includes
generating a deformable volumetric reconstruction for a periocular region associated with the eye (Wood, FIG. 5 teaches a generated deformable model of the periocular region; Note: it should be noted that the periocular region is the surrounding area of the eye, which can include eyelids, eyebrows, eye bags, etc.).
PNG
media_image3.png
515
791
media_image3.png
Greyscale
Wood is analogous art with respect to Knorr, in view of Berard because they are from the same field of endeavor, namely generating realistic 3D deformable eye models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a generative eye region that includes eye gaze parameters, and camera parameters as taught by Wood into the teaching of Knorr, in view of Berard. The suggestion for doing so would allow for real-time modifiable editing and adjustments of the 3D eye model, thereby improving the user experience. Therefore, it would have been obvious to combine Wood with Knorr, in view of Berard.
Regarding Claims 8 and 17, the combination of Knorr, Berard, and Wood discloses the methods of Claim 1 and 11 respectively. The combination of Knorr and Wood does not expressly disclose the limitations of Claims 8 and 17; however, Berard discloses wherein the generating of the image point includes
changing a view direction of the eye (Berard, [0106]: teaches multi-pose fitting for the eyeball, where the pupil position affects the view direction as shown in FIG. 8A).
PNG
media_image4.png
290
330
media_image4.png
Greyscale
Berard is analogous art with respect to Knorr, in view of Wood because they are from the same field of endeavor, namely modelling detailed 3D facial representations of a user. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a parametric eyeball model to fit into a non-rigid 3D face as taught by Berard into the teaching of Knorr, in view of Wood. The suggestion for doing so would result in a detailed and accurate 3D face that is modifiable by the user, thereby allowing the user to create various and natural facial expressions with the 3D face model. Therefore, it would have been obvious to combine Berard with Knorr, in view of Wood.
Claims 2-4, 7, 9-10, 12-16, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Knorr et al. (US 20150279113 A1, previously cited), hereinafter referenced as Knorr, in view of Bêrard et al. (US 20180012418 A1, previously cited), hereinafter referenced as Berard, and further in view of Wood et al. ("Leaning an Appearance-Based Gaze Estimator from One Million Synthesized Images"), hereinafter referenced as Wood as applied to Claims 1, 11, and 18 above respectively, and further in view of Zhao et al. (US 20230027890 A1, previously cited), hereinafter referenced as Zhao.
Regarding Claims 2 and 12, the combination of Knorr, Berard, and Wood discloses the methods of Claims 1 and 11 respectively. The combination of Knorr, Berard, and Wood does not expressly disclose the limitations of Claims 2 and 12; however, Zhao discloses
storing a plurality of image points using the generated image point (Zhao, [0098]: teaches encoding that are "sent through the first and second neural networks with a plurality of RGB image representations," where "the output length of the second neural network (that is, volumetric light map network) is 12 for each queried 3D location <read on stored image points> while that of the first neural network (that is, material network) is 4 including one channel for skin roughness and 3 channels for skin sub-surface scattering"); and
generating an image representing the avatar based on a plurality of image points (Zhao, [0098]: teaches encoding that are "sent through the first and second neural networks with a plurality of RGB image representations," where "the output length of the second neural network (that is, volumetric light map network) is 12 for each queried 3D location while that of the first neural network (that is, material network) is 4 including one channel for skin roughness and 3 channels for skin sub-surface scattering"; [0098]: further teaches "transport aggregation is performed to obtain a weighted value for each ray based on a density distribution among the sampled points along the ray, where the values of each component on rays are aggregated with introduced values from preprocessed albedo and normal maps, and then visualized as an image with pixels representing their intensities following a rendering equation (1)," where "the rendered RGB values <read on image points> are constrained with ground truth pixel colors").
Zhao is analogous art with respect to the combination of Knorr, Berard, and Wood because they are from the same field of endeavor, namely utilizing neural networks for lighting 3D face models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a volumetric and material neural networks as taught by Zhao into the combined teaching of Knorr, Berard, and Wood. The suggestion for doing so would allow the neural networks to better predict lighting effects on the face model, including the eyes, thereby resulting in a more natural-looking face model. Therefore, it would have been obvious to combine Zhao with the combination of Knorr, Berard, and Wood.
Regarding Claims 3 and 13, the combination of Knorr, Berard, Wood, and Zhao discloses the methods of Claims 2 and 12 respectively. The combination of Knorr, Berard, and Wood does not expressly disclose the limitations of Claims 3 and 13; however, Zhao discloses wherein the generating of the image representing the avatar includes
rendering the image representing the avatar using raytracing to compute reflection rays and refraction rays (Zhao, [0099]: teaches "generating a volumetric neural radiance field based on at least the volumetric skin reflectance field (obtained at 306) and the volumetric light map (obtained at 310)," where "based on one or more input rendering assets, including the three-dimensional geometry, the albedo map, and the high-frequency map, a density field is generated based on an absolute depth of a sample to ray-mesh intersections <read on computing reflection and refraction rays>, which is a distance from any sample to an intersecting point on the three-dimensional geometric mesh along a ray direction").
Zhao is analogous art with respect to the combination of Knorr, Berard, and Wood because they are from the same field of endeavor, namely utilizing neural networks for lighting 3D face models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a volumetric and material neural networks as taught by Zhao into the combined teaching of Knorr, Berard, and Wood. The suggestion for doing so would allow the neural networks to better understand lighting behaviors on the face model, including the eyes, thereby resulting in a more natural-looking face model. Therefore, it would have been obvious to combine Zhao with the combination of Knorr, Berard, and Wood.
Regarding Claim 19, the combination of Knorr, Berard, and Wood discloses the non-transitory computer-readable storage medium of Claim 18. The combination of Knorr, Berard, and Wood does not expressly disclose the limitations of Claim 19; however, Zhao discloses wherein the instructions are further configured to cause the computing system to
render a plurality of image points representing the avatar using raytracing to compute reflection rays and refraction rays to generate an image representing the avatar (Zhao, [0099]: teaches "generating a volumetric neural radiance field based on at least the volumetric skin reflectance field (obtained at 306) and the volumetric light map (obtained at 310)," where "based on one or more input rendering assets, including the three-dimensional geometry, the albedo map, and the high-frequency map, a density field is generated based on an absolute depth of a sample to ray-mesh intersections <read on computing reflection and refraction rays>, which is a distance from any sample to an intersecting point on the three-dimensional geometric mesh along a ray direction").
Zhao is analogous art with respect to the combination of Knorr, Berard, and Wood because they are from the same field of endeavor, namely utilizing neural networks for lighting 3D face models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a volumetric and material neural networks as taught by Zhao into the combined teaching of Knorr, Berard, and Wood. The suggestion for doing so would allow the neural networks to better understand lighting behaviors on the face model, including the eyes, thereby resulting in a more natural-looking face model. Therefore, it would have been obvious to combine Zhao with the combination of Knorr, Berard, and Wood.
Regarding Claim 4, the combination of Knorr, Berard, and Wood discloses the method of Claim 1. The combination of Knorr, Berard, and Wood does not expressly disclose the limitations of Claim 4; however, Zhao discloses wherein the SH coefficients include
specular SH coefficients (Zhao, [0132]: teaches a "direct specular component <read on specular SH coefficients> models the light that comes from the light source and is reflected directly on the surface without experiencing energy absorption," where "given all possible incident ray directions
ω
i
at a single point x, the direct specular radiance
L
d
s
of outgoing direction
ω
o
is calculated") and
diffuse SH coefficients (Zhao, [0135]: teaches a direct diffuse component <read on diffuse SH coefficients>, which "takes into account the energy absorption between the incident light and particles of the material" and "to measure diffuse reflection of radiance out of the total incident radiance, the direct diffuse radiance Lad in outgoing direction
ω
o
is calculated").
Zhao is analogous art with respect to the combination of Knorr, Berard, and Wood because they are from the same field of endeavor, namely utilizing neural networks for lighting 3D face models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a volumetric and material neural networks as taught by Zhao into the combined teaching of Knorr, Berard, and Wood. The suggestion for doing so would allow the neural networks to predict how light affects certain surface points of the face model, such as the eyes, resulting in a more accurate and realistic output. Therefore, it would have been obvious to combine Zhao with the combination of Knorr, Berard, and Wood.
Regarding Claims 7 and 16, the combination of Knorr, Berard, and Wood discloses the methods of Claims 1 and 11 respectively. The combination of Knorr, Berard, and Wood does not expressly disclose the limitations of Claims 7 and 16; however, Zhao discloses
disentangling a reflectance associated with environmental lighting (Zhao, [0128]: teaches "lighting <read on reflectance associated with environmental lighting> can be disentangled from texture by adjusting weight in density and visibility field while keeping the lightmap fully differentiable"); and
relighting the image point based on an environmental map (Zhao, [0072]: teaches specular reflectance reconstruction <read on relighting image point>, where "with reconstructed assets, camera calibration, and all-white baked environmental texture, ray-based rendering may be performed under all white, matching the environment <read on environmental map> of capture and training on combined color images").
Zhao is analogous art with respect to the combination of Knorr, Berard, and Wood because they are from the same field of endeavor, namely utilizing neural networks for lighting 3D face models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a volumetric and material neural networks as taught by Zhao into the combined teaching of Knorr, Berard, and Wood. The suggestion for doing so would allow the neural networks to better understand lighting behaviors on the face model, including the eyes, thereby resulting in a more natural-looking face model. Therefore, it would have been obvious to combine Zhao with the combination of Knorr, Berard, and Wood.
Regarding Claims 9, 14, and 20, the combination of Knorr, Berard, and Wood discloses the methods and the non-transitory computer-readable storage medium of Claims 1, 11, and 18 respectively. Additionally, Knorr further discloses wherein the model includes
[[a trained Neural Radiance Fields (NeRF) with Spherical Harmonics Lighting (SHL) model trained to]]
generate the indicator of reflection as an albedo and trained (Knorr, [0128]: teaches a human face specific characteristic being a reflection in the eyes <read on generating indicator of reflection as albedo>, which can be used to determine the light incident on the face) to
generate the spherical indicator as spherical harmonic coefficients using the transformed first point and the warped second point as inputs to the trained model (Knorr, [0129]: teaches a human face specific characteristic being "some pre-modeled radiance transfer for a particular face region like basis images of the face or face region under basis illumination (Spherical Harmonics basis, or point lights) <read on generating spherical indicator as spherical harmonic coefficients> which can be used to find the best fitting linear combination resulting in an estimation for the incident lighting").
However, the combination of Knorr, Berard, and Wood does not expressly disclose
a trained Neural Radiance Fields (NeRF) with Spherical Harmonics Lighting (SHL) model trained to…
Zhao discloses
a trained Neural Radiance Fields (NeRF) with Spherical Harmonics Lighting (SHL) model trained to (Zhao, [0064]: teaches memory 120 storing a volumetric light map generation module, where "the volumetric light map generation module 126 may include a second neural network (also referred to herein as volumetric light map network) for generating a set of representations including spherical harmonic representations <read on SHL model> that model indirect light behavior based on a desired lighting condition"; [0065]: teaches memory 120 storing "a rendering module 128 that includes instructions for generating a volumetric neural radiance field based on one or more of the skin reflectance representations from the first neural network, the volumetric light map from the second neural network <read on trained NeRF>, and the one or more rendering assets")…
Zhao is analogous art with respect to the combination of Knorr, Berard, and Wood because they are from the same field of endeavor, namely utilizing neural networks for lighting 3D face models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a volumetric and material neural networks as taught by Zhao into the combined teaching of Knorr, Berard, and Wood. The suggestion for doing so would allow the neural networks to better understand lighting behaviors on the face model, including the eyes, thereby resulting in a more natural-looking face model. Therefore, it would have been obvious to combine Zhao with the combination of Knorr, Berard, and Wood.
Regarding Claims 10 and 15, the combination of Knorr, Berard, Wood, and Zhao discloses the methods of Claim 9 and 14 respectively. The combination of Knorr, Berard, and Wood does not expressly disclose the limitations of Claims 10 and 15; however, Zhao discloses wherein the NeRF with SHL model is trained with a subject at least one of:
following a mobile camera with a gaze of the subject while keeping their head static and forward facing (Zhao, [0059]: teaches the computing system 100 being a mobile phone <read on mobile camera>; [0079]: teaches one or more rendering assets 202 that correspond to one or more digital images including a subject, where one of the views is a front view <read on head being static and forward facing>; [0127]: teaches the visibility at each point is defined to model whether the point is visible from view direction
ω
o
<read on gaze>);
focusing on a first static camera (Zhao, FIG. 21 teaches camera locations 2002 for the left, front, and right cameras, where the cameras are stationary <read on focus on first static camera>) and
PNG
media_image5.png
356
561
media_image5.png
Greyscale
changing the gaze of the subject to a second static camera (Zhao, FIG. 21 teaches camera locations 2002, where the left, front and right cameras have different views of the subject <read on change gaze of subject to second static camera>); and
focusing on the first static camera (Zhao, FIG. 21 teaches camera locations 2002 for the left, front, and right cameras, where the cameras are stationary <read on focus on first static camera>) and
rotating a head of the subject in a pattern with eyes of the subject static (Zhao, [0068]: teaches "the platform may rotate the object <read on head of subject> with various rotation angle steps to allow the object and/or the subject to be seen by the camera(s) 174 at desired angles"; [0067]: teaches "the one or more light source(s) 172 may be configured to illuminate the subject and/or the object with a controllable field of illumination using threshold-order (e.g., up to 5th order) Spherical Harmonics (SH) or Fourier Series (FS) illumination patterns").
Zhao is analogous art with respect to the combination of Knorr, Berard, and Wood because they are from the same field of endeavor, namely utilizing neural networks for lighting 3D face models. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to capture the head of the user with stationary cameras as taught by Zhao into the combined teaching of Knorr, Berard, and Wood. The suggestion for doing so would result in a more accurate representation of the user's head, thereby preparing the model for modification. Therefore, it would have been obvious to combine Zhao with the combination of Knorr, Berard, and Wood.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Schwartz et al. (US 11010951 B1) discloses generating an avatar eye with a 3D mesh and eyeball texture individually; and
Zhang et al. (US 20220319055 A1) discloses using a NERF framework to extract feature maps from multi-view images.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KARL TRUONG whose telephone number is (703)756-5915. The examiner can normally be reached 7:30 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.D.T./Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614