Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on July 7th, 2026, has been entered.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-2, 4-15, 17-18 and 21-24 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the enablement requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to enable one skilled in the art to which it pertains, or with which it is most nearly connected, to make and/or use the invention.
Claim 1 recites “obtaining initial values of the PCA coefficients for the eye patch areas by interpolating 3D vertex positions at barycentric coordinates corresponding to two-dimensional (2D) face landmarks detected in the one or more images”. However, the Applicant Specification does not specify such a limitation. The closest support available in the Specification is in Paragraph [0054], which specifies that “Given a set of (new) input images, such as at 502 in FIG. 5, initial values for the model parameters can obtained from the input images. Optimization on the initial values can then be performed. Given the initial values of the model parameters, some embodiments can evaluate the 3D position of all vertices. From the vertex positions, we can interpolate the 3D position at the barycentric coordinates with a calibrated pinhole camera model that can compute the projection of the face landmark points”. “Model parameters” is contextualized in Paragraph [0055]: “In one implementation, the head and eye patch parameters includes: principal component analysis (PCA) shape coefficients”.
However, Paragraph [0054] specifies that 3D positions of vertices can be evaluated given initial values of model parameters (which include PCA coefficients), where from the vertex positions, interpolation of the 3D position at the barycentric coordinates can be performed with a calibrated pinhole camera model that computes the projection of the face landmark points.
The nature of the invention is centered around the generation of a 3D model of a head. The method of doing so depends on optimizing model parameters, of which performing PCA and optimizing PCA coefficients is a central component. The optimization of the model for generating the 3D model becomes unclear because there is not support in the specification for “obtaining initial values of the PCA coefficients for the eye patch areas by interpolating 3D vertex positions at barycentric coordinates corresponding to two-dimensional (2D) face landmarks detected in the one or more images”, as the Specification indicates that interpolating vertex positions at barycentric coordinates is performed given the initial values, and does not teach obtaining the initial values as a result of interpolating vertex positions at barycentric coordinates. The breadth of the claims becomes unclear due to a lack of support for the optimization of the model in the Specification regarding the aforementioned limitation.
Considering the Specification does not provide clear instruction regarding “obtaining initial values of the PCA coefficients for the eye patch areas by interpolating 3D vertex positions at barycentric coordinates corresponding to two-dimensional (2D) face landmarks detected in the one or more images”, a person having ordinary skill in the art would not be able to perform the method of the invention.
As a result of the aforementioned analysis, undue experimentation would be necessitated to make the invention based on the contents of the disclosure.
Claim 14, being similar in scope to Claim 1, is rejected under the same rationale for lack of enablement.
Claim 18, being similar in scope to Claim 1, is rejected under the same rationale for lack of enablement.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 4-5, 10, 14, 17 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Wood (A 3D Morphable Eye Region Model for Gaze Estimation, 2016), in view of Ploumpis (Towards a complete 3D morphable model of the human head, 2019), Ciuc (Pub. No. US 2019/0279393 A1) and Park (3D face econstruction from a single 2D face image, 2008).
Regarding Claim 1, Wood teaches A method for generating a three-dimensional model, comprising:
Obtaining one or more images of the head, wherein the head includes eyes (Section 1, An Eye Region 3DMM: “We constructed a 3DMM of the facial eye region by carefully registering a set of high-quality 3D head scans”, where the image and scans are visualized in Figure 1);
retrieving a parametric model for the eyes that includes a set of parameters (Section 2.1: “A 3D morphable model is a statistically-derived generative model, parameterized by shape and texture coefficients”; Section 4: “While this model is simple, we found it to be sufficient. If we considered a larger facial region, or fit models to both eyes at once, we would explore more advanced material or illumination models”, where extending the model to both eyes is suggested to be an anticipated and largely possible extension);
assigning values for each parameter in the set of parameters of the parametric model for the eyes based on the one or more images (Figure 3: “An overview our fitting process: We localize landmarks L in an image, and use them to initialze our 3DMM. We then use analysis-by-synthesis to render a [synthesized image] that best matches [an observed image]”, where Figure 3 clearly depicts using the Landmarks to fill the initial parameters);
generating eye patch areas of areas surrounding the eyes based on the values of the parameters in the set of parameters of the parametric model for the eyes (Section 1, An Eye Region 3DMM: “We constructed a 3DMM of the facial eye region by carefully registering a set of high-quality 3D head scans, and extracting modes of shape and texture variation using PCA”, where Figure 4 and Figure 5 visualize the eye patch area model);
wherein the eye patch areas are represented by principle component analysis (PCA) coefficients obtained based on a database of head meshes, each head mesh in the database including eyes (Figure 4: “We re-parameterize high-resolution 3D head scan data (left)”; Refer to Figure 4, which shows a head model (mesh) with eyes; Section 4, 3D Eye Region Model, Morphable facial eye region model: “We started by acquiring 22 high-quality head scans as source data”; Section 1, An Eye Region 3DMM: “We constructed a 3DMM of the facial eye region by carefully registering a set of high-quality 3D head scans, and extracting modes of shape and texture variation using PCA”; Section 4, 3D Eye Region Model, Morphable facial eye region model: “We then performed Principle Component Analysis (PCA) on our set of c ordered scans to extract orthogonal shape and texture basis functions: U … and V… Facial eye region shapes s and textures t can then be generated from shape (
β
f
a
c
e
)… and texture coefficients (
τ
f
a
c
e
)”; Equation 5 and Equation 6 define the coefficients as being coefficients as being PCA coefficients, since U and V are derived via PCA. Notes: The broadest reasonable interpretation of a mesh is a collection of vertices defining a 3D model, as is apparent in Figure 4);
optimizing the initial values of the PCA coefficients to obtain optimized PCA coefficients by performing gradient descent on an objective function that directly minimizes in an image space, distances between the 2D face landmarks and projections of 3D vertices of the head and the eye patch areas (Section 5, Analysis-by-synthesis for Gaze Estimation: We cast this as an unconstrained energy minimization problem for unknown ϕ”; Refer to Equation 9, which describes the minimization problem as an optimization through minimizing the energy objection function E(ϕ); Equation 10 defines the Energy function as being the sum between the Energy of the image and the Energy of the landmarks given parameters ϕ; Section 5, Analysis-by-synthesis for Gaze Estimation, Landmark similarity metric: “The face contains important landmark feature points that can be localized reliably [13]. These can be used to efficiently consider the appearance of the whole face, as well as the local appearance of the eye region. We use a state-of-the-art face tracker [15] to localize 14 landmarks L around the eye region in image-space (see Fig. 8). For each landmark
l
∈
L
we compute a corresponding synthesized landmark
l
'
using our 3DMM. The sparse landmark-similarity term is calculated as the distance between both sets of landmarks, normalized by the foreground area to avoid bias from image or eye region size. This acts as a regularizer to prevent our pose from drifting too far from a reliable estimate”; Refer to Equation 12 regarding the distance difference between the synthetic landmark derived from the 3DMM (projected 3D vertices) and the 2D landmark. Section 1, Analysis-by-Synthesis: “We iteratively fit our model using gradient descent with numerical derivatives efficiently calculated with a tailored GPU rasterizer”; Figure 7 also demonstrates how the model appears after each iteration, where each iteration visualizes a model in which loss is minimized via gradient descent. Notes: The broadest reasonable interpretation of applying a gradient descent algorithm to raw data from an image is utilizing gradient descent as a means for minimizing loss between the output of a generation model and a target image, where the raw data includes image data, which is subsequently used for image processing); and
generating, based on the optimized PCA coefficients, the 3D model of the eyes and the eye patch areas (Section 1, An Eye Region 3DMM: “We constructed a 3DMM of the facial eye region by carefully registering a set of high-quality 3D head scans, and extracting modes of shape and texture variation using PCA. We combined this with an anatomy-based eyeball model that can be posed separately to simulate changes in eye gaze”, where Figure 4 and Figure 5 visualize the eye patch area model and Figure 6 visualizes the eye model; See Equation 1 and Equation 9).
Wood does not teach obtaining initial values of the PCA coefficients for the eye patch areas by interpolating 3D vertex positions at barycentric coordinates corresponding to two-dimensional (2D) face landmarks detected in the one or more images.
However, Park teaches obtaining initial values that can define the initial value of the PCA coefficients for the eye patch areas by interpolating 3D vertex positions at barycentric coordinates corresponding to two-dimensional (2D) face landmarks detected in the one or more images (Section 4, 3D Face Reconstruction: “In this chapter, we propose a way to reconstruct a 3D face from feature points and pose estimation by interpolating more points in an input 2D image and estimating the corresponding vertices in an output 3D face. After feature detection and pose estimation by the proposed method, we obtain the 3D positions of the 79 feature points, but they are not enough to reconstruct a whole 3D face. For example, when we model a 3D face based on the 3D geometrical structure of the USF Human-ID database, it is required to obtain both shape and texture of 75,972 vertices. As the first step to get this information, we interpolate 75,893 points in an input 2D image by a simple way using barycentric coordinates. Next, we estimate the 3D vertices with the depths in a 3D face by a linear deformable model; Refer to Figure 3 for a visualization” Section 4, 3D Face Reconstruction, Interpolating 2D points in a Frontal View: “Fortunately, the difference between the shapes of a specific face and the average face is not significant. Inspired by this idea, first of all for interpolation, pose correction of the 2D feature points to the frontal view is applied by the estimated pose parameters. After pose correction to the frontal view, the transformed feature points can be easily compared to the corresponding points in the average shape obtained from a training set consisting of the 3D frontal faces. The process for interpolation is shown in Figure 3”. Notes: The broadest reasonable interpretation of 2D landmark points are feature points that describe structure).
While Park does not explicitly state using the obtained values for initial PCA values, a person having ordinary skill in the art would appreciate that Wood uses shape parameters (defined by 3D vertices, which are derived in Park) to define the PCA coefficients (Refer to Equation 5 and Equation 6), where given a set of vertices that define shape (such as those of Park), Equation 5 or Equation 6 can be used to determine initial PCA coefficient values by interpolating 3D vertex positions.
Wood and Park are considered analogous in the art with respect to the generating models of human faces utilizing PCA. facial reconstruction of an image is well known within the art. One would be motivated to utilize 2D facial landmarks to derive 3D correspondences to generate face/head models of a face or head from an image, as is evident in Park; such an application has known uses in character customization and identification.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the face/eye patch area 3D model generation utilizing PCA and head scans of Wood with the interpolation of 3D vertex positions of Park; Doing so would yield the predictable result of obtaining initial PCA coefficient values from interpolated 3D vertex positions to be optimized through a face/head generation model.
Wood as modified does not teach that each head mesh in the database including eyes is normalized to be spaced a fixed distance apart from one another, and does not teach a generated 3D model of the head with eyes is normalized to be spaced the fixed distance apart from one another in the 3D model, and wherein a size of the head in the 3D model is scaled based on the fixed distance between the eyes
However, Ciuc teaches modifying a head from a database of heads with eyes such that each head mesh in the database including eyes are normalized to be spaced a fixed distance apart from one another and a generated 3D model of the head with eyes is normalized to be spaced the fixed distance apart from one another in the 3D model, and wherein a size of the head in the 3D model is scaled based on the fixed distance between the eyes (Paragraph [0065]: “In an implementation, a database of 3D models of human heads 108, such as the set of 35, is artificially generated using a modeler… In an implementation, the models 108 are scaled such that the interpupillary distance is 63 mm, which is the average distance for an adult person”; Paragraph [0064]: “In an implementation, the example features tracker 100 operates with averaged 3D models 108 of the head, with facial landmarks represented by feature points at average positions, on which is applied a set of transformations to match a 3D model 108 to the head of the subject face 102 in the current image 104”. Notes: The database of heads are derived from images containing faces, where each head in the database is scaled such that the interpupillary distance, which describes the distance between the eyes, is fixed at 63 mm).
Wood as modified and Ciuc are considered analogous in the art with respect to modeling with respect to a human face from an image. A common motivation in the art is to normalize faces to be modeled such that faces can be more easily generalized for modeling purposes.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the method of 3D head model generation with eyes of Wood as modified with the eye space normalization and head scaling of Ciuc; Doing so would yield the predictable result of a more easily generated 3D head model.
Wood as modified does not teach generating the 3D model of the head based on the optimized PCA coefficients.
However, Ploumpis teaches generating, based on optimized PCA coefficients, the 3D model of the head with the eye and eye patch areas (Abstract: “Abstract—Three-dimensional Morphable Models (3DMMs) are powerful statistical tools for representing the 3D shapes and textures of an object class. Here we present the most complete 3DMM of the human head to date that includes face, cranium, ears, eyes, teeth and tongue. To achieve this, we propose two methods for combining existing 3DMMs of different overlapping head parts: i. use a regressor to complete missing parts of one model using the other, ii. use the Gaussian Process framework to blend covariance matrices from multiple models. Thus we build a new combined face-and-head shape model that blends the variability and facial detail of an existing face model (the LSFM) with the full head modelling capability of an existing head model (the LYHM). Then we construct and fuse a highly-detailed ear model to extend the variation of the ear shape. Eye and eye region models are incorporated into the head model … We use our model to reconstruct full head representations from single, unconstrained images allowing us to parameterize craniofacial shape and texture, along with the ear shape, eye gaze and eye color”, where Figures 7-9 clearly depict a 3DMM model of the eye and eye patch area similar to that of Wood; the eye and eye patch area models are integrated into the 3D model of the head).
Wood as modified and Ploumpis are considered analogous in the art with respect to working with 3DMMs for generating eye and eye patch areas. Given that the method for generating a 3D head model of Ploumpis integrates 3DMM models of the eyes and eye patch areas, it is not a novel concept to integrate models of the eyes and eye patch areas (such as those of Wood) into a 3D head model; the motivation for doing so would be to create a more detailed and accurate 3D head model, especially with regards to how the region around the eyes, and the eyes themselves, visually change as a person looks in different directions.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the method of generating eye and eye patch area models of Wood as modified with the 3D model generation of the head with eye and eye patch areas of Ploumpis; Doing so would yield the predictable result of the generation of a 3D head model with specifically fitted eye and eye patch areas.
Regarding Claim 4, the method of Claim 1 is rejected over Wood as modified.
Wood as modified teaches normalizing the eyes of each head mesh by spacing the distance between the eyes using the average distance between the eyes (See Ciuc, “Paragraph [0065]: “In an implementation, a database of 3D models of human heads 108, such as the set of 35, is artificially generated using a modeler… In an implementation, the models 108 are scaled such that the interpupillary distance is 63 mm, which is the average distance for an adult person”).
However, Wood as modified does not explicitly teach computing the average distance between the eyes, nor does it explicitly teach computing the average distance for head meshes in particular.
While Wood as modified does not explicitly teach computing the average distance between the eyes, calculating the average distance between the eyes is clearly known in the art through the established average for an adult person; said established average would have been calculated from a collection of human heads, wherein the collection is akin to a database. Therefore, through the broadest reasonable interpretation of an average, the established average distance between the eyes would have been calculated in a manner analogous with the method of calculating an average distance between the eyes of a database of heads in the claimed invention. A motivation for applying the average to a specific database of heads not necessarily representative of the general human population would be to obtain an average representative of the specific database of heads.
Furthermore, while Wood as modified does not explicitly teach applying the average distance between the eyes to a database of head meshes (as opposed to heads in general), Ciuc clearly teaches normalizing the eyes of each head mesh by spacing the distance between the eyes using the average distance between the eyes. Considering that the eyes in the meshes are spaced to be 63 mm, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention that the average between the eyes can be applied to heads and head meshes alike, especially in a scenario where eyes in head meshes are themselves normalized to a specific distance (63 mm).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to integrate calculating average eye spacing for heads/head meshes in a database of Ciuc with the database of heads and use of an average distance between the eyes of Wood as modified; doing so would yield the predictable result of a plurality of anatomically accurate 3D models of heads with an eye spacing representative of the average eye spacing of the heads within the database.
Regarding Claim 5, the method of Claim 1 is rejected over Wood as modified.
Wood as modified teaches assigning the values for each parameter in the set of parameters of the parametric model for the eyes based on the one or more images, wherein doing so comprises:
Obtaining an initial set of parameter values for the set of parameters of the parametric model for the eyes based on the one or more images (Wood, Figure 3 demonstrates that an initial set of parameters
PNG
media_image1.png
1
1
media_image1.png
Greyscale
𝜙 is derived from an input image); and
assigning the values for each parameter in the set of parameters of the parametric model by performing gradient descent on the initial set of parameter values to optimize the parameter values (Wood, Figure 3 demonstrates the optimization of the energy function that describes fitting the model to the eye of the subject; Wood, Section 5.2 further describes the process of using gradient descent to optimize the initial parameter set 𝜙, as demonstrated in Wood, Equation 13-14).
Regarding Claim 10, the method of Claim 1 is rejected over Wood as modified.
Wood as modified teaches rendering an image of the 3D model of the head using differential rendering (Wood, 5.2, Optimization Procedure: “Computing our gradients is expensive, requiring rendering and differencing two images per parameter. Their efficient computation is possible with our tailored GPU DirectX rasterizer that can render [a synthesized image] at over 5000fps”; Notes: Differential rendering within the context of generating images from a 3D model, in its broadest reasonable interpretation, is iteratively comparing a synthesized image with an observed image, and computing a gradient from the difference of the two images. Differential rendering is often combined with parametric models describing the 3D model. In Wood, the difference between the synthesized image and the observed image is defined as the energy function E in Wood, Equation 10; the gradients are defined in Wood, Equation 13 and 14, where Wood, Equation 14 describes how the gradient of the difference E itself is computed using the current parameter values, and Wood, Equation 13 describes how the parameter set is optimized using the acquired gradient E (gradient descent equation). The result of each iteration of gradient descent can be observed in Wood, Fig 7, wherein each iteration generated an image of the eye and eye patch region in accordance with the set of parameters that were updated through gradient descent; With regards to generating an image of the head, it was established in the rejection of Claim 1 that integrating the eye and eye patch models into a 3D head model is not novel. It is noted that while the claim language does not specify a parametric model for the head, a parametric model of the head is present in Ploumpis as indicated by the Abstract: “The new model achieves state-of-the-art performance. We use our model to reconstruct full head representations from single, unconstrained images allowing us to parameterize craniofacial shape and texture, along with the ear shape, eye gaze and eye color”).
Claim 14, being similar in scope to Claim 1, is rejected under the same rationale.
Claim 17, being similar in scope to Claim 4, is rejected under the same rationale.
Claim 18, being similar in scope to Claim 1, is rejected under the same rationale.
Claims 2 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Wood (A 3D Morphable Eye Region Model for Gaze Estimation, 2016), in view of Ploumpis (Towards a complete 3D morphable model of the human head, 2019), Ciuc (Pub. No. US 2019/0279393 A1) and Park (3D face econstruction from a single 2D face image, 2008), and in further view of Kuang (Towards an Accurate 3D Deformable Eye Model for Gaze Estimation, 2022) and Crouch (Parametric Eye Models, 2007).
Regarding Claim 2, the method of Claim 1 is rejected over Wood as modified.
Wood as modified teaches a set of parameters of the parametric model for the eyes (Section 2.1: “A 3D morphable model is a statistically-derived generative model, parameterized by shape and texture coefficients”; Section 4: “While this model is simple, we found it to be sufficient. If we considered a larger facial region, or fit models to both eyes at once, we would explore more advanced material or illumination models”) that broadly encompass geometric parameters such as shape, texture, and pose.
Wood does not teach a set of parameters of the parametric model for the eyes that include iris diameter, cornea radius of curvature, eye width, eye axial length, and iris depth.
However, Kuang teaches a model defined by parameters such as iris diameter, cornea radius of curvature, eye width, and iris depth (Table 1 lists the aforementioned parameters for a 3D eyeball model, with Figure 1 illustrating the model and associated parameters; Notes: iris radius is treated as being analogous to iris diameter, considering the ease in which diameter can be derived from radius. Cornea radius of curvature is analogous to cornea radius, as supported by Figure 1. Eye width is synonymous with Eyeball radius as demonstrated by Figure 1. Lastly, iris depth can easily be derived as the eyeball radius parameter value summed with the distance between eyeball center parameter value and cornea center parameter value).
Crouch teaches a parametric eye model including the parameter of eye axial length (“The eye model presented contains analytically defined shape equations that produce models matching user-specified physical measurements such as … eye axial length”).
Eye modeling is well established in the art. There are numerous parameters to define a parametric eye model, many of which are well documented characteristics of the eyes, or are otherwise easily derived from existing well-established parameters. One ordinarily skilled in the art would be motivated to select certain parameters to best define an eye model, or increase the volume of parameters to have a more detailed and accurate eye model. Combining the parameters used by others in the field is a known approach for defining a parameter set. Consequently, combining the parameter set of Kuang and the parameter set of Crouch would result in a set of parameters containing iris diameter, cornea radius of curvature, eye width, iris depth, and eye axial length without significant experimentation.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date to combine the set of parameters for the parametric eye models of Kuang and Crouch together; doing so would yield the predictable result of a more detailed parametric eye model defined by parameters including iris diameter, cornea radius of curvature, eye width, iris depth, and eye axial length.
Claim 15, being similar in scope to Claim 2, is rejected under the same rationale.
Claims 6-7 and Claim 9 are rejected under 35 U.S.C. 103 as being unpatentable over Wood (A 3D Morphable Eye Region Model for Gaze Estimation, 2016), in view of Ploumpis (Towards a complete 3D morphable model of the human head, 2019), Ciuc (Pub. No. US 2019/0279393 A1) and Park (3D face econstruction from a single 2D face image, 2008), and further in view of Berard (Lightweight eye capture using a parametric model, 2016).
Regarding Claim 6, the method of Claim 1 is rejected over Wood as modified.
Wood as modified teaches generating an iris texture for an iris of the eyes (Wood, Section 4, Parametric eyeball model: “We used a collection of aligned high-resolution iris photos to build a generative model… of iris texture using PCA”).
Wood as modified does not teach modeling the iris using polar coordinates.
However, Berard teaches generating an iris texture for an iris of the eyes, wherein the iris is modeled using polar coordinates (Section 6.1: “The structure of an iris is arranged radially around the pupil. Operating in polar coordinates (angle/radius) unwraps the radial structure (Fig. 5) and presents itself well for synthesis with rectangular patches”, where Figure 5 shows the iris modeled in polar coordinates).
Wood as modified and Berard are considered analogous in the art, as they both generate iris textures for an eye model. One ordinarily skilled in the art of 3D modeling of heads and their associated organs and components would be generally motivated to 3D model a head more realistically, as well as with more detail. Combining the more detailed method of iris generation of Berard with the general 3D modeling method of a head with eyes of Wood as modified would be a logical and obvious step for generating a more detailed and realistic 3D model of a head.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the method of iris texture generation using the polar coordinates of said iris of Berard with the 3D modeling method of a head with eyes of Wood as modified; doing so would yield the predictable result of a more detailed and realistic 3D model of a head, especially with regards to the iris texturing.
Regarding Claim 7, the method of Claim 1 is rejected over Wood as modified.
Wood as modified teaches receiving an image of an iris, wherein the iris is circular, and generating a texture for the iris (Wood, Section 4, Parametric eyeball model: “We used a collection of aligned high-resolution iris photos to build a generative model… of iris texture using PCA”).
Wood as modified does not teach transforming the iris to polar coordinates, wherein the iris transformed to polar coordinates can be represented by a rectangle, and subsequently used to generate a texture for the iris based on the polar coordinates.
However, Berard teaches transforming the iris to polar coordinates, wherein the iris transformed to polar coordinates can be represented by a rectangle, and subsequently used to generate a texture for the iris based on the polar coordinates (Section 6.1: “The structure of an iris is arranged radially around the pupil. Operating in polar coordinates (angle/radius) unwraps the radial structure (Fig. 5) and presents itself well for synthesis with rectangular patches”, where Figure 5 shows the iris modeled in polar coordinates, as well as the texture generated from the iris represented in polar coordinates).
The motivation for combining Wood as modified with Berard from Claim 6 is applied in Claim 7.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the method of iris texture generation using the polar coordinates represented as a rectangle of said iris of Berard with the 3D modeling method of a head with eyes of Wood as modified; doing so would yield the predictable result of a more detailed and realistic 3D model of a head, especially with regards to the iris texturing.
Regarding Claim 9, the method of Claim 7 is rejected by Wood as modified.
Wood as modified teaches a texture for the iris that includes a diffuse color map and a height map for the eyes (See Berard, Figure 14 demonstrates the height maps (left most image); Berard, Figure 5 demonstrates a control map: “Synthesizing an iris consists of capturing initial textures (a), from which control maps are generated by removing specular highlights (b)”. Notes: the iris geometry (left most images) are consistent with the definition of a height map, as they demonstrate the vertical changes through 3D representation of the iris. The control map is considered synonymous with diffuse color map. A diffuse color map, in its broadest reasonable interpretation, is a diffuse map when dealing with non-metallic materials; a diffuse map is an image where specular highlights pertaining to lighting are removed. Berard, Figure 5a demonstrates an initial image, in which an image Berard, Figure 5b is derived by removing specular highlights, therefore resulting in a diffuse map, and consequently a diffuse color map representative of the iris).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Wood (A 3D Morphable Eye Region Model for Gaze Estimation, 2016), in view of Ploumpis (Towards a complete 3D morphable model of the human head, 2019), Ciuc (Pub. No. US 2019/0279393 A1), Park (3D face econstruction from a single 2D face image, 2008) and Berard (Lightweight eye capture using a parametric model, 2016), and further in view of Benalcazar (A 3D iris Scanner From a Single Image Using Convolutional Neural Networks, 2020) and Paumard (Image Reassembly Combining Deep Learning and Shortest Path Problem, 2018).
Regarding Claim 8, the method of Claim 7 is rejected over Wood as modified.
Wood as modified teaches a rectangle that represents the iris transformed to polar coordinates (Section 6.1: “The structure of an iris is arranged radially around the pupil. Operating in polar coordinates (angle/radius) unwraps the radial structure (Fig. 5) and presents itself well for synthesis with rectangular patches”, where Figure 5 shows the iris modeled in polar coordinates, as well as the texture generated from the iris represented in polar coordinates).
However, Benalcazar teaches dividing the rectangle that represents the iris transformed to polar coordinates into slices (Section E: “We reconstructed the 3D rubber sheet from the 3D model in Figure 12c by obtaining one 2D slice every 1 [degree]”, where Figure 15 demonstrates the manner in which the slices are concatenated such that they result in an elongated rectangle: “3D Rubber Sheet obtained from 360 slices of the 3D model in Figure 12c. The iris image of the subject is shown on the bottom left corner along with the 0 [degree] line of the slicing process”. Notes: The method of representing the iris in polar coordinates is commonly referred to as the rubber sheet model. The 3D rubber sheet is a 3D version of the rectangle resulting from transforming a 3D representation of the iris to polar coordinates. The method of transformation is identical save for the iris being represented in the 2D or 3D space. Benalcazar demonstrates that in performing the transformation to polar coordinates for an iris, the result would be 360 slices (one for each degree); in other words, the slices are inherent in forming the rectangular representation of the iris by transforming the iris to polar coordinates).
Paumard teaches a machine learning model that represents an image based on the slices of the image (Conclusion: “In this paper, we tackled the image reassembly problem where given a unordered list of image fragments, we want to recover the original image. To that end, we proposed a deep neural network architecture that predicts the relative position of a given pair of fragments. Then, we cast the reassembly problem into a shortest path in a graph algorithm for which we propose several construction algorithms depending on whether the puzzle is complete or if there are missing pieces”, where the reconstruction is visualized in Figure 1; Notes: Considering Benalcazar teaches dividing the iris into segments as claimed in the invention, the task of representing the iris in its broadest reasonable interpretation includes representing the rectangular representation of the iris after transformation to polar coordinates. Therefore, given the slices of the rectangular representation of the iris, the method of Wood as modified can be utilized to reconstruct the rectangular representation of the iris).
Benalcazar explicitly teaches what Wood as modified implies, in that transforming an iris to polar coordinates, a rectangle representative of a slice of the iris can be derived, of which said slices can be sequentially joined together to form a single rectangle representative of the iris. Paumard teaches a machine learning model that can reconstruct an image; Therefore, Benalcazar and Paumard are considered analogous in the art of image reconstruction. One ordinarily skilled in the art would be motivated to automate the process of reconstructing an image of an iris using the slices derived from Wood as modified (implicitly) and Benalcazar, as doing so would eliminate the need to manually piece together the slices to form the rectangular representation of an iris image.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the transformation of an iris to slices of Benalcazar and Wood as modified with the image reconstruction machine learning model of Paumard; doing so would yield the predictable result of a representation of the iris as an output from the machine learning model of Paumard, with the slices derived as specified by Benalcazar and Wood as modified as in input into the machine learning of Paumard.
Claims 11-13 are rejected under 35 U.S.C. 103 as being unpatentable over Wood (A 3D Morphable Eye Region Model for Gaze Estimation, 2016), in view of Ploumpis (Towards a complete 3D morphable model of the human head, 2019), Ciuc (Pub. No. US 2019/0279393 A1) and Park (3D face econstruction from a single 2D face image, 2008), and further in view of Ablavatski (Pub. No. US 2021/0104096 A1).
Regarding Claim 11, the method of Claim 1 is rejected over Wood as modified.
Wood as modified teaches generating eye patch areas, and obtaining initial eye patch vertex locations based on the values of the parameters in the set of parameters of the parametric model of the eyes (Wood, Section 4: “the facial eye regions are represented as a combination of 3D shape s (n vertices) and 2D texture t (m texels), encoded as 3n and 3m dimensional vectors respectively”; Wood, Section 4: “Facial eye region shapes s and textures t can then be generated from shape and texture coefficients”, where the shape and texture coefficients are used to calculate the shape and texture in Wood, Equations 5 and 6, and the coefficients are described in the passage between Wood, Equation 4 and Equation 5 as being “the average 3D shape and 2D texture”, as well as “the Gaussian distributions of each shape and texture basis function”. Notes: Parameters are defined broadly within Wood as encompassing shape and texture. Considering shape and texture are defined by the above values, the above values are considered parameters);
wherein the initial eye patch vertex locations are subdivided into groups (Wood, Section 4, Morphable facial eye region model: “Additionally, we maintain correspondences for detailed parts, e.g. the interior eyelid margins, which are poorly defined for previous models” Notes: Woods explicitly states that the correspondence between the scan vertices and the simplified initial vertex set is maintained for the eyelid area; hence, the eyelids are considered to be a distinct subgroup within the initial vertex set; Wood, Section 4, Posing our multi-part model: “each eyelid vertex is rotated about the inter-eye-corner axis, with rotational amounts chosen to match measurements from an anatomical study. As our multi-part model contains disjoint parts, we also “shrinkrwap” the eyelid skin to the eyeball, projecting eyelid vertices onto the eyeball mesh to avoid gaps and clipping issues”; Notes: Wood clearly differentiates between the eyelid vertices and other eye patch area vertices, in that the eyelid vertices group has to conform to the eyeball vertices, while the eye patch area vertices group does not).
Wood as modified does not teach generating the eye patch areas based on optimizing the initial eye patch vertex locations using Catmull-Clark subdivision surface equations.
However, Ablavatski teaches generating a face model with optimized face vertex locations using Catmull-Clark subdivision surface equations on an initial set of face vertex locations (Paragraph [0032]: “The mesh representation 140 enables the building a plausible smooth surface representation 130 of the human face using algorithmic or machine leaning techniques. For example, Catmull-Clark subdivision can be used, resulting in the representation 150”, with corresponding Figures 1A and 1B).
While Ablavatski does not teach generating eye patch areas being generated from initial eye patch vertex locations being optimized using Catmull-Clark subdivision, it does demonstrate that using Catmull-Clark sub-division within the realm of 3D modeling of faces for smoothing is well established. There is common motivation within the art to smooth edges of a model to make it look more appealing and realistic to the user. With regards to a specific part of a head, such as the eye patch areas, one ordinarily skilled in the art would use Catmull-Clark subdivision to populate the 3D model of the eye patch areas with more vertices or otherwise optimize the vertices to smooth the 3D model of the eye patch areas and make them look more realistic. It is also worth noting that using Catmull-Clark sub-division for subdivision surface smoothing on any 3D mesh is well established to the point that 3D modeling software applications such as Blender incorporate it as a tool.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the initial eye patch vertex locations of Wood as modified with the use of Catmull-Clark subdivision to smooth a 3D model of a face of Ablavatski; doing so would yield the predictable result of generating eye patch areas that have optimized eye patch vertex locations, resulting in smoothed eye patch areas that are realistic and appealing to look at.
Regarding Claim 12, the method of Claim 11 is rejected over Wood as modified.
Wood as modified teaches a first group of initial eye patch vertex locations defining vertices of an eyelid and a second group of initial eye patch vertex locations defining vertices of an eyeball, and where in optimizing the initial eye patch vertex locations comprises constraining the vertices in the first group of initial eye patch vertex locations that defines vertices of the eyelid to approximate a curvature of the vertices in the second group of initial eye patch vertex locations that defines vertices of the eyeball (Wood, Section 4, Posing our multi-part model: “each eyelid vertex is rotated about the inter-eye-corner axis, with rotational amounts chosen to match measurements from an anatomical study. As our multi-part model contains disjoint parts, we also “shrinkrwap” the eyelid skin to the eyeball, projecting eyelid vertices onto the eyeball mesh to avoid gaps and clipping issues”; Notes: eyeball mesh by nature of being a mesh is a set of vertices defining the eyeball. Therefore, we have a group of vertices pertaining to the eyelid that are positioned such that they are constrained to the vertices of the eyeball, such that the eyelid vertices are projected onto the eyeball mesh vertices, which approximates a curvature of the vertices of the eyeball by nature of projection).
Regarding Claim 13, the method of Claim 1 is rejected over Wood as modified.
Wood as modified teaches subdividing vertices of an iris of an eye in the 3D model to obtain a dense sampling of vertices of the iris (Ablavatski, Paragraph [0032]: “The mesh representation 140 enables the building a plausible smooth surface representation 130 of the human face using algorithmic or machine leaning techniques. For example, Catmull-Clark subdivision can be used, resulting in the representation 150”, with corresponding Ablavatski, Figures 1A and 1B);
Computing refraction values for the iris based on the dense sampling of vertices (Wood, Section 4, Parametric eyeball model: “We model changes in the iris size geometrically, by scaling vertices on the iris boundary about the 3D iris centre as specified by iris diameter… This can be used to generate new iris textures… we avoid explicitly modelling this by computing refraction effects in texture-space”); and
rendering the iris based on applying a texture to vertices of the iris and the refraction values for the iris (Wood, Figure 6 demonstrates applying a texture to an eyeball 3D mesh, where a mesh is a collection of vertices; note that the mean texture is actually both the vertices and the 2d texture combined, as specified with the symbol
PNG
media_image1.png
1
1
media_image1.png
Greyscale
𝜇 representing “the average 3D shape and 2D texture” (Wood, Section 4, Morphable facial eye region model). Note that iris variations and refraction are captured in Wood, Figure 6 as well).
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Wood (A 3D Morphable Eye Region Model for Gaze Estimation, 2016), in view of Ploumpis (Towards a complete 3D morphable model of the human head, 2019), Ciuc (Pub. No. US 2019/0279393 A1), Park (3D face econstruction from a single 2D face image, 2008) and Ablavatski (Pub. No. US 2021/0104096 A1), and further in view of Wood.B (Learning an Appearance-Based Gaze Estimator from One Million Synthesised Images, 2016), Lyu (Differentiable Refraction-Tracing for Mesh Reconstruction of Transparent Objects, 2020) and Laine (Modular Primitives for High-Performance Differentiable Rendering, 2020).
Regarding Claim 21, the method of Claim 13 is rejected over Wood as modified
Wood as modified teaches subdividing an iris to generate multiple vertices (Ablavatski, Paragraph [0032]: “The mesh representation 140 enables the building a plausible smooth surface representation 130 of the human face using algorithmic or machine leaning techniques. For example, Catmull-Clark subdivision can be used, resulting in the representation 150”, with corresponding Ablavatski, Figures 1A and 1B. Notes: Refer to obviousness for applying Catmull-Clark subdivision for smoothing body components).
Wood as modified does not teach generating the texture using differentiable refraction by computing texture coordinates with gradients based on view rays interacting with the cornea that refract at each vertex of the multiple vertices of the subdivided vertices.
However, Wood.B teaches generating the texture with regards to a refraction at a cornea vertex with regards to view rays (Figure 3: “We model iris refraction by altering texture look-ups. In (a), a viewed pixel is refracted correctly to show black (pupil) instead of blue (geometry surface). Example renders with (top) and without (bottom) refraction are shown in (b)”; Section 3, Approximate eyeball model, Physically based Refraction; “In reality, the iris is a flat disk of muscle that appears distorted through the refractive corneal bulge. This phenomenon is particularly apparent when the eye is viewed at an angle (Figure 3b), so it was important to model it”; refer to Figure 3 for an illustration of calculating refraction).
Wood as modified and Wood.B are considered analogous in the art with respect to the modeling of the eye; specifically, Wood references Wood.B for eye modeling. A common motivation in modeling is to improve modeling detail, and would therefore utilize refraction modeling of eyes in Wood.B to improve modeling of the eyes of Wood.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the eye modeling method of Wood as modified with the refraction modeling of view rays passing through the cornea of an eye of Wood.B; Doing so would yield the predictable result of more accurately modeled eyes.
Wood as modified does not teach the use of differentiable refraction with gradients.
However, Lyu teaches performing differentiable refraction with respect to view rays using gradients (Figure 4 clearly illustrates a view ray passing through a curved surface, resulting in the refraction, and is differentiable refraction because the loss between Q and Q’, represented by Equation 2, is minimized; Section 4.1, Refraction Loss: “Q′, obtained by intersecting the ray with the background monitor, is also a function of the associated vertices. Since all the operations to obtain Q′ are differentiable, the gradient of Eq. (2) is easily calculated using the chain rule. The effect of the refraction loss is visualized in Fig. 5”).
Wood as modified and Lyu are considered analogous in the art with respect to modeling refraction with respect to view rays. A common motivation in the art is to use differentiable modeling to improve predictability and efficiency. One would be motivated to utilize differentiable refraction when seeking to decrease time and/or resources spent modeling refractive effects on a model.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the method of refraction modeling the eye using view rays of Wood as modified with the differentiable refraction method utilizing gradients of Lyu; Doing so would yield the predictable result of increased efficiency and predictability when modeling refractive effects in an eye model.
Wood as modified does not explicitly teach computing texture coordinates utilizing differentiable refraction.
However, Laine teaches computing texture coordinates with gradients through differentiable rendering (Section 3.4, Interpolation: “Generally, vertex attributes can be used for arbitrary purposes. One of their typical uses, however, is to provide 2D coordinates for texture mapping”; Refer to Figure 1, which clearly illustrates the use of gradients for calculating texture coordinates via differentiable rendering. Notes: Differentiable refraction is a form of differentiable rendering)
Wood as modified and Laine are considered analogous in the art with respect to the appearance of models with respect to light effects (texture effects). A common motivation in the art is to utilize differentiable rendering to efficiently model complex visual effects; this is apparent with differentiable refraction, which takes into account how a view ray may change direction when interacting with curved surfaces or materials with different refractive effects.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the method differentiable refraction of Wood as modified with the computation of texture coordinates with gradients through differentiable rendering of Laine; Doing so would yield the predictable result of generating textures with specific values at certain texture coordinates, providing a dynamic, realistic view of the model from different angles.
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Wood (A 3D Morphable Eye Region Model for Gaze Estimation, 2016), in view of Ploumpis (Towards a complete 3D morphable model of the human head, 2019), Ciuc (Pub. No. US 2019/0279393 A1) and Park (3D face econstruction from a single 2D face image, 2008), and in further view of Ascust (3DMM-Fitting-Pytorch, 2021) and Game Development Stack Exchange (Should a mesh consist of triangles or quads?, 2015).
Regarding Claim 22, the method of Claim 1 is rejected over Wood as modified.
Wood as modified teaches generating textures for the head, the eye patch areas, and the eyes by combining the head, the eye patch areas, and the eyes to generate a mesh model (Ploumpis, Introduction: “Therefore, we present a general approach that can be employed to combine 3DMMs from different parts of an object class into a single 3DMM. Due to their widespread use in the computer vision community, we fuse 3DMMs of the human face and the full human head as our exemplar. We add detailed models of the ears, eyes and eye regions to our head model, along with a basic model of the oral cavity, tongue and teeth”; Ploumpis, Figure 10: “Head texture completion given an unseen facial texture”, where Ploumpis, Figure 13 demonstrates the overall head texture for a model; Ploumpis, Figure 7: “The bank of iris textures in our model along with our eye mesh structure”; Ploumpis, Figure 9: “Illustration of the first five components of the eye region shape model”; Wood, Introduction, An Eye Region 3DMM: “We constructed a 3DMM of the facial eye region by carefully registering a set of high-quality 3D head scans, and extracting modes of shape and texture variation using PCA”. Notes: 3DMM models contain meshes (groups of vertices)).
Wood as modified does not teach generating a differentiable model that can be rasterized and shaded.
However, Ascust teaches that 3DMM models are differentiable (3DMM model fitting using Pytorch: “This is a fitting framework implemented in Pytorch for reconstructing the face in an image or a video using a 3DMM model. The framework only uses Pytorch modules and a differentiable renderer from pytorch3d. The whole module is differentiable and can be integrated into other systems for the gradient propagation”. Notes: note that Ploumpis teaches combining 3DMM models into a single 3DMM model. Furthermore, rasterization and shading are well known in the art as being achieved by differential rendering).
Wood as modified and Ascust are considered analogous in the art with respect to the use of 3DMM models. A common motivation in the art is to utilize differentiable 3D models for machine learning tasks; this is evident in Ascust, as a 3DMM undergoes differentiable rendering to integrate into videos.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the combination of multiple models and associated textures into a 3DMM mesh model of Wood as modified with the method of differentiable rendering of 3DMM models of Ascust; Doing so would yield the predictable result of enabling machine learning tasks utilizing gradient propagation via differentiable rendering.
Wood as modified does not teach generating a triangle mesh.
However, Game Development Stack Exchange teaches that generating triangle meshes is common and necessary in the art (user 1430: “However, modern graphics cards only work with triangles, so at some point the mesh data must be converted to triangles”).
Wood as modified and Game Development Stack Exchange are considered analogous in the art with respect to the use of mesh models. A common motivation for representing mesh models as triangle mesh models is that they work with modern graphics cards, as is pointed out in Game Development Stack Exchange.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the mesh model generation of Wood as modified with the conversion of mesh models to triangle mesh models of Game Development Stack Exchange; Doing so would yield the predictable result of allowing graphics cards to work with the mesh models.
Claim 23 is rejected under 35 U.S.C. 103 as being unpatentable over Wood (A 3D Morphable Eye Region Model for Gaze Estimation, 2016), in view of Ploumpis (Towards a complete 3D morphable model of the human head, 2019), Ciuc (Pub. No. US 2019/0279393 A1) and Park (3D face econstruction from a single 2D face image, 2008), and in further view of Stack Overflow (Way to scale all assets in a Unity3D scene, 2013).
Regarding Claim 23, the method of Claim 1 is rejected over Wood as modified.
Wood as modified teaches the method of Claim 1, wherein the eyes of each head mesh of the head meshes in the database are normalized to be spaced the fixed distance apart from one another (Ciuc, Paragraph [0065]: “In an implementation, a database of 3D models of human heads 108, such as the set of 35, is artificially generated using a modeler… In an implementation, the models 108 are scaled such that the interpupillary distance is 63 mm, which is the average distance for an adult person”; Ciuc, Paragraph [0064]: “In an implementation, the example features tracker 100 operates with averaged 3D models 108 of the head, with facial landmarks represented by feature points at average positions, on which is applied a set of transformations to match a 3D model 108 to the head of the subject face 102 in the current image 104”. Notes: The database of heads are derived from images containing faces, where each head model (head mesh) in the database is scaled such that the interpupillary distance, which describes the distance between the eyes, is fixed at 63 mm), and
Wherein performing the gradient descent on the object function is executed under a calibrated pinhole camera model (Section 4, 3D Eye Region Model, Camera Projection: Camera projection: “For a complete model of image formation, we also consider camera projection. We fix our axis-aligned camera at world origin, allowing us to set our world-to-view transform as the identity I_4. We assume knowledge of intrinsic camera calibration parameters K, and use these to build a full projection transform P. A local point in our model can then be transformed into image space using the model-view-projection transform”. Notes: the BRI of a pinhole camera model is a mathematical relationship between coordinates of a point in 3D space and its projection onto the image plane of a pinhole camera (which generally reads on a camera, where in non pinhole cameras, there may be distortion, thus making the pinhole camera model an approximation for non pinhole cameras depending on the level of distortion). Hence, the camera projection defined by Wood is analogous to a pinhole camera model since it maps 3d points in the model to image space via a projection transform that has camera calibration parameters K)
Wood as modified does not teach that the size of the head in the 3D model is set by applying an isotropic scale factor equal to a ratio between an interpupillary distance estimated for the one or more images and the fixed distance between the eyes.
However, Stack Overflow teaches the size of a 3D model is set by applying an isotropic scale factor equal to a ratio (Answer by Happy Apple: “Select any number of objects in the Hierarchy pane (using Ctrl or Shift for selection modifiers) and manipulate the Scale X/Y/Z in the Transform panel. That's if you wish to override the scale for everything you've selected, if you wish to scale them relative to how they currently are, use the Scaling tool (Press R over Scene view) and drag the scaling widget. You can scale all axis at the same time by dragging the scaling widget's central cube”).
Wood as modified and Stack Overflow are considered analogous in the art with respect to the scaling of 3D models. Isotropic scaling is well known in the art, and is commonly utilized to proportionately scale an entire model to a specific desired distance on the model, as is evident in Stack Overflow. While Stack Overflow does not teach that the isotropic scale factor is applied to a 3D head model, where the ratio defines a relationship between an interpupillary distance (IPD) estimated for a head in an image and the fixed distance between the eyes, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to apply an isotropic scaling factor equal to a ratio between an interpupillary distance estimated for the one or more images and the fixed distance between the eyes to set a size of the head in the 3D model; The motivation for scaling can be observed in Ciuc, which teaches scaling heads (which are initially captured through images) to a specific fixed IPD (where IPD is a measure describing the distance between the eyes).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the scaling of head meshes of Wood as modified with the isotropic scaling of 3D models of Stack Overflow; Doing so would yield the predictable result of scaling 3D head models proportionately in accordance with a ratio defined by an estimated interpupillary distance and a fixed distance between the eyes of the model.
Claim 24 is rejected under 35 U.S.C. 103 as being unpatentable over Wood (A 3D Morphable Eye Region Model for Gaze Estimation, 2016), in view of Ploumpis (Towards a complete 3D morphable model of the human head, 2019), Ciuc (Pub. No. US 2019/0279393 A1) and Park (3D face econstruction from a single 2D face image, 2008), and in further view of Dai (Statistical Modeling of Craniofacial Shape and Texture, 2019).
Regarding Claim 24, the method of Claim 1 is rejected over Wood as modified.
Wood as modified does not explicitly teach that generating the 3D model of the head is further based on using a head mesh PCA model trained on the database of head meshes, although it does indirectly teach so (Ploumpis, Section 2.1: “The most accurate craniofacial 3DMM of the human head both in terms of shape and texture, is the Liverpool-York Head model (LYHM)”; Ploumpis, Section 3.1, Regression modelling: “The LYHM is a PCA generative head model”).
However, Dai teaches generating the 3D model of the head is further based on using a head mesh PCA model trained on the database of head meshes (Section 4, Overview of the Headspace Dataset: “The Headspace dataset comprises 3D images of the human head for 1519 subjects. The data was collected and annotated by the Alder Hey Children’s Hospital (AHCH) Craniofacial Unit (Liverpool, UK), who employed 3dMD Ltd’s static 5-view 3dMDhead scanning system, using the five 3D camera configuration shown in Fig. 2. This dataset has been structured and made available online for research purposes, in a collaboration between AHCH and the Department of Computer Science, University of York. Access to the dataset is via the author’s Headspace web page… A typical output of this system rendered from different viewpoints, both with and without texture, is shown in Fig. 3. Vertex resolution is variable but typically there are around 180K vertices. All subjects are wearing tight fitting latex caps to reduce the effect of hairstyles. For subjects with relatively low-volume hairstyles, the shape of the cranium is clearly revealed. If this is not the case, we exclude them from the 3DMM training data, filtering on the basis of the hair bulge flag in the metadata”; Figure 1: “The proposed global Liverpool–York Head Model (LYHM) trained using 1212 subjects (606 male, 606 female) from the Headspace dataset”).
Wood as modified and Dai are considered analogous in the art with respect to the use of a PCA generative head model. A common motivation in the art is to train PCA generative head models off of a database of head meshes, as is implicit in Wood as modified and evident in Dai.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the PCA generative head model of Wood as modified with the PCA generative head model trained off of a database of head meshes of Dai; Doing so would yield the predictable result of a PCA generative head model trained off of head meshes.
Response to Arguments
Applicant’s arguments, see Applicant Remarks pages 9-10, filed July 7th, 2026, with respect to the rejection(s) of claim(s) 1, 14 and 18 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Wood (A 3D Morphable Eye Region Model for Gaze Estimation, 2016), in view of Ploumpis (Towards a complete 3D morphable model of the human head, 2019), Ciuc (Pub. No. US 2019/0279393 A1) and Park (3D face econstruction from a single 2D face image, 2008).
The Applicant asserts that Wood, Ploumpis, and Ciuc fail to teach “generating eye patch areas of areas surrounding the eyes based on the values of the parameters in the set of parameters of the parametric model for the eyes, wherein the eye patch areas are represented by principle component analysis (PCA) coefficients obtained based on a database of head meshes, each head mesh in the database including eyes that are normalized to be spaced a fixed distance apart from one another”.
However, Wood in combination with Ploumpis and Ciuc, teach the aforementioned limitation (Wood, Section 1, An Eye Region 3DMM: “We constructed a 3DMM of the facial eye region by carefully registering a set of high-quality 3D head scans, and extracting modes of shape and texture variation using PCA”, where Wood, Figure 4 and Wood, Figure 5 visualize the eye patch area model); Figure 4: “We re-parameterize high-resolution 3D head scan data (left)”; Refer to Wood, Figure 4, which shows a head model (mesh) with eyes; Wood, Section 4, 3D Eye Region Model, Morphable facial eye region model: “We started by acquiring 22 high-quality head scans as source data”; Wood, Section 1, An Eye Region 3DMM: “We constructed a 3DMM of the facial eye region by carefully registering a set of high-quality 3D head scans, and extracting modes of shape and texture variation using PCA”; Wood, Section 4, 3D Eye Region Model, Morphable facial eye region model: “We then performed Principle Component Analysis (PCA) on our set of c ordered scans to extract orthogonal shape and texture basis functions: U … and V… Facial eye region shapes s and textures t can then be generated from shape (
β
f
a
c
e
)… and texture coefficients (
τ
f
a
c
e
)”; Wood, Equation 5 and Wood, Equation 6 define the coefficients as being coefficients as being PCA coefficients, since U and V are derived via PCA. Notes: The broadest reasonable interpretation of a mesh is a collection of vertices defining a 3D model, as is apparent in Wood, Figure 4; Paragraph [0065]: “In an implementation, a database of 3D models of human heads 108, such as the set of 35, is artificially generated using a modeler… In an implementation, the models 108 are scaled such that the interpupillary distance is 63 mm, which is the average distance for an adult person”; Ciuc, Paragraph [0064]: “In an implementation, the example features tracker 100 operates with averaged 3D models 108 of the head, with facial landmarks represented by feature points at average positions, on which is applied a set of transformations to match a 3D model 108 to the head of the subject face 102 in the current image 104”. Notes: The database of heads are derived from images containing faces, where each head in the database is scaled such that the interpupillary distance, which describes the distance between the eyes, is fixed at 63 mm). For more details regarding the combination of references, refer to the rejection of Claim 1.
The Applicant asserts that Wood, in combination with Ploumpis and Ciuc, fail to teach “obtaining initial values of the PCA coefficients for the eye patch areas by interpolating 3D vertex positions at barycentric coordinates corresponding to two-dimensional (2D) face landmarks detected in the one or more images”. The Examiner agrees with the Applicant’s assertion.
The Examiner bases a new rejection on new grounds over Wood, in view of Poumpis, Ciuc, and Park. Park teaches portions of the aforementioned limitation (Park, Section 4, 3D Face Reconstruction: “In this chapter, we propose a way to reconstruct a 3D face from feature points and pose estimation by interpolating more points in an input 2D image and estimating the corresponding vertices in an output 3D face. After feature detection and pose estimation by the proposed method, we obtain the 3D positions of the 79 feature points, but they are not enough to reconstruct a whole 3D face. For example, when we model a 3D face based on the 3D geometrical structure of the USF Human-ID database, it is required to obtain both shape and texture of 75,972 vertices. As the first step to get this information, we interpolate 75,893 points in an input 2D image by a simple way using barycentric coordinates. Next, we estimate the 3D vertices with the depths in a 3D face by a linear deformable model; Refer to Park, Figure 3 for a visualization” Park, Section 4, 3D Face Reconstruction, Interpolating 2D points in a Frontal View: “Fortunately, the difference between the shapes of a specific face and the average face is not significant. Inspired by this idea, first of all for interpolation, pose correction of the 2D feature points to the frontal view is applied by the estimated pose parameters. After pose correction to the frontal view, the transformed feature points can be easily compared to the corresponding points in the average shape obtained from a training set consisting of the 3D frontal faces. The process for interpolation is shown in Figure 3”. Notes: The broadest reasonable interpretation of 2D landmark points are feature points that describe structure). While Park does not explicitly state using the obtained values for initial PCA values, a person having ordinary skill in the art would appreciate that Wood uses shape parameters (defined by 3D vertices, which are derived in Park) to define the PCA coefficients (Refer to Wood, Equation 5 and Wood, Equation 6), where given a set of vertices that define shape (such as those of Park), Wood, Equation 5 or Wood, Equation 6 can be used to determine initial PCA coefficient values by interpolating 3D vertex positions. Refer to the rejection of Claim 1 for more details regarding obviousness and reason for combination.
Applicant asserts that Wood, in combination with Ploumpis and Ciuc, fail to teach optimizing the initial values of the PCA coefficients to obtain optimized PCA coefficients by performing gradient descent on an objective function that directly minimizes, in an image space, distances between the 2D face landmarks and projections of 3D vertices of the head and the eye patch areas”.
However, Wood in combination with Ploumpis and Ciuc teaches the aforementioned limitation (Wood, Section 5, Analysis-by-synthesis for Gaze Estimation: We cast this as an unconstrained energy minimization problem for unknown ϕ”; Refer to Wood, Equation 9, which describes the minimization problem as an optimization through minimizing the energy objection function E(ϕ); Wood, Equation 10 defines the Energy function as being the sum between the Energy of the image and the Energy of the landmarks given parameters ϕ; Wood, Section 5, Analysis-by-synthesis for Gaze Estimation, Landmark similarity metric: “The face contains important landmark feature points that can be localized reliably [13]. These can be used to efficiently consider the appearance of the whole face, as well as the local appearance of the eye region. We use a state-of-the-art face tracker [15] to localize 14 landmarks L around the eye region in image-space (see Fig. 8). For each landmark
l
∈
L
we compute a corresponding synthesized landmark
l
'
using our 3DMM. The sparse landmark-similarity term is calculated as the distance between both sets of landmarks, normalized by the foreground area to avoid bias from image or eye region size. This acts as a regularizer to prevent our pose from drifting too far from a reliable estimate”; Refer to Wood, Equation 12 regarding the distance difference between the synthetic landmark derived from the 3DMM (projected 3D vertices) and the 2D landmark. Wood, Section 1, Analysis-by-Synthesis: “We iteratively fit our model using gradient descent with numerical derivatives efficiently calculated with a tailored GPU rasterizer”; Wood, Figure 7 also demonstrates how the model appears after each iteration, where each iteration visualizes a model in which loss is minimized via gradient descent. Notes: The broadest reasonable interpretation of applying a gradient descent algorithm to raw data from an image is utilizing gradient descent as a means for minimizing loss between the output of a generation model and a target image, where the raw data includes image data, which is subsequently used for image processing).
Dependent Claims of 1, 14 and 18 are not considered allowable based on dependence to Independent Claims 1, 14 and 18.
Claim 23 is rejected over Wood (A 3D Morphable Eye Region Model for Gaze Estimation, 2016), in view of Ploumpis (Towards a complete 3D morphable model of the human head, 2019), Ciuc (Pub. No. US 2019/0279393 A1) and Park (3D face econstruction from a single 2D face image, 2008), and in further view of Stack Overflow (Way to scale all assets in a Unity3D scene, 2013). Refer to the Rejection of Claim 23 for more detail and reasons for combination.
Claim 24 is rejected over Wood (A 3D Morphable Eye Region Model for Gaze Estimation, 2016), in view of Ploumpis (Towards a complete 3D morphable model of the human head, 2019), Ciuc (Pub. No. US 2019/0279393 A1) and Park (3D face econstruction from a single 2D face image, 2008), and in further view of Dai (Statistical Modeling of Craniofacial Shape and Texture, 2019). Refer to the Rejection of Claim 24 for more detail and reasons for combination.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RAYMOND CHUN LAM LI whose telephone number is (571)272-5124. The examiner can normally be reached M-F 8:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at 571-272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RAYMOND CHUN LAM LI/Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights