Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Amendment
This is in response to applicant’s amendment/response filed on 03/03/2026, which has been entered and made of record. Claims 1, 17 and 20 have been amended. No claim has been cancelled. No claim has been added. Claims 1-20 are pending in the application.
Response to Arguments
Applicant’s arguments on 03/03/2026 have been fully considered but are moot because the arguments do not apply to any of the references being used in the current rejection.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 1-7, 14-20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang, Longwen, et al. ("Dreamface: Progressive generation of animatable 3d faces under text guidance." arXiv preprint arXiv:2304.03117 (2023).) in view of Bradley et al. (US Pub 2024/0249459 A1).
As to claim 1, Zhang discloses a system comprising: at least one processor; and at least one memory component storing instructions that, when executed by the at least one processor (Zhang, Page 14, “We train the texture LDM on two Nvidia A6000 GPUs”” enabling the generation of a high-quality facial asset within 5 minutes on a single Nvidia A6000 GPU.”), cause the at least one processor to perform operations comprising:
receiving a prompt from a developer (Zhang, Page 2, “a progressive scheme to generate personalized 3D faces with text guidance. As shown in Fig. 1, with only prompt controls, DreamFace enables layman users to customize 3D facial assets with the desired shape and physically based textures, as well as empowered animation capabilities.”);
accessing a default head mesh rigged to facial features of a default head (Zhang, Page 3, “We adopt the U-Net architecture to transform the neutral asset (in terms of geometry images) into the generated facial mesh under various expressions.” “DreamFace applies the explicit face mesh with the same geometric topology as ICT-FaceKit [Li et al. 2020a] to utilize displacement and tangent space normal maps to achieve fine-grained rendering.”);
modifying the default head mesh by (Zhang, Page 6, “We use the geometry corresponding to the best match as our coarse head mesh.” Equation (1), “T is the mean face, S𝑖 is the shape components, and T are corresponding generated head mesh.” “Recall that the coarse mesh is selected from a set of randomly generated geometries using parametric representations. The selected coarse mesh is over-smoothing and deviates from the text prompt.”):
inputting the prompt into a stable diffusion model (Zhang, Page 6, “We rely on the Score Distillation Sampling (SDS) loss of the pretrained generic LDM, i.e., Stable Diffusion [Rombach et al. 2022], for guiding the details carving. We also use the pre-trained LDM image autoencoder from Stable Diffusion with E as encoder and D as decoder, which converts a 512×512 image to 64×64 latent code and back.” “we use the CLIP model [Radford et al. 2021] to extract features from the images and select the one that best matches the input user prompt.”);
retrieving gradients from the stable diffusion model (Zhang, Page 6, “gradient of the generic LDM with rendered faces as input can be passed using the Score Distillation Sampling technique for learning V𝑑 and N𝑑.”); and
adjusting a plurality of vertices on the default head mesh according to the gradients (Zhang, Page 6, “To increase realism and make the geometry more closely match the input prompt, we develop a detail carving process. We model the face details using two components, i.e., the vertex displacement V𝑑 and geometric detailed tangent space normal map N𝑑, on top of the coarse mesh T∗. The vertex displacement imposes geometric correction to T∗, while the geometric detailed tangent space normal map provides more details during rendering, such as deep wrinkles. As illustrated in Fig. 3, we use the generic LDM, i.e., Stable Diffusion, to add facial details to the coarse geometry with prompt guidance. Specifically, the vertex displacement and tangent space normal map takes effect during the face rendering process, and hence gradient of the generic LDM with rendered faces as input can be passed using the Score Distillation Sampling technique for learning V𝑑 and N𝑑.”);
accessing a camera feed from a camera system of a user, the camera feed including a head of the user (Zhang, Page 4, “Some face-tracking techniques [Apple 2023; Feng et al. 2021; Fyffe et al. 2015; Somepalli et al. 2021] can provide a lightweight capture solution for facial animation with a single RGB camera input.” Page 8, “We collect the UV texture dataset from multiple sources, including our captured facial scans using the multi-view photometric capture system”); and
applying a first content augmentation corresponding to the modified head mesh to the head of the user in the camera feed, wherein the first content augmentation overlays one or more digital elements onto a live feed of the camera feed, wherein one or more digital elements include at least one of: an image or an animation (Zhang, Page 4, “Facial Animation. Unlike implicit representation, explicit representation supports generating expressions directly by generic expression blendshapes. Some face-tracking techniques [Apple 2023;Feng et al. 2021; Fyffe et al. 2015; Somepalli et al. 2021] can providea lightweight capture solution for facial animation with a single RGB camera input.” “enables video-driven animation from VR Headset cameras. To allow for cross-identity retargeting, Moser et al. [2021] applied image-to image translation and extracted a common representation between the input video and rendered CG sequence to predict blendshape weights.” Page 10, “our framework also empowers the animatability of the generated facial asset” Fig. 8. Fig. 10. Page 12, Fig. 9, “Through our animatability empowerment, the generated facial assets can be animated using a single RGB image and rendered photo-realistically in modern CGpipelines.”).
Zhang does not explicitly disclose wherein the gradients include individual rates of change for corresponding vertices of the default head mesh, the adjusting of the plurality of vertices based on the individual rates of change.
Bradley discloses the gradients include individual rates of change for corresponding vertices of the default head mesh, the adjusting of the plurality of vertices based on the individual rates of change (Bradley, ¶0083, “The training engine 122 can use gradient descent and backpropagation to update neural network weights or other parameters of the control networks 254 and/or the material to canonical space mapping networks 212 in a way that reduces the measure of error. For example, the training engine 122 can calculate the gradient accumulated on the vertices of the soft body 230 and back propagate the gradient to the control networks 254. The back propagation to the control networks 254 updates parameters of the control networks 254.” ¶0084, “The training engine 122 trains the actuation network 256 and/or the bone network 258 based on the facial geometry loss 286 as described above with respect to FIG. 2A by propagating the gradient accumulated on the vertices of the actuated simulation mesh of the simulated active soft body 284 to the identity-specific control values 270 and then to the canonical control values 260. The gradient accumulated on the vertices can be propagated to the identity-specific control values 270 by multiplying the gradient accumulated on the vertices by a sensitivity matrix.”).
Zhang and Bradley are considered to be analogous art because all pertain to latent diffusion model. It would have been obvious before the effective filing date of the claimed invention to have modified Zhang with the features of “the gradients include individual rates of change for corresponding vertices of the default head mesh, the adjusting of the plurality of vertices based on the individual rates of change” as taught by Bradley. The suggestion/motivation would have been in order to enable the use of gradient-based methods for solving complex control problems (Bradley, ¶0048). All the claimed elements were known in the prior art and one skilled in the art could have combined the elements as claimed by known methods with no change in their respective functions, and the combination would have yielded predictable results to one of ordinary skill in the art at the time of the invention.
As to claim 2, claim 1 is incorporated and the combination of Zhang and Bradley discloses the operations further comprise training the stable diffusion model to generate images based on inputted prompts (Zhang, Fig. 6, Page 9, “The overview of our Texture LDM training pipeline.”), wherein during the generation of an image based on the inputted prompt, the stable diffusion model generates the gradients (Zhang, Page 2, “we learn the detailed displacements and normal maps using the Score Distillation Sampling technique (SDS) to compute gradients from the generic Latent Diffusion Model (LDM), i.e., Stable Diffsuion [Rombach et al. 2022].”).
As to claim 3, claim 2 is incorporated and the combination of Zhang and Bradley discloses the gradients including encoded information that includes instructions on changes to the vertices of the default head mesh is to be adjusted to correspond to the inputted prompt (Zhang, Page 6, “The selected coarse mesh is over-smoothing and deviates from the text prompt. To increase realism and make the geometry more closely match the input prompt, we develop a detail carving process. We model the face details using two components, i.e., the vertex displacement V𝑑 and geometric detailed tangent space normal map N𝑑, on top of the coarse mesh T∗. The vertex displacement imposes geometric correction to T∗, while the geometric detailed tangent space normal map provides more details during rendering, such as deep wrinkles. As illustrated in Fig. 3, we use the generic LDM, i.e., Stable Diffusion, to add facial details to the coarse geometry with prompt guidance. Specifically, the vertex displacement and tangent space normal map takes effect during the face rendering process, and hence gradient of the generic LDM with rendered faces as input can be passed using the Score Distillation Sampling technique for learning V𝑑 and N𝑑.”).
As to claim 4, claim 2 is incorporated and the combination of Zhang and Bradley discloses the gradients include individual rates of change for corresponding vertices of the default head mesh (Zhang, Page 6, “The selected coarse mesh is over-smoothing and deviates from the text prompt. To increase realism and make the geometry more closely match the input prompt, we develop a detail carving process. We model the face details using two components, i.e., the vertex displacement V𝑑 and geometric detailed tangent space normal map N𝑑, on top of the coarse mesh T∗. The vertex displacement imposes geometric correction to T∗, while the geometric detailed tangent space normal map provides more details during rendering, such as deep wrinkles. As illustrated in Fig. 3, we use the generic LDM, i.e., Stable Diffusion, to add facial details to the coarse geometry with prompt guidance. Specifically, the vertex displacement and tangent space normal map takes effect during the face rendering process, and hence gradient of the generic LDM with rendered faces as input can be passed using the Score Distillation Sampling technique for learning V𝑑 and N𝑑.” “where Laplacian(·) represents the Laplacian smooth loss between two meshes, and L𝑚𝑎𝑝 regularizes both the gradient and divergence of the detailed normal map for smoothing.”).
As to claim 5, claim 1 is incorporated and the combination of Zhang and Bradley discloses the operations further comprise: comparing the modified default head mesh with a rendered mesh to identify a loss, the rendered mesh representing a desired mesh outcome based on the inputted prompt (Zhang, Page 6, “We rely on the Score Distillation Sampling (SDS) loss of the pretrained generic LDM, i.e., Stable Diffusion [Rombach et al. 2022], for guiding the details carving.); and further modifying the modified default head mesh causing a reduction in the loss (Page 6, equation (6), “We can write the SDS loss of generic LDM on rendered image I as follows: where 𝑥𝑑 = [V𝑑,N𝑑,𝜷∗] are the optimizing parameters” Equation (7), “Besides the SDS loss, we further add regularization terms to ensure the rationality of generated details. The additional regularization losses include, where Laplacian(·) represents the Laplacian smooth loss between two meshes, and L𝑚𝑎𝑝 regularizes both the gradient and divergence of the detailed normal map for smoothing.” Also see Page 9, Latent space SDS. Page 10, image space SDS.).
As to claim 6, claim 1 is incorporated and the combination of Zhang and Bradley discloses inputting the prompt into the stable diffusion model causes the stable diffusion model to (1) identify relevant features from a dataset of a plurality of head meshes and corresponding prompts (Zhang, Fig. 3, “Geometry generation pipeline. Given the input prompt, we utilize the CLIP model to select the coarse geometry candidates with the highest matching score.” Page 6, “we use the CLIP model [Radford et al. 2021] to extract features from the images and select the one that best matches the input user prompt.”)
and (2) generate the gradients based on the identified relevant features (Zhang, Page 6, “gradient of the generic LDM with rendered faces as input can be passed using the Score Distillation Sampling technique for learning V𝑑 and N𝑑.” “Laplacian(·) represents the Laplacian smooth loss between twomeshes, and L𝑚𝑎𝑝 regularizes boththegradient and divergence of the detailed normal map for smoothing.”).
As to claim 7, claim 1 is incorporated and the combination of Zhang and Bradley discloses subsequent to modifying the default head mesh, generating an application configured to be used by a plurality of users to apply the modified head mesh to corresponding users of the system via camera feeds from corresponding user devices (Zhang, Page 1, “It enables layman users to naturally customize 3D facial assets that are compatible with CG pipelines, with desired shapes, textures and fine-grained animation capabilities.” Page 11, “one could utilize existing facial performance capture techniques [Apple 2023; Li et al. 2017; Moser et al. 2021] to obtain the corresponding expression blendshape parameters from images and videos to animate our generated facial asset.” “produce personalized expressions while preserving the unique characteristics of our generated facial asset and creating more realistic and believable animations.”);
and receiving a selection of the application by the user, wherein accessing the camera feed is in response to receiving the selection of the application (Zhang, Page 12, Fig. 9, “Through our animatability empowerment, the generated facial assets can be animated using a single RGB image and rendered photo-realistically in modern CGpipelines.” Page 18, “We believe that our approach renews the generation of accessible 3D facial assets with physically-based rendering quality and rich animation ability in the neural and prompt-interaction era. It not only benefits the CG production industry but also incentivizes numerous innovative applications for VR/AR and the emerging Metaverse.”).
As to claim 14, claim 1 is incorporated and the combination of Zhang and Bradley discloses the operations further comprise: accessing a default body mesh rigged to features of a default body head (Zhang, Page 3, “We adopt the U-Net architecture to transform the neutral asset (in terms of geometry images) into the generated facial mesh under various expressions.” “DreamFace applies the explicit face mesh with the same geometric topology as ICT-FaceKit [Li et al. 2020a] to utilize displacement and tangent space normal maps to achieve fine-grained rendering.”);
modifying the default body mesh by: inputting the prompt into a stable diffusion model; retrieving gradients from the stable diffusion model (Zhang, Page 6, “We rely on the Score Distillation Sampling (SDS) loss of the pretrained generic LDM, i.e., Stable Diffusion [Rombach et al. 2022], for guiding the details carving. We also use the pre-trained LDM image autoencoder from Stable Diffusion with E as encoder and D as decoder, which converts a 512×512 image to 64×64 latent code and back.” “we use the CLIP model [Radford et al. 2021] to extract features from the images and select the one that best matches the input user prompt.”); and adjusting a plurality of vertices on the default body mesh according to the gradients (Zhang, Page 6, “To increase realism and make the geometry more closely match the input prompt, we develop a detail carving process. We model the face details using two components, i.e., the vertex displacement V𝑑 and geometric detailed tangent space normal map N𝑑, on top of the coarse mesh T∗. The vertex displacement imposes geometric correction to T∗, while the geometric detailed tangent space normal map provides more details during rendering, such as deep wrinkles. As illustrated in Fig. 3, we use the generic LDM, i.e., Stable Diffusion, to add facial details to the coarse geometry with prompt guidance. Specifically, the vertex displacement and tangent space normal map takes effect during the face rendering process, and hence gradient of the generic LDM with rendered faces as input can be passed using the Score Distillation Sampling technique for learning V𝑑 and N𝑑.”); accessing a camera feed from a camera system of a user, the camera feed including a body of the user; and applying the first content augmentation that also corresponds to the modified body mesh to the body of the user in the camera feed (Zhang, Page 4, “Some face-tracking techniques [Apple 2023; Feng et al. 2021; Fyffe et al. 2015; Somepalli et al. 2021] can provide a lightweight capture solution for facial animation with a single RGB camera input.” Page 8, “We collect the UV texture dataset from multiple sources, including our captured facial scans using the multi-view photometric capture system” A user’s head is the uppermost part of the body.).
As to claim 15, claim 1 is incorporated and the combination of Zhang and Bradley discloses wherein the operations further comprise: displaying a selectable user interface element (Zhang, Fig. 1, “DreamFace generates personalized 3D physically-based facial assets under text guidance, which are compatible with the existing CG pipeline, with desired shapes, textures, and fine-grained animations for realistic rendering. Go to DreamFace project page https://sites.google.com/view/dreamface, watch our video at https://youtu.be/yCuvzgGMvPM and experience DreamFace online at https://hyperhuman.top !”); and in response to a user selection of the selectable user interface element, capturing a picture or video of the camera feed with the applied first content augmentation (Zhang, Fig. 9, Page 12).
As to claim 16, claim 1 is incorporated and the combination of Zhang and Bradley discloses the first content augmentation augments, modifies, or overlays content from the camera feed with one or more digital elements, wherein one or more digital elements include at least one of: an image, an animation, or audio (Zhang, Fig. 9, Page 11, “Beyond generic animatability using blendshapes, we explore introducing an enhanced animation scheme to produce personalized expressions while preserving the unique characteristics of our generated facial asset and creating more realistic and believable animations.” Page 12, “Video-driven animation.”).
As to claim 17, the combination of Zhang and Bradley discloses a method comprising: receiving a prompt from a developer; accessing a default head mesh rigged to facial features of a default head; modifying the default head mesh by: inputting the prompt into a stable diffusion model; retrieving gradients from the stable diffusion model; and adjusting a plurality of vertices on the default head mesh according to the gradients, wherein the gradients include individual rates of change for corresponding vertices of the default head mesh, the adjusting of the plurality of vertices based on the individual rates of change; accessing a camera feed from a camera system of a user, the camera feed including a head of the user; and applying a first content augmentation corresponding to the modified head mesh to the head of the user in the camera feed, wherein the first content augmentation overlays one or more digital elements onto a live feed of the camera feed, wherein one or more digital elements include at least one of: an image or an animation. (See claim 1 for detailed analysis.).
As to claim 18, claim 17 is incorporated and the combination of Zhang and Bradley discloses training the stable diffusion model to generate images based on inputted prompts, wherein during the generation of an image based on the inputted prompt, the stable diffusion model generates the gradients (See claim 2 for detailed analysis.).
As to claim 19, claim 17 is incorporated and the combination of Zhang and Bradley discloses comparing the modified default head mesh with a rendered mesh to identify a loss, the rendered mesh representing a desired mesh outcome based on the prompt; and further modifying the modified default head mesh causing a reduction in the loss (See claim 5 for detailed analysis.).
As to claim 20, the combination of Zhang and Bradley discloses a non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: receiving a prompt from a developer; accessing a default head mesh rigged to facial features of a default head; modifying the default head mesh by: inputting the prompt into a stable diffusion model; retrieving gradients from the stable diffusion model; and adjusting a plurality of vertices on the default head mesh according to the gradients, wherein the gradients include individual rates of change for corresponding vertices of the default head mesh, the adjusting of the plurality of vertices based on the individual rates of change; accessing a camera feed from a camera system of a user, the camera feed including a head of the user; and applying a first content augmentation corresponding to the modified head mesh to the head of the user in the camera feed, wherein the first content augmentation overlays one or more digital elements onto a live feed of the camera feed, wherein one or more digital elements include at least one of: an image or an animation. (See claim 1 for detailed analysis.).
Claim 8-13 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang, Longwen, et al. ("Dreamface: Progressive generation of animatable 3d faces under text guidance." arXiv preprint arXiv:2304.03117 (2023).) in view of Moser, Lucio, et al. ("Semi-supervised video-driven facial animation transfer for production." ACM Transactions on Graphics (TOG) 40.6 (2021): 1-18.).
As to claim 8, claim 1 is incorporated and Zhang discloses the operations further comprise:
assessing the camera feed to identify facial features of the head of the user (Zhang, Fig. 9. Generated facial assets from descriptions. Our approach generates facial assets that faithfully match the characteristics described in the prompts. Through our animatability empowerment, the generated facial assets can be animated using a single RGB image and rendered photo-realistically in modern CGpipelines.);
identifying locations of the facial features relative to the head (Zhang, Page 11, “The geometry expression encoder encodes different facial expressions into a unified expression latent code, while the geometry generator uses this code, along with the neutral geometry of a specific identity, to generate the desired facial geometry with corresponding expressions.”); and
customizing the default head mesh such that the location of the facial features of the default head mesh correspond to the location of the facial features of the head of the user, the first content augmentation comprising applying the customized head mesh to the head of the user in the camera feed (Zhang, Fig. 9. Generated facial assets from descriptions. Our approach generates facial assets that faithfully match the characteristics described in the prompts. Through our animatability empowerment, the generated facial assets can be animated using a single RGB image and rendered photo-realistically in modern CGpipelines. Page 12, “Once trained, the geometry generator Ggeo is capable of producing geometries with the desired expressions for a specific identity using the unified expression latent code.” “. After training, Eexp is able to extract expression latent code from in-the-wild videos, and Ggeo could further produce personalized animations for our generated facial asset.”);
Zhang teaches one could utilize existing facial performance capture techniques [Apple 2023; Li et al. 2017; Moser et al. 2021] to obtain the corresponding expression blendshape parameters from images and videos to animate our generated facial asset (Page 11). Therefore, more details relate to blendshape are in Moser et al.
Moser teaches customizing the default head mesh such that the location of the facial features of the default head mesh correspond to the location of the facial features of the head of the user, the first content augmentation comprising applying the customized head mesh to the head of the user in the camera feed (Moser, Page 3, “our approach that relies on a PCA blendshape model that captures full facial motions.” Fig. 2. “works with a single camera and is capable of transferring the expressions of multiple users to arbitrary characters.”).
Zhang and Moser are considered to be analogous art because all pertain to automatic transfer of facial expressions. It would have been obvious before the effective filing date of the claimed invention to have modified Zhang with the features of “customizing the default head mesh such that the location of the facial features of the default head mesh correspond to the location of the facial features of the head of the user, the first content augmentation comprising applying the customized head mesh to the head of the user in the camera feed” as taught by Moser. The suggestion/motivation would have been Zhang explicitly incorporated the teaching of Moser in Zhang’s paper. All the claimed elements were known in the prior art and one skilled in the art could have combined the elements as claimed by known methods with no change in their respective functions, and the combination would have yielded predictable results to one of ordinary skill in the art at the time of the invention.
As to claim 9, claim 8 is incorporated and the combination of Zhang and Moser discloses customizing the default head mesh includes deforming the default head mesh by adjusting weights of predefined blend shapes to match the location of the facial features of the head of the user (Zhang, Page 2, “Our generated neutral assets naturally support blendshapes-based facial animations, thanks to the unified geometric topology. We further improve the animation ability with personalized deformation characteristics.” “DreamFace can generate realistic 3D facial assets with physically-based rendering quality and rich animation ability from video footage, even for fashion icons or exotic characters in cartoons and fiction movies.” Page 3, “For animating the generated neutral geometry and appearance, the brute-force approach would adopt the default blendshapes provided by the parametric model ICT-FaceKit [Li et al. 2020a]” Page 4, “To allow for cross-identity retargeting, Moser et al. [2021] applied image-to-image translation and extracted a common representation between the input video and rendered CG sequence to predict blendshape weights.” Page 11, “blendshape-based animation due to its consistent geometric topology, one could utilize existing facial performance capture techniques [Apple 2023; Li et al. 2017; Moser et al. 2021] to obtain the corresponding expression blendshape parameters from images and videos to animate our generated facial asset.”).
As to claim 10, claim 9 is incorporated and the combination of Zhang and Moser discloses the weights are further adjusted to control a magnitude of intensity for a corresponding blend shape (Zhang, Page 4, “To allow for cross-identity retargeting, Moser et al. [2021] applied image-toimage translation and extracted a common representation between the input video and rendered CG sequence to predict blendshape weights.” Page 17, “To adapt FLAME [Li et al. 2017] blendshapes to our asset, we transfer the expression parameters (blendshape weights and jaw rotation) to control our generated facial asset with the rig that same as FLAME.” See Moser for control a magnitude of intensity for blendshape. Fig. 5, “Pre-computed blendshapes multiplied by the weights recover the animated geometry”).
As to claim 11, claim 9 is incorporated and the combination of Zhang and Moser discloses the operations further comprise: assessing the camera feed to identify a facial expression of the head of the user; and further customizing the default head mesh such that the facial expression of the default head mesh corresponds to the facial expression of the head of the user (Zhang, Page 11, “one could utilize existing facial performance capture techniques [Apple 2023; Li et al. 2017; Moser et al. 2021] to obtain the corresponding expression blendshape parameters from images and videos to animate our generated facial asset.” See Moser et al. 2021 for more details.).
As to claim 12, claim 8 is incorporated and the combination of Zhang and Moser discloses customizing the default head mesh includes adjusting vectors of the default head mesh, the vectors determined based on a difference between a location of a facial feature in the camera feed to a location of a corresponding facial feature in the default head mesh (Zhang, Page 4, “Facial Animation. Unlike implicit representation, explicit representation supports generating expressions directly by generic expression blendshapes. Some face-tracking techniques [Apple 2023; Feng et al. 2021; Fyffe et al. 2015; Somepalli et al. 2021] can provide a lightweight capture solution for facial animation with a single RGB camera input. To further enhance the inaccurate result of generic facial animation, many research efforts made huge progress. Laine et al. [2017] trained a characteristic neural network with registered 4D scans. Lombardi et al. [2018] encodes geometry and view-dependent texture information with a VAE framework that enables video-driven animation from VR Headset cameras. To allow for cross-identity retargeting, Moser et al. [2021] applied image-toimage translation and extracted a common representation between the input video and rendered CG sequence to predict blendshape weights. The state-of-the-art neural animation method [Zhang et al. 2022] proposed a production-level animation pipeline that generates high-quality facial geometry for person-specific performance. A very recent work Cao et al. [2022] proposed a Universal Prior Model (UPM), which can produce personalized expressions from unseen identities. Inspired by this work, we train a cross-identity hypernetwork together with an image expression encoder to enable personalized animation for our generated facial assets by novice users.” Fig. 9, Page 12. Moser, Page 6-7, “3.6 Image-to-geometry inference”).
As to claim 13, claim 8 is incorporated and the combination of Zhang and Moser discloses customizing the default head mesh comprises rendering the default head mesh as a 2D image (Moser, Page 4, “First, the bounding box containing the face is determined for each frame using a face detection model. Then a 2D landmark detection model [Bulat and Tzimiropoulos 2017] locates fiducial points on the face. A subset of these landmarks encompassing eyebrows, nose and eyes is used for computing the 2D affine transformation [Umeyama 1991] for the alignment to a canonical frontal head pose [Perov et al. 2019]. The aligned face image is produced with the 2D affine transformation applied on the input image. The head pose on each frame is estimated [Ruiz et al. 2018] and stored. In the final step, the detected landmarks are used to build a segmentation mask [Naruniec et al. 2020], corresponding to the face region only.”),
computing a loss function that measures a discrepancy between the 2D image and an expected image (Moser, Page 5, “Equation 1 combines three terms. L𝐿1 and L𝑆𝑆𝐼𝑀 are image reconstruction loss terms, important for the task of face swapping [Naruniec et al. 2020; Yan et al. 2018]. Both terms are defined by equation 2 using different criterion functions 𝑓𝑐, respectively L1 and structural similarity (SSIM) [Wang et al. 2004]. In order to encourage the network to focus on the face region, both the input image(𝑥𝑖)and the reconstructed image (˜𝑥𝑖) from domain 𝑖 are masked by the background mask (𝑚𝑥𝑖 ), using element-wise multiplication for each image channel. The reconstructed image is computed with the shared encoder and the input domain decoder using ˜𝑥𝑖 = 𝐷𝑒𝑐𝑖(𝐸𝑛𝑐(𝑥𝑖)). The losses are aggregated and averaged over all images in the batch 𝑋𝑖.”),
adjusting the default head mesh based on the computed loss function, and propagating gradients back to the adjusted default head mesh (Moser, Page 11, “Cycle consistency loss” section. Fig. 15).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YU CHEN whose telephone number is (571)270-7951. The examiner can normally be reached on M-F 8-5 PST Mid-day flex.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached on 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YU CHEN/
Primary Examiner, Art Unit 2613
Prosecution Insights