Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Applicant’s Arguments
Applicant’s arguments filed 12/22/2025 have been fully considered, but they are not deemed to be persuasive. Applicant argues that the cited references do not teach the claimed “training the IRN together with an expression extraction network and an identity extraction network using a two dimensional (2D) input image of a face, wherein the expression extraction network is trained to generate an expression embedding from the 2D input image, the identity extraction network is trained to generate an identity embedding from the 2D input image, and the IRN comprises a neural network that is trained to obtain the expression embedding and the identity embedding and to generate an implicit 3D representation of the face, wherein the implicit 3D representation of the face comprises an estimation of an implicit function describing the face.” Specifically, Applicant argues, “Since the cited Zheng uses the 3D face scans for calculating the loss function used for training its ImFace model, Zheng cannot replace the ground truth 3D face scans with 2D images since such a replacement will make calculating the loss function required by Zheng impossible” which is not persuasive. Zheng teaches that ‘the use of inputting 3D image improves over the inputting of 2D image because the 3D image has “high fidelity and fine details” than the low resolution of input 2D image’ (see Zheng, 2. Related Work: 3D Morphable Face Models - A number of models were learned from 2D images, but they mostly lacked high fidelity and fine details due to the low resolution of input images in this ill-posed inverse problem); furthermore, Applicant’s argument, “a replacement (i.e., 3D image by 2D image) will make calculating the loss function required by Zheng impossible,” is not persuasive because, in Zheng’s ImFace calculation for a Loss in the Nonlinear 3D Morphable Face Model with Implicit Neural Representations, by assuming the depth z=0 to the sample of 2D image, the Zheng’s proposed INR network (e.g., Figure 2) can work with the input data of a “modified” 2D image data (i.e., modifying a 2D sample (x, y) to a 3D sample (x, y, 0)) as claimed. Accordingly, the claimed invention as represented in the claims does not represent a patentable distinction over the art of record.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over LI et al (Facial Expression Recognition with Identity and Emotion Joint Learning) in view of Zheng et al (ImFace: A Nonlinear 3D Morphable Face Model with Implicit Neural Representations) and ZHANG et al (FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning).
As per claim 1, LI teaches the claimed "method for using an implicit representation network to produce three-dimensional (3D) animation," the method comprising: "training the network together with an expression extraction network and an identity extraction network, wherein the expression extraction network is trained to generate an expression embedding from an input image of a face, the identity extraction network is trained to generate an identity embedding from the input image" (Li, figure 2 - Our model consists of two convolutional neural networks. The left one represents the Deep/D network learning the identity features. The right deep residual network is trained with facial expression databases. After training separately, the identity feature and the deep- learned emotion feature are concatenated as the TFE features and feed to the subsequent fully connected layers), and "controlling at least one of the expression and identity of the implicit 3D representation of the face by changing at least one of the identity embedding and the expression embedding” (Li, 3.1 Overview of our Network Architecture - In order to guide this merged network to extract identity and emotion features, we do not train from scratch but pre-train the weights of both two sub- networks separately with the corresponding datasets except the fully connected layer; Figure 2 - The left one represents the DeepID network learning the identity features. The right deep residual network is trained with facial expression databases. After training separately, the identity feature and the deep-learned emotion feature are concatenated; 3.2 Identity and Emotion Feature Concatenation - Suppose that the identity and emotion features of an arbitrary inout image are represented as Zi and Ze, respectively. Then, we can reconstruct the new TFE representation Ztfe by concatenating Zi and Ze together. However, sometimes the deep-learned Zi and Ze features are not in the same scale because both network structure and training data are different). It is noted that Li does not explicitly teach the generation "an implicit 3D representation of the face, wherein the implicit 3D representation of the face comprises an estimation of an implicit function describing the face" and "implicit representation network (IRN) comprising a neural network" as claimed. However, Li's "representation networks” (figure 2 - Our model consists of two convolutional neural networks. The left one represents the DeepID network learning the identity features. The right deep residual network is trained with facial expression databases) suggests the representation network can be any conventional representation neural network can be used, such as the well-known "implicit representation network (IRN)" to generate "an implicit 3D representation of the face comprising an estimation of an implicit function describing the face (see Zheng, 1 Introduction - Recently, several studies on Implicit Neural Representations (INRs) have shown that 3D geometries can be precisely modeled by learning continuous deep implicit functions; 3. Method - The proposed ImFace explicitly disentangles facial shape morphs into two separate deformation fields associated with identity and expression, respectively, and a deep SDF is learned to represent the template shape. All the fields are blended with a series of local implicit functions for more detailed representation; Figure 2. ImFace overview - the Expression and Identity Mini-Nets blocks are associated with expression and identity deformations). It is noted that Zheng is using 3D image sample to get “high fidelity and fine details” than the low resolution of the claimed “input 2D image” (see Zheng, 2. Related Work: 3D Morphable Face Models - A number of models were learned from 2D images, but they mostly lacked high fidelity and fine details due to the low resolution of input images in this ill-posed inverse problem); furthermore, Applicant’s argument, “a replacement (i.e., 3D image by 2D image) will make calculating the loss function required by Zheng impossible,” is not persuasive because, in Zheng’s ImFace calculation for a Loss of the Nonlinear 3D Morphable Face Model with Implicit Neural Representations, by assuming the depth z=0 to the sample of 2D image, the Zheng’s proposed INR network (e.g., Figure 2) can work with the input data of a “modified” 2D image data (i.e., modifying a 2D sample (x, y) to a 3D sample (x, y, 0)) as claimed.
The Zheng reference also teaches the claimed "training the IRN together with an expression extraction network and an identity extraction network using a two dimensional (2D) input image of a face, wherein the expression extraction network is trained to generate an expression embedding from the 2D input image, the identity extraction network is trained to generate an identity embedding from the 2D input image, and the IRN is trained to obtain the expression embedding and the identity embedding and to generate an implicit 3D representation of the face (Zheng, Figure 2(a) - ImFace overview. (a) The proposed network consists of three Mini-Nets blocks to explicitly disentangle shape morphs into separate deformation fields, where the Expression and Identity Mini-Nets blocks are associated with expression and identity deformations, respectively); furthermore, Zheng teaches the new features on the amended claims, such as "training the IRN together with an expression extraction network and an identity extraction network using a two dimensional (2D) input image of a face, wherein the expression extraction network is trained to generate an expression embedding from the 2D input image, the identity extraction network is trained to generate an identity embedding from the 2D input image, and the IRN is trained to obtain the expression embedding and the identity embedding and to generate an implicit 3D representation of the face" (Zheng, 3.1. Disentangled INRs Network - The fundamental idea of INRs is to train a neural network to fit a continuous function f, which implicitly represents surfaces through level-sets… Expression Mini-Nets (ExpNet)). The facial deformations incurred by expressions are represented by ExpNet Ɛ… Identity Mini-Nets (IDNet)…; see also Zhang, Figure 2 - Given input audio, the proposed FACIAL-GAN aims to generate the explicit attributes (expression) and implicit attributes (eye blinking AU45, head pose) with jointly temporal correlations and local phonetic features) (Noted: Zhang’s expression attribute can be implicitly represented as showed in Zheng’s Expression Mini-Nets). Since the cited references related to the neural network training the identity and expression of a human, it would have been obvious, in view of Zhang and Zheng, to configure Li's system as claimed by using a neural network "to train the IRN together with an expression extraction network and an identity extraction network using a two dimensional (2D) input image of a face, wherein the implicit 3D representation of the face comprises an estimation of an implicit function describing the face." The motivation is to an identity and emotion joint learning approach with implicit neural networks to enhance the performance of facial expression recognition (FER) tasks (Zheng, Abstract), and to use a desired facial representation data format to implement the learning network (Zheng, Figure 2 - Overview of the proposed implicit attribute learning framework).
Claim 2 adds into claim 1 "wherein changing the identity embedding comprises providing a second input image of a face having a required identity to the identity extraction network and wherein changing the expression embedding comprises providing a second input image of a face having a required expression to the expression extraction network" (Zheng, Figure 2. ImFace overview - the Expression and Identity Mini-Nets blocks are associated with expression and identity deformations; 3.1. Disentangled INRs Network - The fundamental idea of INRs is to train a neural network to fit a continuous function f, which implicitly represents surfaces through level-sets… Expression Mini-Nets (ExpNet)). Since all the cited references related to the neural network training the identity and expression of a human, it would have been obvious, in view of Zhang and Zheng, to configure Li's system as claimed by using a neural network "for changing the identity embedding and expression embedding." The motivation is to an identity and expression joint learning approach with implicit neural networks to enhance the performance of facial expression recognition (FER) tasks (Zheng, Abstract), and to use a desired facial representation data format to implement the learning network (Zheng, Figure 2).
Claim 3 adds into claim 1 "wherein the IRN further obtains a speech embedding generated from the input image of a face" (Zhang, Figure 2 - Given input audio, the proposed FACIAL-GAN aims to generate the explicit attributes (expression) and implicit attributes (eye blinking AU45, head pose) with jointly temporal correlations and local phonetic features) (Noted: Zhang’s expression attribute can be implicitly represented as showed in Zheng’s Expression Mini-Nets). Since the cited references related to the neural network training the identity and expression of a human, it would have been obvious, in view of Zhang and Zheng, to configure Li's system as claimed by using a neural network "to obtain a speech embedding generated from the input image of a face." The motivation is to an implement a speech for the inputted face with implicit neural networks to enhance the performance of facial expression recognition (FER) tasks (Zheng, Abstract), and to use a desired facial representation data format to implement the learning network.
Claim 4 adds into claim 3 "wherein the IRN comprises: "an identity embedder being a network configured to obtain the identity embedding and generate an identity representation" (Zheng, Figure 2. ImFace overview - the Expression and Identity Mini- Nets blocks are associated with expression and identity deformations; 3.1. Disentangled INRs Network - The fundamental idea of INRs is to train a neural network to fit a continuous function f, which implicitly represents surfaces through level-sets… Expression Mini-Nets (ExpNet)). The facial deformations incurred by expressions are represented by ExpNet Ɛ… Identity Mini-Nets (IDNet)…), "an expression embedder being a network configured to obtain the expression embedding and the speech embedding generated from the 2D input image of the face, and generate an expression representation" (Zheng, Figure 2. ImFace overview - the Expression and Identity Mini- Nets blocks are associated with expression and identity deformations); and "a fuser being a network configured to obtain the identity representation and the expression representation and generate a fused identity and expression representation" (Zheng, Figure 2. ImFace overview - the Expression and Identity Mini- Nets blocks are associated with expression and identity deformations). Since all the cited references related to the neural network training the identity and expression of a human, it would have been obvious, in view of Zhang, Zheng, to configure Li's system as claimed by using a neural network "to train and to obtain the expression embedding and the identity embedding and to generate an implicit 3D representation of the face." The motivation is to an identity and emotion joint learning approach with implicit neural networks to enhance the performance of facial expression recognition (FER) tasks, and to use a desired facial representation data format to implement the learning network.
Claim 5 adds into claim 1 "wherein the IRN comprises: an identity embedder being a network configured to obtain the identity embedding and generate an identity representation" (Zheng, Figure 2. ImFace overview - the Expression and Identity Mini- Nets blocks are associated with expression and identity deformations; 3.1. Disentangled INRs Network - The fundamental idea of INRs is to train a neural network to fit a continuous function f, which implicitly represents surfaces through level-sets… Expression Mini-Nets (ExpNet)). The facial deformations incurred by expressions are represented by ExpNet Ɛ… Identity Mini-Nets (IDNet)…), "an expression embedder being a network configured to obtain the expression embedding and generate an expression representation" (Zheng, Figure 2. ImFace overview - (a) The proposed network consists of three Mini-Nets blocks to explicitly disentangle shape morphs into separate deformation fields, where the Expression and Identity Mini-Nets blocks are associated with expression and identity deformations, respectively), "a fuser being a network configured to obtain the identity representation and the expression representation and generate a fused identity and expression representation" (Zheng, Figure 2. ImFace overview - It is tailed by a Fusion Network for more comprehensive representations. (d) The Fusion Network is a lightweight module conditioned on the query point position, which adaptively blends the local field functions, resulting in an elaborate Neural Blend-Field). Since all the cited references related to the neural network training the identity and expression of a human, it would have been obvious, in view of Zhang and Zheng, to configure Li's system as claimed by using a neural network "to train and to obtain the expression embedding and the identity embedding and to generate an implicit 3D representation of the face." The motivation is to an identity and emotion joint learning approach with implicit neural networks to enhance the performance of facial expression recognition (FER) tasks, and to use a desired facial representation data format to implement the learning.
Claim 6 adds into claim 5 "converting the implicit 3D representation into an explicit 3D representation by: a shape predictor being a network configured to obtain the fused identity and expression representation and predict a shape representation" (Zheng, Figure 2. ImFace overview - (a) The proposed network consists of three Mini- Nets blocks to explicitly disentangle shape morphs into separate deformation fields, where the Expression and Identity Mini-Nets blocks are associated with expression and identity deformations, respectively, and the Template Mini-Nets block learns the SDF of a template face space. (b) The Mini-Nets block is a shared architecture, which decomposes an entire facial feature into semantically meaningful parts and encodes them by a set of local field functions. It is tailed by a Fusion Network for more comprehensive representations. (c) The Landmark-Net is introduced to softly partition the entire facial surface. (d) The Fusion Network is a lightweight module conditioned on the query point position, which adaptively blends the local field functions, resulting in an elaborate Neural Blend-Field), and "a texture field being a network configured to obtain the fused identity and expression representation F and the shape representation and predict a texture representation" (Zheng, 5. Discussion - a basic texture model can be achieved by plugging a color field). Since all the cited references related to the neural network training the identity and expression of a human, it would have been obvious, in view of Zhang and Zheng, to configure Li's system as claimed by using a neural network for "converting the implicit 3D representation into an explicit 3D representation by: a shape predictor being a network configured to obtain the fused identity and expression representation and predict a shape representation." The motivation is to an identity and emotion joint learning approach with implicit neural networks to enhance the performance of facial expression recognition (FER) tasks, and to use a desired facial representation data format to implement the learning network (Zheng, 3.1. Disentangled INRs Network - The fundamental idea of INRes is to train a neural network to fit a continuous function f, which implicitly represents surfaces through level- sets. The function can be defined in various formats, e.g. occupancies, SDF, or UDF. We exploit a deep SDF conditioned on the latent embeddings of both expression and identity for comprehensive face representations).
Claim 7 adds into claim 6 "wherein the shape representation comprises a set of tuples {v_i0,v_i1, ...v_ij} i:0->N describing at least one of a topology in 3D and a value of occupancy in vertices {v}ij' which is a conventional method to describe a shape in space by the connections between elements of a given set of vertices (Zheng, 3.5 Data Processing - Pseudo watertight face generation in which mesh is formed by connecting between elements of a given set of vertices). It would have been obvious, in view of Zhang and Zheng to represent a shape on space by using the points, or vertices, on the space because a connection of collection of these points, or vertices, can represent, or form, a shape of object on space. The motivation is to an identity and emotion joint learning approach with implicit neural networks to enhance the performance of facial expression recognition (FER) tasks, and to use a desired facial representation data format to implement the learning network (Zheng, 3.1. Disentangled INRs Network - The fundamental idea of INRes is to train a neural network to fit a continuous function f, which implicitly represents surfaces through level-sets. The function can be defined in various formats, e.g. occupancies, SDF, or UDF. We exploit a deep SDF conditioned on the latent embeddings of both expression and identity for comprehensive face representations).
Claim 8 adds into claim 7 "wherein the value of occupancy comprises a signed distance function (SDF) value of the implicit representation at the vertices {v}ij' which is conventional well-known in the art to represent a object's shape by the vertices in which each vertex represents a value of measurement function such as the well-known "signed distance function (SDF)" (Zheng, 3.5 Data Processing - SDF computation on facial surfaces). It would have been obvious to use the conventional SDF to represent a point, or vertex, on the space because a collection of these points, or vertices, can be used to form a shape of object on space. Since all the cited references related to the neural network training the identity and expression of a human, it would have been obvious, in view of Zhang and Zheng, to configure Li's system as claimed by using a neural network "to train and to obtain the expression embedding and the identity embedding and to generate an implicit 3D representation of the face." The motivation is to an identity and emotion joint learning approach with implicit neural networks to enhance the performance of facial expression recognition (FER) tasks, and to use a desired facial representation data format to implement the learning network.
Claim 9 adds into claim 1 "wherein training the IRN is performed using a training dataset of facial images, wherein the dataset comprises facial images extracted from a plurality of videoclips of a plurality of persons, with a plurality of viewpoints per identity and a plurality of timepoints per viewpoint." As showed above, in both of Li and Zheng's teaching, the facial features, e.g., landmarks used in training, are collected from the input image and scan, respectively, of a 3D face. It is well- known in the art, to convert the facial data between a 3D facial model and 2D images of the facial model. Since both of Li and Zheng references teach a facial process using the combination of an identity network and an expression network, it would have been obvious, in view of Zhang and Zheng, to modify the input data of Li's 2D facial images to a 3D facial model by adding a depth z=0 to the 2D sample data for the purpose of inputting into the Zheng network using inputted 3D image (see also Li, 7 INTRODUCTION - FACIAL Expression Recognition (FER) is a well defined task, aiming to recognize facial expressions with discrete categories (e.g., neutral, sad, contempt, happy, surprise, angry, fear, disgust, etc.) or continuous levels (e.g., valance, arousal) from still images or videos) (see example of captured images in Li, figures 1 and 7).
Claim 10 adds into claim 9 "wherein training the IRN is performed using at least one of: an adversarial loss term generated by rendering two-dimensional (2D) images from the 3D model of the face and applying a pre-trained discriminator that is trained on the domain of human face images on the 2D images, a sync loss term generated by providing a sequence of input facial images taken from a speaking human in a videoclip, generating the implicit 3D representation of the face for each of the input facial images, generating an animation from the generated implicit 3D representations, and measuring a level of discrepancy between the generated animation and the original speech in the videoclip, and a reconstruction loss term generated by calculating a distance function between an original 2D input facial image and the rendered 2D image, wherein the original input facial image and the rendered 2D image have same extrinsic camera parameters" (Zheng, 3.4. Loss Functions - Reconstruction Loss). As showed above, in both of Li and Zheng's teaching, the facial features, e.g., landmarks used in training, are collected from the input image and scan, respectively, of a 3D face. It is well-known in the art, to convert the facial data between a 3D facial model and 2D images of the facial model (e.g., in Zheng’s ImFace calculation for a Loss in the Nonlinear 3D Morphable Face Model with Implicit Neural Representations, by assuming the depth z=0 to the sample of 2D image, the Zheng’s proposed INR network (e.g., Figure 2) can work with the input data of a “modified” 2D image data (i.e., modifying a 2D sample (x, y) to a 3D sample (x, y, 0)) as claimed). Since the cited references related to the neural network training the identity and expression of a human, it would have been obvious, in view of Zhang and Zheng, to configure Li's system as claimed by using a neural network "for reconstruction loss term generated by calculating a distance function between an original 2D input facial image and the rendered 2D image." The motivation is to implement a learning approach with implicit neural networks to enhance the performance through a loss function (Zheng, 3.4. Loss Functions - Reconstruction Loss).
Claims 11-20 claim a system based on the method of the claims 1-10; therefore, they are rejected under a similar rationale.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHU K NGUYEN whose telephone number is (571)272-7645. The examiner can normally be reached M-F 8-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel F. Hajnik can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHU K NGUYEN/ Primary Examiner, Art Unit 2616
Prosecution Insights