DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This action is in response to the amendment filed on 12th May 2026. Claims 1, 15, and 29-30 have been amended. Claims 5-6, 10-12, 14, 19-20, and 24-26 have been cancelled. Claims 1-4, 7-9, 13, 15-18, 21-23, and 27-30 remain rejected in the application.
Response to Arguments
Applicant's arguments with respect to Claims 1, 15, and 29-30 filed on 12th May, 2026, with respect to the rejection under 35 U.S.C. § 103, regarding that the prior art does not teach the limitation(s): "obtain an image of at least a portion of a mouth of the face of the user", "process the image of the at least the portion of the mouth to generate mouth-image features using a mouth-image-encoder machine-learning model, wherein the mouth-image-encoder machine-learning model is trained to generate mouth-image features based on training mouth images", "obtain a view for a three-dimensional model of the face, wherein the view for the three-dimensional model of the face is based on an angle from which the three-dimensional model of the face is to be viewed", "process the view using a view-encoder machine-learning model to generate view features, wherein the view-encoder machine-learning model is trained to generate view-based features based on training views", "combine the image features, mouth-image features, the audio features, and the view features to obtain combined features", and "process the combined features using a model-generator machine-learning model to generate a three-dimensional model of the face, wherein the model-generator machine-learning model is trained to generate three-dimensional models based on training image features, training mount-image features, training audio features, and training view features" have been fully considered, but are moot because of new grounds for rejection. It has now been taught by the combination of Richard and Cao.
Regarding arguments to Claims 2-4, 7-9, 13, 16-18, 21-23, and 27-28, they directly/indirectly depend on independent Claims 1, 15, and 29-30 respectively. Applicant does not argue anything other than independent Claims 1, 15, and 29-30. The limitations in those claims, in conjunction with combination, was previously established as explained.
Claim Objections
Claims 1, 15, and 29-30 are objected to because of the following informalities:
Claim 1 recites the limitation(s): "training mount-image features" on PG(s). 3, Line(s) 10; examiner suggest amending this to "training mouth-image features";
Claim 15 recites the limitation(s): "training mount-image features" on PG(s). 5, Line(s) 24-25; examiner suggest amending this to "training mouth-image features";
Claim 29 recites the limitation(s): "training mount-image features" on PG(s). 8, Line(s) 17; examiner suggest amending this to "training mouth-image features"; and
Claim 30 recites the limitation(s): "training mount-image features" on PG(s). 9, Line(s) 19-20; examiner suggest amending this to "training mouth-image features".Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7-9, 15-18, 21-23, and 28-30 are rejected under 35 U.S.C. 103 as being unpatentable over Richard et al. (US 20220309724 A1, previously cited), hereinafter referenced as Richard, in view of Cao et al. (US 20230245365 A1, previously cited), hereinafter referenced as Cao.
Regarding Claim 1, Richard discloses an apparatus for generating models of faces, the apparatus (Richard, [0089]: teaches a computer system 1200 <read on apparatus> used for generating 3D facial models as shown in FIG. 12) comprising:
PNG
media_image1.png
417
451
media_image1.png
Greyscale
at least one memory (Richard, [0090]: teaches the computer system 1200 including memory 1204); and
at least one processor coupled to the at least one memory and configured to (Richard, [0089]: teaches the computer system 1200 including processor 1202):
obtain one or more images of one or both eyes of a face of a user (Richard, [0082]: teaches receiving audio and image capture of the face of a subject <read on obtain images of both eyes of face of user>);
process the one or more images of the one or both eyes of the user using one or more image-encoder machine-learning models to generate image features (Richard, [0033]: teaches a facial expression encoder, which is a part of a multimodal encoder 240 <read on image-encoder machine-learning models>, that identifies an expression-like facial feature of the subject to generate a second mesh for an upper portion of the face of the subject <read on generate image features>, where sampling of multiple facial expressions of the subject is performed), wherein
the image features comprise a vector of values representative of the one or more images (Richard, [0040]: teaches training categorical latent space <read on vector of values>, which represents a sequence of face meshes 329; Note: it should be noted that latent space uses vector values) and wherein
the one or more image-encoder machine-learning models are trained to generate image-based features based on training images (Richard, [0032]: teaches a training database 252 that includes "training archives and other data files that may be used by 3D speech animation engine 232 in the training of a machine learning model," which includes training one of the encoders of multimodal encoder 240, such as facial expression encoder 244, being trained on images <read on training images> of the user's face to synthesize a portion of a facial expression <read on generated image-based features>);
obtain an image of at least a portion of a mouth of the face of the user (Richard, [0033]: teaches the facial expression encoder 244 identifying expression-like facial features of the subject <read on obtained image>, such as the mouths <read on portion of mouth>, lips, cheek, and cheeks, to generate a first mesh portion; Note: it should be noted that the facial expression encoder processes images of the user's face, where it uses a pair of inputs, one of which is the ground-truth image of the user's face; additionally, the image is being interpreted as not being exclusively the mouth (i.e., a full picture of the face));
process the image of the at least the portion of the mouth to generate mouth-image features using a mouth-image-encoder machine-learning model (Richard, [0033]: teaches the facial expression encoder <read on mouth-image-encoder machine-learning model> identifying an expression-like facial feature of the subject to generate a second mesh <read on generated mouth-image features> for an upper portion of the face of the subject based on stored facial expressions), wherein
the mouth-image-encoder machine-learning model is trained to generate mouth-image features based on training mouth images (Richard, [0033]: teaches the facial expression encoder <read on mouth-image-encoder machine-learning model> identifying an expression-like facial feature of the subject to generate a second mesh for an upper portion of the face of the subject based on stored facial expressions <read on training mouth images> in training database 252),
obtain audio data based on utterances of the user (Richard, FIG. 9 teaches obtaining speech audio input <read on obtained audio data> of a subject <read on utterance of user> and modifying a mesh template to match facial features to the audio input); and
PNG
media_image2.png
415
548
media_image2.png
Greyscale
process the audio data using an audio-encoder machine-learning model to generate audio features (Richard, [0039]: teaches recording a speech signal 328 <read on audio data>, where "for each tracked mesh, a Mel spectrogram is generated <read on generated audio features>, including a 600 ms audio snippet starting 500 ms before and ending 100 ms after the respective visual frame"; [0033]: teaches using an audio encoder <read on audio-encoder machine-learning model> that "identifies audio-correlated facial features to generate a first mesh for a lower portion of a face of a subject, according to a classification scheme that is learned by training"), wherein
the audio features comprise a vector of values representative of the audio data (Richard, [0039]: teaches the generated Mel spectrogram for each tracked mesh including 80-dimensional Mel spectral features <read on vector of values>; Note: it should be noted that vectors are used to represent dimensions) and wherein
the audio-encoder machine-learning model is trained to generate audio-based features based on training audio data (Richard, [0033]: teaches using an audio encoder <read on audio-encoder machine-learning model> that "identifies audio-correlated facial features to generate a first mesh <read on generated audio-based features> for a lower portion of a face of a subject, according to a classification scheme that is learned by training <read on training audio data>");
[[obtain a view for a three-dimensional model of the face, wherein]]
[[the view for the three-dimensional model of the face is based on an angle from which the three-dimensional model of the face is to be viewed;]]
[[process the view using a view-encoder machine-learning model to generate view features, wherein]]
[[the view-encoder machine-learning model is trained to generate view-based features based on training views;]]
combine the image features, mouth-image features, the audio features, [[and the view features]] to obtain combined features (Richard, [0037]: teaches a synthetic encoder 348 including "a fusion block 330 to map a sequence of input animated face meshes 329 <read on image and mouth-image features> (the expression signal) and speech signal 328 to an encoded expression 341 <read on audio features> in a categorical latent space 340, via a synthetic encoder 348," where "a decoder 350 animates neutral face mesh 327 from encoded expression 341"); and
process the combined features using a model-generator machine-learning model to generate a three-dimensional model of the face (Richard, FIG. 10 teaches generating a synthesized mesh <read on generate 3D model of face> based on the first and second meshes, where the first mesh is generated for a lower portion of a face of the subject based on audio-correlated facial feature, and the second mesh is generated for an upper portion of a face of the subject based on expression-like facial feature, which uses a multimodal encoder; [0034]: teaches the synthetic decoder 248 <read on model-generator machine-learning model> generating "a synthetic mesh of the full face of the subject with the first mesh provided by audio encoder 242 and the second mesh provided by facial expression encoder 244," where "synthetic decoder 248 merges <read on process combined features> continuously and seamlessly a lip shape in the first mesh provided by audio encoder 242 into an eye closure in the second mesh provided by facial expression encoder 244, across the face of the subject"), wherein
PNG
media_image3.png
412
511
media_image3.png
Greyscale
the model-generator machine-learning model is trained to generate three-dimensional models based on training image features, training mount-image features, training audio features, [[and training view features]] (Richard, [0083]: teaches "training a 3D model to create real-time 3D speech animation <read on generate 3D models> of a subject"; [0085]: teaches "generating a first mesh for a lower portion of a human face, based on the facial feature <read on training image> and the first correlation value"; [0033]: teaches the facial expression encoder 244 selects "the expression-like facial feature based on a prior sampling of multiple subject's facial expressions").
However, Richard does not expressly disclose
obtain a view for a three-dimensional model of the face, wherein
the view for the three-dimensional model of the face is based on an angle from which the three-dimensional model of the face is to be viewed;
process the view using a view-encoder machine-learning model to generate view features, wherein
the view-encoder machine-learning model is trained to generate view-based features based on training views;
combine the image features, mouth-image features, the audio features, and the view features to obtain combined features; and
the model-generator machine-learning model is trained to generate three-dimensional models based on training image features, training mount-image features, training audio features, and training view features.
Cao discloses
obtain a view for a three-dimensional model of the face (Cao, FIG. 7 teaches obtaining a plurality of views of the user's face), wherein
PNG
media_image4.png
302
604
media_image4.png
Greyscale
the view for the three-dimensional model of the face is based on an angle from which the three-dimensional model of the face is to be viewed (Cao, [0073]: teaches generating tracked meshes, where "a high-coverage landmark detector runs multiple views <read on angle> of each frame," in which "the detected landmarks are then used to initialize a Principal Component Analysis (PCA) model-based tracking method to produce the final tracked mesh");
process the view using a view-encoder machine-learning model to generate view features (Cao, [0042]: teaches "a universal prior model <read on view-encoder machine-learning model> that is trained on high resolution multi-view video captures of facial performances of hundreds of human subjects," where the 3D head avatar output matches the facial shape and appearance <read on generate view features> of the user from a plurality of angles), wherein
the view-encoder machine-learning model is trained to generate view-based features based on training views (Cao, [0042]: teaches "a universal prior model <read on view-encoder machine-learning model> that is trained on high resolution multi-view video captures <read on training views> of facial performances of hundreds of human subjects," where the 3D head avatar output matches the facial shape and appearance <read on generate view-based features> of the user from a plurality of angles);
combine the image features, mouth-image features, the audio features, and the view features to obtain combined features (Cao, [0042]: teaches "a universal prior model that is trained on high resolution multi-view video captures of facial performances of hundreds of human subjects," where the 3D head avatar output matches the facial shape and appearance <read on view-based features> of the user from a plurality of angles); and
the model-generator machine-learning model is trained to generate three-dimensional models based on training image features, training mount-image features, training audio features, and training view features (Cao, [0042]: teaches "a universal prior model that is trained on high resolution multi-view video captures <read on training views> of facial performances of hundreds of human subjects").
Cao is analogous art with respect to Richard because they are from the same field of endeavor, namely utilizing machine learning models to generate 3D face meshes based on training data. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement a universal prior model to capture and update viewing angles of the captured 3D face mesh as taught by Cao into the teaching of Richard. The suggestion for doing so would allow the multimodal system to incorporate the universal prior model as part of the encoder stack, thereby offering multi-view images of the user, thus resulting in a more accurate output result. Therefore, it would have been obvious to combine Cao with Richard.
Regarding Claim 15, it recites the limitations that are similar in scope to Claim 1, but in a method. As shown in the rejection, the combination of Richard and Cao discloses the limitations of Claim 1. Additionally, Richard discloses a method for generating models of faces, the method (Richard, [0074]: teaches method 1000 for embedding a 3D speech animation model <read on generating face models> in a VR environment) comprising:…
Thus, Claim 15 is met by Richard according to the mapping presented in the rejection of Claim 1, given the apparatus corresponds to a method.
Regarding Claim 29, it recites the limitations that are similar in scope to Claim 1, but in a non-transitory computer-readable storage medium. As shown in the rejection, the combination of Richard and Cao discloses the limitations of Claim 1. Additionally, Richard discloses a non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to (Richard, [0020]: teaches "a non-transitory, computer-readable medium stores instructions which, when executed by a processor, cause a computer to perform a method"):…
Thus, Claim 29 is met by Richard according to the mapping presented in the rejection of Claim 1, given the apparatus corresponds to a non-transitory computer-readable storage medium.
Regarding Claim 30, it recites the limitations that are identical in scope to Claim 1. As shown in the rejection, the combination of Richard and Cao discloses the limitations of Claim 1. Additionally, Richard discloses an apparatus for generating models of faces, the apparatus (Richard, [0089]: teaches a computer system 1200 <read on apparatus> used for generating 3D facial models as shown in FIG. 12) comprising:…
Thus, Claim 30 is met by Richard according to the mapping presented in the rejection of Claim 1.
Regarding Claims 2 and 16, the combination of Richard and Cao discloses the apparatus and the method of Claims 1 and 15 respectively. Additionally, Richard further discloses wherein
a mouth portion of the three-dimensional model of the face is based on the audio data (Richard, [0025]: teaches "the latent space is trained based on a novel cross-modality loss that encourages the model to have an accurate upper face reconstruction independent of the audio input and accurate mouth area <read on mouth portion of 3D model of face> that only depends on the provided audio input").
Regarding Claims 3 and 17, the combination of Richard and Cao discloses the apparatus and the method of Claims 1 and 15 respectively. Additionally, Richard further discloses wherein the three-dimensional model comprises
a three-dimensional morphable model (3DMM) of the face (Richard, FIG. 9 teaches obtaining speech audio input and modifying a mesh template <read on 3DMM of face> to match facial features to the audio input).
Regarding Claims 4 and 18, the combination of Richard and Cao discloses the apparatus and the method of Claims 1 and 15 respectively. Additionally, Richard further discloses wherein the three-dimensional model comprises
a plurality of vertices corresponding to points of the face (Richard, [0038]: teaches face meshes including 6, 172 vertices <read on points of face> with a high level of detail including eye lids, upper face structure, and different hair styles).
Regarding Claims 7 and 21, the combination of Richard and Cao discloses the apparatus and the method of Claims 1 and 15 respectively. Additionally, Richard further discloses wherein the audio data comprises
perception-based representation of the utterances of the user (Richard, [0061]: teaches model 500 generating two different sets of latent representations <read on perception-based representation>,
S
a
u
d
i
o
and
S
e
x
p
r
, where "
S
a
u
d
i
o
contains latent codes (lower face meshes 521A) obtained by fixing the expression input to facial expression encoder (e.g., facial expression encoders 244 and 344) and varying the audio signal <read on utterances of user>").
Regarding Claims 8 and 22, the combination of Richard and Cao discloses the apparatus and the method of Claims 7 and 21 respectively. Additionally, Richard further discloses wherein the perception-based representation of the utterances comprises
a representation of the audio data based on perceptually-relevant frequencies and perceptually-relevant amplitudes (Richard, [0075]: teaches "identifying an intensity and a frequency of the audio capture <read on audio data representation> from the subject and correlating an amplitude <read on perceptually-relevant amplitudes> and a frequency <read on perceptually-relevant frequencies> of an audio waveform with a geometry of the lower portion of the face of the subject"; Note: it should be noted that amplitudes are used to distinguish sounds from one another and can also be used for detecting and predicting a current emotion; correlating the amplitude and frequency of the audio capture is being interpreted as perceptually-relevant amplitudes and perceptually-relevant frequencies respectively).
Regarding Claims 9 and 23, the combination of Richard and Cao discloses the apparatus and the method of Claims 1 and 15 respectively. Additionally, Richard further discloses wherein the audio data comprises
a Mel spectrogram representative of the utterances of the user (Richard, [0065]: teaches using Deep-Speech features, which include Mel spectrograms).
Regarding Claim 28, the combination of Richard and Cao discloses the method of Claim 15. Additionally, Richard further discloses
obtaining an image of at least a portion of a mouth of the face of the user (Richard, [0033]: teaches the facial expression encoder 244 identifying expression-like facial features of the subject <read on obtained image>, such as the mouths <read on portion of mouth>, lips, cheek, and cheeks, to generate a first mesh portion); and
generating mouth-image features using a mouth-image-encoder machine-learning model (Richard, [0033]: teaches the facial expression encoder <read on mouth-image-encoder machine-learning model> identifying an expression-like facial feature of the subject to generate a second mesh <read on generated mouth-image features> for an upper portion of the face of the subject based on stored facial expressions), wherein
the mouth-image-encoder machine-learning model is trained to generate mouth-image features based on training mouth images (Richard, [0033]: teaches the facial expression encoder <read on mouth-image-encoder machine-learning model> identifying an expression-like facial feature of the subject to generate a second mesh for an upper portion of the face of the subject based on stored facial expressions <read on training mouth images> in training database 252), wherein
the three-dimensional model of the face is generated based on the mouth-image features (Richard, [0034]: teaches the facial expression encoder generating and providing a second mesh of the face model, which are based on stored facial expressions in training database 242).
Claims 13 and 27 are rejected under 35 U.S.C. 103 as being unpatentable over Richard et al. (US 20220309724 A1, previously cited), hereinafter referenced as Richard, in view of Cao et al. (US 20230245365 A1, previously cited), hereinafter referenced as Cao as applied to Claims 1 and 15 above respectively, and further in view of Kwatra et al. (US 20230343010 A1, previously cited), hereinafter referenced as Kwatra.
Regarding Claims 13 and 27, the combination of Richard and Cao discloses the apparatus and the method of Claims 1 and 15 respectively. The combination of Richard and Cao does not expressly disclose the limitations of Claims 13 and 27;however, Kwatra discloses wherein the at least one processor is further configured to:
generate a UV map of the face based on the three-dimensional model of the face using a first renderer (Kwatra, [0068]: teaches the system predicting both texture and geometry of the user's face by "projecting the corresponding predicted vertices onto the reference cylinder, and using their 2D location as new texture coordinates <read on generating UV map>" to be used to render <read on first renderer> a 3D head model);
generate a texture map based on the UV map of the face using a machine-learning encoder-decoder (Kwatra, [0030]: teaches using an encoder-decoder framework "that computes embeddings from audio spectrograms, and decodes them into 3D geometry and texture <read on generating texture map>"); and
render the three-dimensional model of the face based on the three-dimensional model of the face and the texture map using a second renderer (Kwatra, [0082]: teaches "the computing system can combine the face geometry and the face texture <read on texture map> to generate a three-dimensional face mesh model"; [0084]: teaches "rendering <read on second renderer> the three-dimensional face mesh within the two-dimensional target video at the target position to generate the synthesized video").
Kwatra is analogous art with respect to Richard, in view of Cao because they are from the same field of endeavor, namely generating 3D face models based on audio input data. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to have the system analyze audio input data for phenomes and visemes as taught by Kwatra into the teaching of Richard, in view of Cao. The suggestion for doing so would allow the system to predict the user's facial expression based on spectrogram audio data and facial image data, thereby resulting in a more accurate and realistic 3D model of the user's face when combined together. Therefore, it would have been obvious to combine Kwatra with Richard, in view of Cao.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Berlin et al. (US 20230049729 A1) discloses training an autoencoder for generating a 3D face for face swapping;
Booth et al. (US 20220092853 A1) discloses receiving portions of the face of a user to generate a 3D facial model; and
Bouaziz et al. (US 20190180084 A1) discloses generating a 3D model of a user based on captured images/video of said user.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KARL TRUONG whose telephone number is (703)756-5915. The examiner can normally be reached 10:30 AM - 7:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.D.T./Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights