DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
2. Claim 1 is amended.
3. Claims 2-15 are as previously presented.
Claim Rejections - 35 USC § 103
4. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
5. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
6. Claims 1-3, 8, 10-11, 14 are rejected under 35 U.S.C. 103 as being unpatentable over Albuz (US 10970907 B1), hereinafter Albuz, in view of Miller (US 7643683 B2), hereinafter Miller.
Regarding claim 1, Albuz teaches a method comprising: training, by a processor (Fig. 2, Col. 8 lines 3-25, components and software implemented on a processor), a two-stage machine learning model based on avatar training images and specified blendshape weights (Fig. 1A, Col. 6, lines 3-31, a two-phase machine learning model based on input training images and associated sets of blendshape weights), the machine learning model having a first stage extracting image features from the rendered avatar training images (Fig. 1B, Col. 6 line 55 – Col. 7 line 10, wherein determining and classifying attributes of face parts of the input images during the training stage is interpreted as a first state extracting image features from training images) and a second stage predicting blendshape weights from the extracted image features (Col. 6, lines 49-53; Col. 7, lines 22-47, wherein generating a set of estimated blendshapes based on the attributes and classifications of the face parts of input images is interpreted as predicting blendshape weights based on extracted image features, and suggests the predicting blendshape weights step being performed after the extracting image features step which is interpreted as a second stage); and applying the trained machine learning model to predict the blendshape weights for a facial expression of a wearer of a head-mountable display (HMD) from a set of images captured by the HMD of a face of the wearer when exhibiting the facial expression (Fig. 1C, Col. 7 lines 53-63, rendering an avatar based on generated blendshape weights from input images; Fig. 4A-4C, Fig. 5A-5B, Col. 10 lines 22-562, generating an avatar based on a set of blendshapes determined from images of a wearer of an HMD exhibiting a corresponding facial expression).
Albuz does not teach rendering avatar training images of facial avatars having facial expressions corresponding to specified blendshape weights.
Miller teaches rendering avatar training images of facial avatars having facial expressions corresponding to specified blendshape weights (Fig. 1, Col. 4 lines 19-47, rendering 3D avatar training images wherein the avatars have differing facial expressions corresponding to specific avatar poses, coordinates, and deformations, which is interpreted as having specified blendshape weights).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Albuz to incorporate the teaching of Miller for this method of training a machine learning model for predicting blendshape weights for a wearer wearing an HMD. Albuz discusses a machine learning model for generating blendshape weights based on captured facial expressions from a user wearing an HMD that involves classifying and determining attributes of specific face parts of input images, then estimating blendshapes based on those attributes and classes. Miller teaches a way of generating 2D images based on 3D representations, primarily user avatars, for the purposes of generating robust and multifeatured databases for training computer systems. As Albuz discusses that it can be trained on thousands or millions of images, it would be obvious to one of ordinary skill that the image database generation of Miller could be used to generate that training data. As Albuz discusses a training a machine learning system for generating 3D avatars based on input images, and Miller discusses generating an image database of 3D avatars, it would be obvious to combine these references.
Regarding claim 2, Albuz in view of Miller discloses the method of claim 1. Additionally, Albuz teaches retargeting the predicted blendshape weights for the facial expression of the wearer of the HMD onto a facial avatar corresponding to the face of the wearer to render the facial avatar with the facial expression of the wearer (Col. 12, line 55 – Col. 13, line 2, wherein rendering and deforming an avatar according to an output set of blendshapes and their respective weights is interpreted as retargeting the predicted blendshape weights corresponding to a specific facial expression; Fig. 3, Col. 10 lines 10-21, wherein the HMD may capture a wearer’s facial expression as an input to render an avatar); and displaying the rendered facial avatar corresponding to the face of the wearer (Fig. 3, Fig. 6A-6D, Col. 10, lines 10-21, wherein the HMD renders an avatar based on input images of the wearer).
Regarding claim 3, Albuz in view of Miller discloses the method of claim 1. Additionally, Albuz teaches the method of claim 1, wherein the set of images captured by the HMD of the face of the wearer comprises left and right eye images of facial portions of the wearer respectively including left and right eyes of the wearer and a mouth image of a lower facial portion of the wearer including a mouth of the wearer (Fig. 4A-4C, Col. 10 lines 22-32, wherein the HMD captures images of a right and left eye of the wearer, and the lower face and mouth of the user), the method further comprising: for each avatar training image of a facial avatar having a facial expression, simulating left and right eye avatar training images in correspondence with the left and right eye images captured by the HMD and a mouth avatar training image in correspondence with the mouth image captured by the HMD (Fig. 4A-4C, Col. 10 lines 33-54, wherein the captured face part images can be separate classified and assigned a corresponding set of attributes and expressions is interpreted as simulating avatar training images), and wherein the machine learning model is trained using the left and right eye avatar training images and the mouth avatar training image simulated for each training image (Fig. 4A-4C, Col. 10 lines 33-40, wherein the captured face part images is used for training a machine learning module).
Regarding claim 8, Albuz in view of Miller disclose the method of claim 1. Additionally, Albuz teaches wherein the machine learning model is further trained based on the facial expressions of the facial avatars, and wherein the second stage further predicts the facial expressions from the extracted image features (Fig. 1B, Col. 6 lines 32-52, where the machine learning model is trained based on captured facial expressions, and wherein determining a set of estimated blendshape weights based on attributes of captured face parts and expressions of the input images is interpreted as predicting facial expressions based on extracted image features).
Regarding claim 10, Albuz teaches a non-transitory computer-readable data storage medium storing program code executable by a processor to perform processing comprising: capturing a set of images of a face of a wearer of a head-mountable display (HMD) using one or multiple cameras of the HMD (Fig. 3, Col. 9 line 59 – Col. 10 line 21, HMD comprising a set of cameras that captures images of the wearer; Fig. 4A-4C) applying a machine learning model trained on avatar training images corresponding to specified blendshape weights to the captured set of images to predict blendshape weights for a facial expression of the wearer of the HMD exhibited within the captured set of images (Fig. 1A, Col. 6, lines 3-31, machine learning model based on input training images and associated sets of blendshape weights, which can be used to estimate a set of blendshapes based on a captured set of images of an expression of the wearer); retargeting the predicted blendshape weights for the facial expression of the wearer of the HMD onto a facial avatar corresponding to the face of the wearer to render the facial avatar with the facial expression of the wearer (Col. 12, line 55 – Col. 13, line 2, wherein rendering and deforming an avatar according to an output set of blendshapes and their respective weights is interpreted as retargeting the predicted blendshape weights corresponding to a specific facial expression; Fig. 3, Col. 10 lines 10-21, wherein the HMD may capture a wearer’s facial expression as an input to render an avatar); and displaying the rendered facial avatar corresponding to the face of the wearer (Fig. 3, Fig. 6A-6D, Col. 10, lines 10-21, wherein the HMD renders an avatar based on input images of the wearer).
Albuz does not teach having rendered avatar training images of facial avatars having facial expressions corresponding to specified blendshape weights.
Miller teaches having rendered avatar training images of facial avatars having facial expressions corresponding to specified blendshape weights (Fig. 1, Col. 4 lines 19-47, rendering 3D avatar training images wherein the avatars have differing facial expressions corresponding to specific avatar poses, coordinates, and deformations, which is interpreted as having specified blendshape weights).
The motivation to combine would be the same as that set forth in claim 1.
Regarding claim 11, Albuz in view of Miller discloses the non-transitory computer-readable data storage medium of claim 10. Additionally, Albuz teaches wherein the set of images captured by the HMD of the face of the wearer comprises left and right eye images of facial portions of the wearer respectively including left and right eyes of the wearer and a mouth image of a lower facial portion of the wearer including a mouth of the wearer (Fig. 4A-4C, Col. 10 lines 22-32, wherein the HMD captures images of a right and left eye of the wearer, and the lower face and mouth of the user).
Regarding claim 14, Albuz teaches a head-mountable display (HMD) comprising: one or multiple cameras to capture a set of images of a face of a wearer of the HMD (Fig. 3, Col. 9 line 59 – Col. 10 line 21, HMD comprising a set of cameras that captures images of the wearer); a processor; and a memory storing program code executable by the processor (Fig. 2, Col. 8 lines 14-25) to: apply a machine learning model trained on avatar training images corresponding to specified blendshape weights to the captured set of images to predict blendshape weights for a facial expression of the wearer of the HMD exhibited within the captured set of images (Fig. 1A, Col. 6, lines 3-31, machine learning model based on input training images and associated sets of blendshape weights, which can be used to estimate a set of blendshapes based on a captured set of images of an expression of the wearer); retarget the predicted blendshape weights for the facial expression of the wearer of the HMD onto a facial avatar corresponding to the face of the wearer to render the facial avatar with the facial expression of the wearer (Col. 12, line 55 – Col. 13, line 2, wherein rendering and deforming an avatar according to an output set of blendshapes and their respective weights is interpreted as retargeting the predicted blendshape weights corresponding to a specific facial expression; Fig. 3, Col. 10 lines 10-21, wherein the HMD may capture a wearer’s facial expression as an input to render an avatar).
Albuz does not teach having rendered avatar training images of facial avatars having facial expressions corresponding to specified blendshape weights.
Miller teaches having rendered avatar training images of facial avatars having facial expressions corresponding to specified blendshape weights (Fig. 1, Col. 4 lines 19-47, rendering 3D avatar training images wherein the avatars have differing facial expressions corresponding to specific avatar poses, coordinates, and deformations, which is interpreted as having specified blendshape weights).
The motivation to combine would be the same as that set forth in claim 1.
7. Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Albuz in view of Miller as applied to claim 1 above, and further in view of Nojavanasghari (Nojavanasghari, Behnaz, Yuchi Huang, and Saad Khan. "Interactive generative adversarial networks for facial expression generation in dyadic interactions." arXiv preprint arXiv:1801.09092 (2018)), hereinafter Nojavanasghari.
Regarding claim 6, Albuz in view of Miller discloses the method of claim 1. Additionally, Nojavanasghari teaches capturing test facial training images (Fig. 6, Section 4, extracting facial features from a dataset of interviewees for training), wherein the first stage of the machine learning model is trained to extract the image features (Fig. 4, section 3.2, wherein determining facial landmarks from input images is interpreted as extracting image features) from both the rendered avatar training images and the captured test user facial training images in an adversarial training manner such that whether a given training image is of a facial avatar or of a test user cannot be predicted by more than a threshold from the extracted image features (Section 3, Fig. 3, wherein the generative adversarial network increases the probability of generated samples to resemble real data so that the discriminator cannot differentiate between generated samples and real data, which suggests ending training when the discriminator cannot predict a training image is a generated image or test image up to a threshold).
Nojavanasghari does not teach capturing test user facial training images of test users wearing HMDs
Albuz teaches capturing test user facial training images of test users wearing HMDs (Fig. 4A-4C, Col. 10 lines 33-40, wherein the images of test users wearing HMDs are used for training a machine learning module).
It would be obvious before the effective filing date of the claimed invention to have modified Albuz in view of Miller to incorporate the teachings of Nojavanasghari for this method of training this machine learning model for generating 3D avatars. Albuz discusses a machine learning model for rendering a 3D avatar based on the captured facial expressions from a user wearing an HMD that involves classifying and determining attributes of specific face parts of input images, then estimating blendshapes based on those attributes and classes, in order to render a user’s expression more accurately. Similarly, Nojavanasghari discusses a machine learning model for generating accurate facial expressions based on inputted images, in order to better display emotion and subtler expressions, while being more temporally consistent. Miller teaches a way of generating 2D images based on 3D representations, primarily user avatars, for the purposes of generating robust and multifeatured databases for training computer systems. Both Albuz and Nojavanasghari discuss machine learning systems for rendering user expressions more accurately. Additionally, since both are trained on large amounts of input data, it would be obvious to one of ordinary skill that the image database generation of Miller could be used to generate that training data. Therefore, it would be obvious to combine these references together.
8. Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Albuz in view of Miller and Nojavanasghari as applied to claim 6 above, and further in view of Sagar (US 20220092837 A1), hereinafter Sagar.
Regarding claim 7, Albuz in view of Miller and Nojavanasghari discloses the method of claim 6. Additionally, Albuz teaches wherein the second stage of the machine learning model is not trained based on the captured test user facial training images (Col. 6 line 49 – Col. 7 line 30, wherein input images for training are classified with labelled attributes and classes indicating emotion, which suggests that training is not done on input images that cannot be labelled or classified).
Albuz, Miller, and Nojavanasghari do not teach wherein the test users have facial expressions within the captured test user facial training images at unknown or unspecified blendshape weights.
Sagar teaches having test users have facial expressions within the captured test user facial training images at unknown or unspecified blendshape weights (Fig. 1A, paragraph 110-114, wherein motion capture data can be discarded if they are not based on the motion of the tracked subject, where motion capture data is used to determine blendshapes, which is interpreted as capturing and discarding user images with unknown or unspecified blendshape weights).
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Albuz in view of Miller and Nojavanasghari to incorporate the teachings of Sagar for this method of training a machine learning model for rendering 3D avatars. Albuz discusses a machine learning model for rendering and retargeting captured facial expressions to 3D avatar based by classifying and determining attributes of specific face parts of input images, then estimating blendshapes based on those attributes and classes, in order to render and retarget a user’s expression more accurately. Similarly, Nojavanasghari discusses a machine learning model for generating accurate facial expressions based on inputted images, in order to better display emotion and subtler expressions, while being more temporally consistent. While Sagar does not teach a machine learning model, it does disclose analogous art to Albuz and Nojavanasghari by discussing methods of capturing expressions and blendshapes from motion capture data, and retargeting those blendshapes to generate new animations while minimizing errors. Furthermore, Miller teaches a way of generating 2D images based on 3D representations, primarily user avatars, for the purposes of generating robust and multifeatured databases for training computer systems, which would be useful as Albuz and Nojavanasghari discuss using large amounts of input training data for the machine learning models. As Albuz and Nojavanasghari discuss machine learning models for generating and rendering facial expressions, Miller discusses generating training data for training computer models, and Sagar discusses minimizing error rates in generating facial expressions by discarding improper data, it would be obvious to combine them.
Allowable Subject Matter
9. Claims 4-5, 9, 12-13 and 15 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Response to Arguments
10. Applicant’s arguments, see pages 1-2, filed February 23rd, 2026, with respect to the 35 U.S.C. 103 rejections of claims 9 and 12 have been fully considered and are persuasive. The 35 U.S.C. 103 rejection of claims 9 and 12 has been withdrawn.
11. Applicant's arguments filed February 23rd, 2026 with respect to the 35 U.S.C. 103 rejection of claims 1-3, 6-8, 10-11, and 14 have been fully considered but they are not persuasive.
Examiner replies that, during patent examination, the pending claims must be given
their broadest reasonable interpretation consistent with the specification. See MPEP § 2111. Also, it is improper to import claim limitations from the specification. See MPEP § 2111.01(II).
Additionally, Miller clearly teaches “rendering avatar training images of facial avatars having facial expressions corresponding to specified blendshape weights”. Col. 4 lines 19-29 discusses “generating 2D imagery from a 3D avatar” via a rendering engine, which is interpreted as rendering avatar images of facial avatars, and wherein Col. 4 lines 46-47 specify that these avatar images are used to populate a database for training purposes. Additionally, Col. 4 lines 30-41 clearly specify that the “poses” of each image refer to a specific combination of rotations, translations, and deformations of an avatar, wherein the deformations may correspond to specific changes in facial expression or facial movement, and that a wide variety of those poses are uniquely rendered, as seen in Fig. 1. As each rendered image of an avatar corresponds to a specific set of deformations and changes in facial expression, it is interpreted as each rendered image corresponding to a specific set of blendshape weights. Additionally, the rendered images are explicitly stated to be used for generating an image database for training. Miller directly states in Col. 5 lines 29-32 that the image database described within can be used for training a neural network. While Miller primarily states that the image database is used for training image classification tools, it would be obvious to one of ordinary skill in the art that many more neural network systems that train on databases of images could utilize the technique described in Miller, such as with the machine learning system described in Albuz (Fig. 1B, Col. 6 lines 32-48, wherein the machine learning model utilizes a neural network, and takes in an input of thousands or millions of input images). As such, it would be obvious to combine Miller and Albuz.
In conclusion, the rejections set forth in the previous Office Action are shown to have been proper, and the claims are rejected above. To the extent that new citations and parenthetical remarks can be considered new grounds of rejection, such new grounds are necessitated by applicant’s amendments to the claim. Therefore, the present office action is made final.
Conclusion
12. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
13. Any inquiry concerning this communication or earlier communications from the examiner should be directed to JORDAN W YICK whose telephone number is (571)272-4063. The examiner can normally be reached M-F 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said Broome can be reached at (571) 272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JORDAN WAN YICK/ Examiner, Art Unit 2612
/Said Broome/ Supervisory Patent Examiner, Art Unit 2612
Prosecution Insights