Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 12/20/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
The information disclosure statement (IDS) submitted on 9/26/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Chen (US 20230260184 A1).
Regarding claim 1, Chen teaches a computer-implemented method of dynamic head generation for animation (par. 0068: “The pose values and the modified facial expression values are applied to the 3D mesh-based avatar model to generate a digital representation of the video conference participant in an avatar form. As a result, the head pose and facial expressions of the animated avatar then closely mirror the real-world physical head pose and facial expressions expressed by the video conference participant.”), comprising:
computing, by a processor, a linear blend skinning (LBS) rig associated with an avatar head based on a set of mesh deformations for a plurality of poses of the avatar head (par. 0042: “The 3D head mesh model may be rigged to use different blendshapes for natural expressions. In one embodiment, the 3D head mesh model may be rigged to use at least 51 different blendshapes.”), the LBS rig including a set of skinning joints and a set of joint weights (par. 0048: “In some embodiments, the 3D mesh-based models (e.g., in the format of FBX, OBJ, 3ds Max 2012 or Render Vray 2.3 with a textures format of PNG diffuse) may be used as the static avatars rigged using linear blend skinning with joints and bones.”), each joint weight of the set of joint weights being assigned to a vertex of a mesh of the avatar head in a neutral pose (par. 0047: “In some embodiments, to animate the 3D avatar, only expression blendshape weights w would be required (i.e., detected facial expressions).”);
computing, by the processor, an adjusted head cage for the avatar head based on a correspondence between the mesh of the avatar head in the neutral pose and a template head cage using a landmark-prediction model or a regression model (par. 0077: “The data set of ground truth data 536 may include images of human faces that are labeled and identify 2-dimensional facial landmarks in an image. Each human face in an image may include labeled facial landmarks and may be described by q.sub.i. For each facial landmark q.sub.i, the system 100 may perform the optimization process 560 as described below to derive optimal 3DMM parameters (x,y,R,T,s). The optimization process 560 may minimize the distance between projected 3D facial landmarks and input 2D landmarks according to Equation (6), where the subscript i refers to the i-th landmark on the mean face, PCA basis and expressions.”); and
animating, by the processor, the avatar head based on the LBS rig and the adjusted head cage (par. 0068: “At step 450, the system 100 generates or renders a modified video stream depicting a digital representation of the video conference participant in an animated avatar form based at least in part on the pose values and the modified facial expression values. The system 100 may use the modified facial expression values to select one or more blendshape and then apply the one or more blendshape at an associated intensity level to morph the 3D-mesh model. The pose values and the modified facial expression values are applied to the 3D mesh-based avatar model to generate a digital representation of the video conference participant in an avatar form. As a result, the head pose and facial expressions of the animated avatar then closely mirror the real-world physical head pose and facial expressions expressed by the video conference participant.”).
Regarding claim 2, Chen teaches the method of claim 1, wherein animating the avatar head comprises:
animating, by the processor, one or more facial features of the avatar head based on the LBS rig (par. 0061: “At step 360, the system 100 applies the determined one or more pose values and/or facial expression values to render an avatar model. The system 100 may apply the action unit value and corresponding intensity value pairs and/or the 3DMM parameters to render an avatar model. The system 100 may select blendshapes of the avatar model based on the determined action unit values and/or the 3DMM parameters. A 3D animation of the avatar model is then rendered using the selected blendshapes. The selected blend shapes morph or adjust the mesh geometry of the avatar model.”); and
animating, by the processor, one or more of hair or clothing of the avatar head based on the adjusted head cage (par. 0032: “The Avatar Model Customization Module 158 provides system functionality for the customization of features and/or the presented appearance of an avatar. For example, the Avatar Model Customization Module 158 provides for the selection of attributes that may be changed by a user. For example, changes to an avatar model may include hair customization, facial hair customization, glasses customization, clothing customizations, hair, skin and eye coloring changes, facial feature sizing and other customizations made be the user to a particular avatar. The changes made to the particular avatar are stored or saved in the avatar model customization repository 134.”).
Regarding claim 3, Chen teaches the method of claim 1, wherein computing the LBS rig associated with the avatar head based on the set of mesh deformations for the plurality of poses comprises:
generating, by the processor, the set of mesh deformations for the plurality of poses based on the mesh associated with the neutral pose of the avatar head and a plurality of pose vectors using a deformation-prediction model, each of the plurality of pose vectors being associated with a respective pose of the plurality of poses (par. 0051: “In step 310, a machine learning network may be trained on sets of images to determine pose values and/or facial expression parameter values. The training sets of images depict various poses of a person’s head and/or upper body, and depict various facial expressions. The various facial expressions in the images are labeled with a corresponding action number and an intensity value.”); and
performing, by the processor, a Smooth Skinning Decomposition for Rigid Bones (SSDR) procedure based on the set of mesh deformations (par. 0085: “In the pose optimization step 720, the system 100 may estimate neutral landmarks of an image based on the pose, the identity, and the expressions from the previous iteration as illustrated by Equation (14). The estimation and use of neutral landmarks is further described below in reference to FIG. 8. The system 100 may construct a (2n.sub.1+6)×8 matrix A.sub.p, and a (2n.sub.1+6)×1 matrix b.sub.p as illustrated by Equation (15), where 6×8 matrix A.sub.λ, 6×1 matrix b.sub.λ, 2n.sub.1×8 matrix A.sub.F, and 2n.sub.1×1 matrix b.sub.F are defined according to Equation (16) and Equation (17). The system 100 may solve linear equations A.sub.pZ=b.sub.p to determine Equation (18). The system 100 may construct a 3×3 matrix, and apply singular value decomposition (SVD) onto the constructed matrix to obtain matrix U, V according to Equation (19). The system 100 may derive the optimized pose according to Equation (20), with the simplified pose being illustrated by Equation (21). The simplified pose is to be used in Equations (23, 26 and 27).”).
Regarding claim 4, Chen teaches the method of claim 3, wherein computing the LBS rig associated with the avatar head based on the set of mesh deformations for the plurality of poses comprises:
detecting, by the processor, at least one collision between an external surface of the avatar head and internal features of the avatar head based on the set of mesh deformations and the mesh associated with the neutral pose (par. 0042: “In one embodiment, the 3D head mesh model may be rigged to use at least 51 different blendshapes. Also, the 3D head mesh model may have an associated tongue model. The system 100 may detect tongue out positions in an image and render the avatar model depicting a tongue out animation.” NOTE: A tongue is an internal feature of the avatar head; an animation comprising sticking one’s tongue out would require collision physics between internal and external features of the avatar head.); and
adjusting, by the processor, the set of mesh deformations based on the at least one collision, the set of mesh deformations being adjusted with respect of a placement of the internal features in relation to the external surface of the avatar head, and wherein the set of mesh deformations are adjusted prior to performing the SSDR procedure (par. 0042: “The system 100 generates from a 3D mesh-based model, a digital representation of a video conference participant in an avatar form. The avatar model may be a mesh-based 3D model. In some embodiments, a separate avatar head mesh model and a separate body mesh model may be used. The 3D head mesh model may be rigged to use different blendshapes for natural expressions.”).
Regarding claim 5, Chen teaches the method of claim 4, wherein:
detecting the at least one collision between the external surface of the avatar head and the internal features of the avatar head based on the set of mesh deformations and the mesh associated with the neutral pose comprises:
identifying, by the processor, the external surface and the internal features of the avatar head in the neutral pose based on the mesh of the avatar head in the neutral pose (par. 0045: “In some embodiments, the system 100 uses a 3D morphable model (3DMM) to generate rigged avatar models. For example, the following 3DMM may be used to represent a user’s face with expressions: v=m+Pα+Bw, where m is the neutral face, P is the face shape basis and B is the blendshape basis. The neutral face and face shape basis are created from 3D scan data (3DFE/4DFE) using non-rigid registration techniques.”);
identifying, by the processor, a first plurality of depth values associated with the external surface of the avatar head in each of the plurality of poses (par. 0093: “The system 100 may perform additional stabilization processing for pose optimization. Where ground-truth pose (α,β,γ,t.sub.x,t.sub.y,s) is optimized such that (α,β,γ,s) is close to (0,0,0,s.sub.m), the ML Network 516 may learn the same way when inferencing poses for two neighboring frames. As such, the system 100 may improve tracking smoothness and consistency.”);
identifying, by the processor, a second plurality of depth values associated with the internal features of the avatar head in each of the plurality of poses (par. 0051: “In step 310, a machine learning network may be trained on sets of images to determine pose values and/or facial expression parameter values. The training sets of images depict various poses of a person’s head and/or upper body, and depict various facial expressions. The various facial expressions in the images are labeled with a corresponding action number and an intensity value.”); and
determining, by the processor, the at least one collision when at least one of the second plurality of depth values for a pose is greater than a corresponding at least one of the first plurality of depth values of the pose (par. 0095: “The system 100 may project or determine 3D landmarks (e.g., i.sub.0,i.sub.1 3D projected landmarks) corresponding to the 2D landmarks. The projected 3D landmarks may have a distance greater between i.sub.0 and i.sub.1, than i.sub.0 and i.sub.1 of the 2D landmarks. As a result, the gap between the two projected 3D landmarks (i.sub.0, i.sub.1) may increase, leading to an inaccurate expression. In other words, the retargeting may not depict the avatar with its eyes closed.”), and
adjusting the set of mesh deformations comprises:
modifying the at least one of the second plurality of depth values for the pose to be less than the corresponding at least one of the first plurality of depth values for the pose (par. 0096: “To improve the result of the expression retargeting for the eyes and/or mouth, the system 100 may add a distance constraint as described by Equation (45). In rendering the avatar, the tiny gap between the two 2D landmarks (i.sub.0, i.sub.1) may prevent the eyes from closing completely. The system 100 may use different distance constraints for eye regions and mouth regions. For eye regions, a tiny gap between 2D landmark pairs may be removed to make the eye close completely. For the mouth region, the tiny gaps between 2D landmark pairs for mouth expressions may be controlled via a predetermined graph or scale.”).
Regarding claim 6, Chen teaches the method of claim 3, wherein performing the SSDR procedure to compute the LBS rig based on the set of mesh deformations comprises:
computing, by the processor, a set of rigid joint transforms for each of the plurality of poses, wherein a set of input skinning weights are unchanged during the computation of the set of rigid joint transforms (par. 0048: “In some embodiments, the 3D mesh-based models (e.g., in the format of FBX, OBJ, 3ds Max 2012 or Render Vray 2.3 with a textures format of PNG diffuse) may be used as the static avatars rigged using linear blend skinning with joints and bones.”); and
computing, by the processor, a set of final skinning weights for each of the plurality of poses while holding the set of rigid joint transforms constant, wherein the LBS rig includes the set of rigid joint transforms and the set of final skinning weights (par. 0046: “The face shape basis P may be computed using principal component analysis (PCA) on the face meshes. PCA will result in principal component vectors which correspond to the features of the image data set. The blendshape basis B may be derived from the open-source project ICT-FaceKit. The ICT-FaceKit provides a base topology with definitions of facial landmarks, rigid and morphable vertices. The ICT-FaceKit provides a set of linear shape vectors in the form of principal components of light stage scan data registered to a common topology.”).
Regarding claim 7, Chen teaches the method of claim 1, wherein computing the adjusted head cage for the avatar head based on the correspondence between the mesh of the avatar head in the neutral pose and a template head cage comprises:
identifying, by the processor, a correspondence between the mesh of the avatar head in the neutral pose and an initial head cage (par. 0041: “In some embodiments, the system 100 may determine multiple different facial expressions or actions values for an evaluated image. The system 100 may include in the package, a corresponding blendshape for each of the multiple different facial expressions that may be identified by the system.”);
computing, by the processor, an adjusted head cage for the avatar head based on the correspondence (par. 0041: “The system 100 may use the different blendshapes to adjust or deform the 3D mesh-based model (e.g., the head mesh model) when rendering a digital representation of a Video Conference Participant 226 in avatar form.”); and
aligning, by the processor, the adjusted head cage to the mesh associated with the neutral pose of the avatar head (par. 0092: “During pose optimization in Equation (7), the pose in an image may be optimized to achieve the best fitting of projecting 3D landmarks F.sub.i to 2D landmarks q.sub.i. As illustrated, the 2D landmarks 822 and the 2D landmarks 842 change significantly in their position for the 2D landmarks as between image 820 and image 840. In this situation, with a significant distance in the positions of the 2D landmarks from image 820 to image 840, using the optimizing Equation (7) may not provide ideal results. To address this situation, the system 100 may estimate neutral 2D landmarks for each face (such as the neutral 2D landmarks 824 for the mouth open position and with neutral 2D landmarks 844 for the closed mouth position)). This allows the system 100 to reduce the differences in the 2D landmarks (as depicted in 2D landmarks 850), and as such, the system’s optimization of the pose would be more stable.”).
Claim 8 is substantially similar to claim 1, except that it teaches a device rather than a method. It is therefore rejected on a similar basis to claim 1.
Claim 9 is substantially similar to claim 2, except that it depends from claim 8 instead of claim 1. It is therefore rejected on a similar basis to claim 2.
Claim 10 is substantially similar to claim 3, except that it depends from claim 8 instead of claim 1. It is therefore rejected on a similar basis to claim 3.
Claim 11 is substantially similar to claim 4, except that it depends from claim 10 instead of claim 3. It is therefore rejected on a similar basis to claim 4.
Claim 12 is substantially similar to claim 5, except that it depends from claim 11 instead of claim 4. It is therefore rejected on a similar basis to claim 5.
Claim 13 is substantially similar to claim 6, except that it depends from claim 10 instead of claim 3. It is therefore rejected on a similar basis to claim 6.
Claim 14 is substantially similar to claim 7, except that it depends from claim 8 instead of claim 1. It is therefore rejected on a similar basis to claim 7.
Claim 15 is substantially similar to claim 1, except that it teaches a non-transitory computer-readable medium storing instructions, rather than a method. It is therefore rejected on a similar basis to claim 1.
Claim 16 is substantially similar to claim 3, except that it depends from claim 15 instead of claim 1. It is therefore rejected on a similar basis to claim 3.
Claim 17 is substantially similar to claim 4, except that it depends from claim 16 instead of claim 3. It is therefore rejected on a similar basis to claim 4.
Claim 18 is substantially similar to claim 5, except that it depends from claim 17 instead of claim 4. It is therefore rejected on a similar basis to claim 5.
Claim 19 is substantially similar to claim 6, except that it depends from claim 16 instead of claim 3. It is therefore rejected on a similar basis to claim 6.
Claim 20 is substantially similar to claim 7, except that it depends from claim 15 instead of claim 1. It is therefore rejected on a similar basis to claim 7.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN A BARHAM whose telephone number is (571)272-4338. The examiner can normally be reached Mon-Fri, 8:30am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu, can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN ALLEN BARHAM/Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights