DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/16/2026 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement has been considered by the examiner.
Response to Amendment
1 This action is in response to the amendment filed on 3/30/2026. Claims 1-14 have been amended, and claims 16-20 are new additions. Claims 1-15 remain rejected, and claims 16-20 are rejected.
Response to Arguments
2 Applicant’s arguments with respect to claim 1 and 14-15 filed on 3/30/2026, with respect to the rejection under 35 U.S.C. § 102 regarding that the prior art does not teach the following but not limited to “trained machine learning model is configured to predict data representing predicted movement of the user; …wherein the input for varying the degree of the data is for varying a feature or a characteristic movement of an animation”. This argument has been considered, but are moot due to more clarification surrounding the previously used prior art, which is referenced in the rejections below to go along with the amended parts of the claim. For further clarification, the prior art used for the rejection references a deep learning technique (which can be used for machine learning), that can process content data (WO 2022103877 A1; [0025] reciting “In some embodiments, deep learning techniques are applied in the data processing environment 100 to process content data (e.g., video data, visual data, audio data) obtained by an application executed at a client device 104 to identify information contained in the content data, match the content data with other data, categorize the content data, or synthesize related content data.”).
3 Regarding claims 2-13, they directly/indirectly depend on independent claim 1 respectively. Applicant does not argue anything other than independent claims 1 and 14-15. The limitations in those claims, in conjunction with combination, was mostly previously established as explained, with a few changes surrounding the amended dependent claims.
4 Claims 16-20 are new additions as mentioned previously, and are dependent of independent claim 15. However, they are also rejected under 35 U.S.C. § 103.
Claim Rejections - 35 USC § 102
5 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
6 The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
7 Claim(s) 1-3, 6-7, 9-12, and 14-15 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Liu et al. (WO 2022103877 A1).
8 Regarding claim 1, Liu teaches a computer-implemented method comprising:
receiving, from a user device, video data of a user, the video data comprising audio data and image data corresponding to the audio data ([0023] reciting “While the video data is optionally pre-processed on the surveillance camera 104E, the surveillance camera server 102 processes the video data to identify motion or audio events in the video data and share information of these events with the mobile phone 104C, thereby allowing a user of the mobile phone 104 to monitor the events occurring near the networked surveillance camera 104E in the real time and remotely.”; [Abstract] reciting “Additionally, audio-based face parameters are extracted from the audio data, independently of the image. In accordance with the shape parameters, expression parameters, color texture map, displacement map, and audio-based face parameters, the computer system renders the 3D avatar of the person in a video clip in which the 3D avatar is animated for an audio activity synchronous with the audio data.”; [0030] reciting “Model training module 226 for receiving training data and establishing a data processing model for processing content data (e.g., video, image, audio, or textual data) to be collected or obtained by a client device 104”);
training a first machine learning model based on the video data ([0025] reciting “In some embodiments, deep learning techniques are applied in the data processing environment 100 to process content data (e.g., video data, visual data, audio data) obtained by an application executed at a client device 104 to identify information contained in the content data, match the content data with other data, categorize the content data, or synthesize related content data… In these deep learning techniques, data processing models are created based on one or more neural networks to process the content data.”; [0033] reciting “Figure 3 is another example data processing system 300 for training and applying a neural network based (NN-based) data processing model 240 for processing content data (e.g., video, image, audio, or textual data), in accordance with some embodiments.”), thereby resulting in a second, trained machine learning model, the second, trained machine learning model being personalized to the user ([0004] reciting “Accordingly, there is a need for an efficient 3D avatar driving mechanism for creating a 3D personalized avatar from a two-dimensional (2D) image and driving the 3D personalized avatar in synchronization with independent audio data…. such a 3D avatar driving mechanism is implemented by a neural network model that is optimized for a mobile device having limited computational resources.”; [0028] reciting “As explained above, in some embodiments, deep learning techniques are applied in the data processing environment 100 to process video data, static image data, or inertial sensor data captured by the AR glasses 104D. 2D or 3D device poses are recognized and predicted based on such video, static image, and/or inertial sensor data using a first data processing model. Visual content is optionally generated using a second data processing model.”), wherein the second, trained machine learning model is configured to predict data representing predicted movement of the user ([0042] reciting “The avatar generation model 500 includes a coarse reconstruction network (CRN) 508, a fine reconstruction network (FRN) 510, an audio-face neural network 512, and an audio-driven 3D avatar head network 514. These networks 508-514 are configured to process the image 504 and audio data 506 jointly to personalize the 3D avatar 502 and animate the 3D avatar 502 in synchronization with the audio data 506. Specifically, the 3D avatar 502 is animated with head, mouth, eye, and/or facial muscle movements. There movements are synchronized with and dynamically controlled based on the audio data”; [0052] reciting “The mouth keypoint loss 610A indicates a difference of physical locations between predicted mouth keypoints 612 and ground-truth keypoints, and the mouth rendering loss 610B indicates a color difference between a rendered mouth region 618 of a predicted face and the ground-truth mouth region. By optimizing these two mouth losses 610A and 610B, the CRN 508 is adjusted to refine the face parameters 516 around the mouth region iteratively, and the result face parameters 516 can be applied to reconstruct complex lip movement on a human head model.”; [0066] reciting “Lip movement prediction from the audio data 506 is easy to use and has natural results.”);
receiving further audio data from the user device ([0030] reciting “Model training module 226 for receiving training data and establishing a data processing model for processing content data (e.g., video, image, audio, or textual data) to be collected or obtained by a client device 104”; [0042] reciting “The avatar generation model 500 receives the image 504 and audio data 506, and outputs the animated 3D avatar 502.”);
inputting the further audio data into the second, trained machine learning model thereby resulting in the data representing the predicted movements of the user ([0030] reciting “User interface module 218 for enabling presentation of information (e.g., a graphical user interface for application(s) 224, widgets, websites and web pages thereof, and/or games, audio and/or video content, text, etc.) at each client device 104 via one or more output devices 212 (e.g., displays, speakers, etc.)…Model training module 226 for receiving training data and establishing a data processing model for processing content data (e.g., video, image, audio, or textual data) to be collected or obtained by a client device 104”; [0052] reciting “The mouth keypoint loss 610A indicates a difference of physical locations between predicted mouth keypoints 612 and ground-truth keypoints, and the mouth rendering loss 610B indicates a color difference between a rendered mouth region 618 of a predicted face and the ground-truth mouth region. By optimizing these two mouth losses 610A and 610B, the CRN 508 is adjusted to refine the face parameters 516 around the mouth region iteratively, and the result face parameters 516 can be applied to reconstruct complex lip movement on a human head model.”; [0054] reciting “The second audio-face neural network 804 is configured to generate a plurality of face parameters 810 from the audio data 506. The plurality of face parameters 810 include one or more shape parameters describe a shape of a face or one or more expression parameters describing an expression of the face at the time of generating the audio data 506.”; [0066] reciting “Lip movement prediction from the audio data 506 is easy to use and has natural results.”);
receiving an input from the user device varying a degree of the data representing the predicted movements of the user thereby resulting in customized movements for the user, wherein the input for varying the degree of the data is for varying a feature or a characteristic movement of an animation ([0042] reciting “These networks 508-514 are configured to process the image 504 and audio data 506 jointly to personalize the 3D avatar 502 and animate the 3D avatar 502 in synchronization with the audio data 506. Specifically, the 3D avatar 502 is animated with head, mouth, eye, and/or facial muscle movements. There movements are synchronized with and dynamically controlled based on the audio data 506, i.e., each movement vary dynamically and in real time with one or more of content, a volume and a pitch of a voice, a speech rate, and other characteristics of the audio data 506.” ; [0066] reciting “Lip movement prediction from the audio data 506 is easy to use and has natural results.”); and
generating the animation of an avatar of the user using the customized movements for the user ([Abstract] reciting “This application is directed to generation of a 3D avatar that is animated in synchronization with audio data.”).
9 Regarding claim 2, Liu teaches the method according to claim 1 (see claim 1 rejection above), wherein the predicted movements of the user comprise at least one of: a lip movement of the user, a change in head pose of the user, and a change in facial expression of the user ([0005] reciting “The audio activity includes at least lip movement. It is noted that animation of the 3D avatar is not limited to a mouth region and involves movements of one or more of a head, facial expression, mouth, hair, or other regions of the 3D avatar.”; [0052] reciting “The mouth keypoint loss 610A indicates a difference of physical locations between predicted mouth keypoints 612 and ground-truth keypoints, and the mouth rendering loss 610B indicates a color difference between a rendered mouth region 618 of a predicted face and the ground-truth mouth region.”).
10 Regarding claim 3, Lui teaches the method according to claim 1 (see claim 1 rejection above), comprising: communicating with a further user device, wherein the animation of the avatar is used during the communicating with the further user device ([0010] reciting “Figure 1 is an example data processing environment having one or more servers communicatively coupled to one or more client devices, in accordance with some embodiments.”; [0030] reciting “…an audio-face neural network 512, and an audio-driven 3D avatar head network 514, and is applied to render a 3D avatar of a person in a video clip in which the 3D avatar is animated for an audio activity synchronous with audio data, e.g., in Figure 5; and o Content data and results 242 that are obtained by and outputted to the client device 104 of the data processing system 200, respectively…”).
11 Regarding claim 6, Liu teaches the method according to claim 1 (see claim 1 rejection above), wherein the video data comprises more than one video of the user ([0052] reciting “Referring to Figure 6C, in some embodiments, a training dataset 608 includes a number of interview videos, and image frames that contain lip movement are extracted from these interview videos.”).
12 Regarding claim 7, Liu teaches the method according to claim 1 (see claim 1 rejection above), wherein the method comprises: generating a first persona of the user based on the predicted movements and based on a first customization of the animation of the avatar from the user device ([0042] reciting “The avatar generation model 500 includes a coarse reconstruction network (CRN) 508, a fine reconstruction network (FRN) 510, an audio-face neural network 512, and an audio-driven 3D avatar head network 514. These networks 508-514 are configured to process the image 504 and audio data 506 jointly to personalize the 3D avatar 502 and animate the 3D avatar 502 in synchronization with the audio data 506. Specifically, the 3D avatar 502 is animated with head, mouth, eye, and/or facial muscle movements. There movements are synchronized with and dynamically controlled based on the audio data 506, i.e., each movement vary dynamically and in real time with one or more of content, a volume and a pitch of a voice, a speech rate, and other characteristics of the audio data 506.”; [0048] reciting “In some embodiments, those effects are generated by a graphics processing unit (GPU). In various embodiments of this application, the avatar Tenderer 530 is configured to reduce a computational cost of the plurality of human head related visual effects and enable a subset or all of the plurality of human head visual effects on a mobile device (e.g., a mobile phone 104C).”);
generating a second persona of the user ([0005] reciting “The plurality of face parameters includes a first set of shape parameters describing a shape of the face and a second set of expression parameters describing an expression of the face.”; [0028] reciting “Visual content is optionally generated using a second data processing model. Training of the first and second data processing models is optionally implemented by the server 102 or AR glasses 104D.”) based on at least one of: the predicted movements based on the video data of the user and a second customization of the animation of the avatar from the user device, different predicted movements of the user using second video data different from the video data ([0043] reciting “The first set of shape parameters 518 describe a shape of the face of the person in the 2D image 504, and do not change with time. The second set of expression parameters 520 describe an expression of the face, and temporally varies with the person’s activity…By these means, the 3D face model is suitable for animating the 3D avatar 502 for an audio activity (e.g., talking, singing, laughing), and the audio activity includes different movements (e.g., head, facial muscle, eye, and mouth movement) of the 3D avatar 502 which are synchronous with the audio data 506.”), and different predicted movements of the user using third video data of the user and a third customization of the animation of the avatar from the user device, the third video data being different from the video data;
storing the second persona of the user ([0026] reciting “The trained data processing models are optionally stored in the server 102B or storage 106.”); and
providing an option to the user to select the first persona of the user or the second persona of the user to provide animation of the avatar ([0027] reciting “In some embodiments, the display of the AR glasses 104D displays a user interface, and the recognized or predicted device poses are used to render or interact with user selectable display items (e.g., an avatar) on the user interface.”).
13 Regarding claim 9, Liu teaches the method according to claim 1 (see claim 1 rejection above), wherein the method comprises: determining a representation having a similar appearance to the user; basing the appearance of the avatar on the representation ([Abstract] reciting “A computer system generates face parameters of a face associated with the person from an image…The computer system generates a color texture map and a displacement map of a 3D face model of the face associated with the person based on the face parameters.”).
14 Regarding claim 10, Liu teaches the method according to claim 1 (see claim 1 rejection above), wherein a training dataset comprises two or more videos for each of a plurality of users, each of the videos having at least one labelled vertex of a head of the respective user ([0043] reciting “By these means, the 3D face model is suitable for animating the 3D avatar 502 for an audio activity (e.g., talking, singing, laughing), and the audio activity includes different movements (e.g., head, facial muscle, eye, and mouth movement) of the 3D avatar 502 which are synchronous with the audio data 506.”; [0044] reciting “The CRN 508 further includes a graph convolutional network (GCN) to predict a per-vertex color of each vertex of the mesh of the 3D face model.”; [0057] reciting “Further, in some embodiments, the 3D face model includes a plurality of vertices, and the first reconstruction network includes a graph convolutional network (GCN) configured to predict a color for each vertex of the 3D face model”), the method comprising:
i) training a third machine learning model based on at least one video of a user of the plurality of users to provide a fourth machine learning model ([0047] reciting “During the course of training the audio-face neural network 512, a third training dataset includes third data pairs of training audio data and related training face parameters, and is used to train the audio-face neural network 512.”);
ii) predicting head movements of the user of the plurality of users based on at least one portion of audio data and the fourth machine learning model, wherein each of the at least one portion of audio data has a corresponding video ([0004] reciting “The set of animation parameters are predicted from an audio sequence of human voice speaking or singing, and applied to drive and animate the 3D head model. Additionally, the 3D head model is rendered with photo-realistic facial features.”);
iii) computing, using the predicted head movements and the at least one labelled vertex of the head of the user in the corresponding video for each of the at least one portion of audio data, error for the predicted head movements for the at least one portion of audio data ([0036] reciting “…those of the pre- processing modules 308 and covert the content data to a predefined content format that is acceptable by inputs of the model -based processing module 316. Examples of the content data include one or more of: video, image, audio, textual, and other types of data…The model -based processing module 316 can also monitor an error indicator to determine whether the content data has been properly processed in the data processing model 240.”);
iv) updating parameters of the third machine learning model by backpropagating the error for the predicted head movements of the user of the plurality of users ([0041] reciting “In the backward propagation, a margin of error of the output (e.g., a loss function) is measured, and the weights are adjusted accordingly to decrease the error.”);
wherein the method comprises: repeating operations i) to iv) for a random sample of the plurality of users until determining that the error has converged from one user to the next user in the sample ([0034] reciting “The model training engine 310 modifies the data processing model 240 to reduce the loss function, until the loss function satisfies a loss criteria (e.g., a comparison result of the loss function is minimized or reduced below a loss threshold). The modified data processing model 240 is provided to the data processing module 228 to process the content data.”; [0041] reciting “The training process typically includes two steps, forward propagation and backward propagation, which are repeated multiple times until a predefined convergence condition is satisfied.”); and
upon determining that the error has converged, using the third machine learning model as the first machine learning model ([0041] reciting “The training process typically includes two steps, forward propagation and backward propagation, which are repeated multiple times until a predefined convergence condition is satisfied.”; [0062] reciting “In some embodiments, a first training image 606A, a second training image 606B, and a third training image 606C are applied to train the first reconstruction network.”).
15 Regarding claim 11, Liu teaches the method according to claim 10 (see claims 1 and 10 rejections above), wherein at least one of the first machine learning model, the second, trained machine learning model, the third machine learning model and the fourth machine learning model comprises at least one of: a convolutional neural network or a sequential neural network configured to predict a change in head pose of the user, a change in expression of the user, and a lip movement of the user ([0044] reciting “In some embodiments not shown in Figure 5, the CRN 508 includes a convolutional neural network (CNN) configured to regress the face parameters 516 from the 2D image 504. A face differentiable module is optionally coupled to the CNN and configured to utilize a pixel color distribution of the 2D image 504 to regulate the CNN. Further, in some embodiments, a 3D face model includes a mesh, and a topology of the mesh is assumed to be constant. The CRN 508 further includes a graph convolutional network (GCN) to predict a per-vertex color of each vertex of the mesh of the 3D face model.”; [0057] reciting “The computer system generates (906), from the 2D image 504, a plurality of face parameters 516 of a face associated with the person. The plurality of face parameters 516 includes (908) a first set of shape parameters 518 describing a shape of the face and a second set of expression parameters 520 describing an expression of the face. In some embodiments, the plurality of face parameters 516 have (910) a total number of face parameters among which a first number of face parameters describing a mouth region of the person, and a ratio of the first number and the total number exceeds a predefined threshold ratio. In some embodiments, the plurality of face parameters 516 of the face are generated from the 2D image using a first reconstruction network (e.g., a CRN 508), and the first reconstruction network includes a convolutional neural network (CNN).”).
16 Regarding claim 12, Liu teaches the method according to claim 1 (see claim 1 rejection above), wherein the first machine learning model comprises at least one of: a convolutional neural network or a sequential neural network configured to operate on audio data to predict a change in head pose of the user, a change in expression of the user, and a lip movement of the user ([0044] reciting “In some embodiments not shown in Figure 5, the CRN 508 includes a convolutional neural network (CNN) configured to regress the face parameters 516 from the 2D image 504. A face differentiable module is optionally coupled to the CNN and configured to utilize a pixel color distribution of the 2D image 504 to regulate the CNN. Further, in some embodiments, a 3D face model includes a mesh, and a topology of the mesh is assumed to be constant. The CRN 508 further includes a graph convolutional network (GCN) to predict a per-vertex color of each vertex of the mesh of the 3D face model.”; [0057] reciting “The computer system generates (906), from the 2D image 504, a plurality of face parameters 516 of a face associated with the person. The plurality of face parameters 516 includes (908) a first set of shape parameters 518 describing a shape of the face and a second set of expression parameters 520 describing an expression of the face. In some embodiments, the plurality of face parameters 516 have (910) a total number of face parameters among which a first number of face parameters describing a mouth region of the person, and a ratio of the first number and the total number exceeds a predefined threshold ratio. In some embodiments, the plurality of face parameters 516 of the face are generated from the 2D image using a first reconstruction network (e.g., a CRN 508), and the first reconstruction network includes a convolutional neural network (CNN).”).
17 Claim 14 has similar limitations as of claim 1, therefore it is rejected under the same rationale as claim 1.
18 Regarding claim 15, Liu teaches a computer-readable storage device comprising instructions executable by a processor for ([0055] reciting “The computer readable storage medium may include a magnetic or optical disk storage device, solid state storage devices such as Flash memory, or other non-volatile memory device or devices.”):
receiving, from a user device, video data of a user, the video data comprising audio data and image data corresponding to the audio data ([0023] reciting “While the video data is optionally pre-processed on the surveillance camera 104E, the surveillance camera server 102 processes the video data to identify motion or audio events in the video data and share information of these events with the mobile phone 104C, thereby allowing a user of the mobile phone 104 to monitor the events occurring near the networked surveillance camera 104E in the real time and remotely.”; [Abstract] reciting “Additionally, audio-based face parameters are extracted from the audio data, independently of the image. In accordance with the shape parameters, expression parameters, color texture map, displacement map, and audio-based face parameters, the computer system renders the 3D avatar of the person in a video clip in which the 3D avatar is animated for an audio activity synchronous with the audio data.”; [0030] reciting “Model training module 226 for receiving training data and establishing a data processing model for processing content data (e.g., video, image, audio, or textual data) to be collected or obtained by a client device 104”);
training a first machine learning model based on the video data ([0025] reciting “In some embodiments, deep learning techniques are applied in the data processing environment 100 to process content data (e.g., video data, visual data, audio data) obtained by an application executed at a client device 104 to identify information contained in the content data, match the content data with other data, categorize the content data, or synthesize related content data… In these deep learning techniques, data processing models are created based on one or more neural networks to process the content data.”; [0033] reciting “Figure 3 is another example data processing system 300 for training and applying a neural network based (NN-based) data processing model 240 for processing content data (e.g., video, image, audio, or textual data), in accordance with some embodiments.”), thereby resulting in a second, trained machine learning model, the second, trained machine learning model being personalized to the user ([0004] reciting “Accordingly, there is a need for an efficient 3D avatar driving mechanism for creating a 3D personalized avatar from a two-dimensional (2D) image and driving the 3D personalized avatar in synchronization with independent audio data…. such a 3D avatar driving mechanism is implemented by a neural network model that is optimized for a mobile device having limited computational resources.”; [0028] reciting “As explained above, in some embodiments, deep learning techniques are applied in the data processing environment 100 to process video data, static image data, or inertial sensor data captured by the AR glasses 104D. 2D or 3D device poses are recognized and predicted based on such video, static image, and/or inertial sensor data using a first data processing model. Visual content is optionally generated using a second data processing model.”), wherein the second, trained machine learning model is able to predict movement of the user ([0042] reciting “The avatar generation model 500 includes a coarse reconstruction network (CRN) 508, a fine reconstruction network (FRN) 510, an audio-face neural network 512, and an audio-driven 3D avatar head network 514. These networks 508-514 are configured to process the image 504 and audio data 506 jointly to personalize the 3D avatar 502 and animate the 3D avatar 502 in synchronization with the audio data 506. Specifically, the 3D avatar 502 is animated with head, mouth, eye, and/or facial muscle movements. There movements are synchronized with and dynamically controlled based on the audio data”; [0052] reciting “The mouth keypoint loss 610A indicates a difference of physical locations between predicted mouth keypoints 612 and ground-truth keypoints, and the mouth rendering loss 610B indicates a color difference between a rendered mouth region 618 of a predicted face and the ground-truth mouth region. By optimizing these two mouth losses 610A and 610B, the CRN 508 is adjusted to refine the face parameters 516 around the mouth region iteratively, and the result face parameters 516 can be applied to reconstruct complex lip movement on a human head model.”; [0066] reciting “Lip movement prediction from the audio data 506 is easy to use and has natural results.”);
receiving further audio data from the user device ([0030] reciting “Model training module 226 for receiving training data and establishing a data processing model for processing content data (e.g., video, image, audio, or textual data) to be collected or obtained by a client device 104”; [0042] reciting “The avatar generation model 500 receives the image 504 and audio data 506, and outputs the animated 3D avatar 502.”);
inputting the further audio data into the second, trained machine learning model thereby resulting in predicted movements of the user ([0030] reciting “User interface module 218 for enabling presentation of information (e.g., a graphical user interface for application(s) 224, widgets, websites and web pages thereof, and/or games, audio and/or video content, text, etc.) at each client device 104 via one or more output devices 212 (e.g., displays, speakers, etc.)…Model training module 226 for receiving training data and establishing a data processing model for processing content data (e.g., video, image, audio, or textual data) to be collected or obtained by a client device 104”; [0052] reciting “The mouth keypoint loss 610A indicates a difference of physical locations between predicted mouth keypoints 612 and ground-truth keypoints, and the mouth rendering loss 610B indicates a color difference between a rendered mouth region 618 of a predicted face and the ground-truth mouth region. By optimizing these two mouth losses 610A and 610B, the CRN 508 is adjusted to refine the face parameters 516 around the mouth region iteratively, and the result face parameters 516 can be applied to reconstruct complex lip movement on a human head model.”; [0054] reciting “The second audio-face neural network 804 is configured to generate a plurality of face parameters 810 from the audio data 506. The plurality of face parameters 810 include one or more shape parameters describe a shape of a face or one or more expression parameters describing an expression of the face at the time of generating the audio data 506.”; [0066] reciting “Lip movement prediction from the audio data 506 is easy to use and has natural results.”);
receiving at least one input from the user device varying a degree of at least one of the predicted movements of the user thereby resulting in customized movements for the user ([0042] reciting “These networks 508-514 are configured to process the image 504 and audio data 506 jointly to personalize the 3D avatar 502 and animate the 3D avatar 502 in synchronization with the audio data 506. Specifically, the 3D avatar 502 is animated with head, mouth, eye, and/or facial muscle movements. There movements are synchronized with and dynamically controlled based on the audio data 506, i.e., each movement vary dynamically and in real time with one or more of content, a volume and a pitch of a voice, a speech rate, and other characteristics of the audio data 506.” ; [0066] reciting “Lip movement prediction from the audio data 506 is easy to use and has natural results.”); and
generating animation of an avatar of the user using the customized movements for the user ([Abstract] reciting “This application is directed to generation of a 3D avatar that is animated in synchronization with audio data.”).
Claim Rejections - 35 USC § 103
19 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
20 The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
21 Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al. (WO 2022103877 A1) in view of Sachs et al. (US 20190122411 A1).
22 Regarding claim 4, Liu teaches the method according to claim 1 (see claim 1 rejection above), but does not explicitly teach wherein training the first machine learning model based on the video data is performed, at least in part, by a cloud computing device or the user device.
23 Sachs teaches wherein training the first machine learning model based on the video data is performed, at least in part, by a cloud computing device or the user device ([0073] reciting “When the 3D mesh is rigged for animation, the 3D model can be referred to or animation ready. In several embodiments, the system can also animate the rigged or orientation-ready 3D model of the head mapping to rig parameters audio samples and/or video data. In accordance with the some other embodiments, the processes are performed by a “cloud” server system, a user device, and/or combination of devices local and/or remote from a user.”).
24 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Liu) to incorporate the teachings of Sachs to provide a cloud computing device for the 3d head models taught by Liu. Doing so would allow one or more server systems to provide data and/or executable applications to devices over a network as stated by Sachs ([0083] recited).
25 Claim(s) 5 and 16-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al. (WO 2022103877 A1) in view of Assouline et al. (US 20220157000 A1).
26 Regarding claim 5, Liu teaches the method according to claim 1 (see claim 1 rejection above), wherein the input from the user device for varying the degree of the data representing the predicted movement of the user is received ([0042] reciting “These networks 508-514 are configured to process the image 504 and audio data 506 jointly to personalize the 3D avatar 502 and animate the 3D avatar 502 in synchronization with the audio data 506. Specifically, the 3D avatar 502 is animated with head, mouth, eye, and/or facial muscle movements. There movements are synchronized with and dynamically controlled based on the audio data 506, i.e., each movement vary dynamically and in real time with one or more of content, a volume and a pitch of a voice, a speech rate, and other characteristics of the audio data 506.” ; [0066] reciting “Lip movement prediction from the audio data 506 is easy to use and has natural results.”)
27 Liu does not explicitly teach … via a user interface control to increase or decrease at least one of: a movement vector for the predicted movement or a frequency with which the predicted movement occurs.
28 Assouline teaches … via a user interface control to increase or decrease at least one of: a movement vector for the predicted movement or a frequency with which the predicted movement occurs ([0042] reciting “Namely, the skeletal rig of the selected second 3D avatar is modified based on the joint movements, speed, direction, positions and acceleration specified in the movement vector. This way, the second user can apply movements or motion recorded by a first user to an avatar selected by the second user. The second user can also modify the motion or movement vector to increase or decrease the amount or duration of movements specified in the movement vector.”; [0109] reciting “In some embodiments, the augmentation system 208 is trained using a machine learning technique to predict or estimate a position on the real-world object that is associated with the least movement or noise.”; [0161] reciting “As shown in FIG. 10, a graphical user interface 1000 is presented on a display of the client device 102 of the second user, in which a real-world object 1002 (e.g., a person or user) is shown together with a virtual object 1001 (e.g., a 3D avatar)… his way, as the user performs movements, the virtual object 1001 is moved in the same manner so the user can visualize what the virtual object 1001 looks like when mimicking the user's movements. The second user can control when the movements the user performs are stored in a movement vector to subsequently loop animation of the movements by the virtual object 1001 by interacting with the record start/stop option 1003.”).
29 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Liu) to incorporate the teachings of Assouline to provide a method that can utilize a type of user interface control for the predicted movement, which can utilize the predicted movement taught by Liu. Doing so would allow users to record a movement vector and to send them to another user as stated by Assouline ([0017] recited).
30 Regarding claim 16, Liu teaches the computer-readable storage device of claim 15 (see claim 15 rejection above), wherein the input from the user device for varying the degree of the at least one of the predicted movements of the user ([0042] reciting “These networks 508-514 are configured to process the image 504 and audio data 506 jointly to personalize the 3D avatar 502 and animate the 3D avatar 502 in synchronization with the audio data 506. Specifically, the 3D avatar 502 is animated with head, mouth, eye, and/or facial muscle movements. There movements are synchronized with and dynamically controlled based on the audio data 506, i.e., each movement vary dynamically and in real time with one or more of content, a volume and a pitch of a voice, a speech rate, and other characteristics of the audio data 506.” ; [0066] reciting “Lip movement prediction from the audio data 506 is easy to use and has natural results.”)
31 Liu does not explicitly teach … increases or decreases at least one of: a movement vector for one of the predicted movements or a frequency with which the one of the predicted movements occurs.
32 Assouline teaches … increases or decreases at least one of: a movement vector for one of the predicted movements or a frequency with which the one of the predicted movements occurs ([0042] reciting “Namely, the skeletal rig of the selected second 3D avatar is modified based on the joint movements, speed, direction, positions and acceleration specified in the movement vector. This way, the second user can apply movements or motion recorded by a first user to an avatar selected by the second user. The second user can also modify the motion or movement vector to increase or decrease the amount or duration of movements specified in the movement vector.”; [0109] reciting “In some embodiments, the augmentation system 208 is trained using a machine learning technique to predict or estimate a position on the real-world object that is associated with the least movement or noise.”).
33 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Liu) to incorporate the teachings of Assouline to provide a method that can determine an increase or decrease of the predicted movement vector, which can utilize the predicted movement taught by Liu. Doing so would allow users to record a movement vector and to send them to another user as stated by Assouline ([0017] recited).
34 Regarding claim 17, Liu in view of Assouline teaches the computer-readable storage device of claim 16 (see claims 15-16 rejections above),
35 Assouline from claim 16 can further teach the limitations, specifically wherein the input is received via a user interface control ([0160] reciting “In response to receiving input from the second user selecting the option 902 to start motion capture, the augmentation system 208 begins recording movement of the second user and generates a first movement vector, as illustrated in the example user interfaces of FIG. 10.”; [0161] reciting “As shown in FIG. 10, a graphical user interface 1000 is presented on a display of the client device 102 of the second user, in which a real-world object 1002 (e.g., a person or user) is shown together with a virtual object 1001 (e.g., a 3D avatar)… his way, as the user performs movements, the virtual object 1001 is moved in the same manner so the user can visualize what the virtual object 1001 looks like when mimicking the user's movements. The second user can control when the movements the user performs are stored in a movement vector to subsequently loop animation of the movements by the virtual object 1001 by interacting with the record start/stop option 1003.”).
36 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Liu) to incorporate the teachings of Assouline to provide a type of user interface control for the predicted movement methods that are provided by the teachings of Liu. Doing so would allow users to record a movement vector and to send them to another user as stated by Assouline ([0017] recited).
37 Regarding claim 18, Liu in view of Assouline teaches the computer-readable storage device of claim 16, further comprising instructions executable by the processor for (see claims 15-16 rejections above):
38 Assouline from claim 16 can further teach the limitations, specifically communicating with a second user device of a second user, wherein the animation of the avatar is used during the communicating with the second user device ([0017] reciting “The second user can then use the movement vector recorded by the first user to animate the 3D avatar selected by the second user. The second user can update or modify the movement vector based on recorded movement of the second user.”; [0041] reciting “The augmentation system 208 allows a first user to record and generate a movement or motion vector and share the movement vector with one or more other users. For example, after recording movement of the first user to generate a movement vector, the first user can select an option to send a communication (e.g., a chat message) to a second user.”).
39 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Liu) to incorporate the teachings of Assouline to provide access to a second user that can be communicated with the first users, the communications being animation data provided by Liu. Doing so would allow users to record a movement vector and to send them to another user as stated by Assouline ([0017] recited).
40 Regarding claim 19, Liu teaches the computer-readable storage device of claim 15, further comprising instructions executable by the processor for (see claim 15 rejection above): but does not explicitly teach communicating the animation of the avatar to a second user device of a second user; and receiving a second input from the second user device for varying a degree of the at least one of the predicted movements of the user to increase or decrease at least one of: a movement vector for one of the predicted movements or a frequency with which the one of the predicted movements occurs.
41 Assouline teaches communicating the animation of the avatar to a second user device of a second user ([0017] reciting “The second user can then use the movement vector recorded by the first user to animate the 3D avatar selected by the second user. The second user can update or modify the movement vector based on recorded movement of the second user.”; [0041] reciting “The augmentation system 208 allows a first user to record and generate a movement or motion vector and share the movement vector with one or more other users. For example, after recording movement of the first user to generate a movement vector, the first user can select an option to send a communication (e.g., a chat message) to a second user.”); and receiving a second input from the second user device for varying a degree of the at least one of the predicted movements of the user to increase or decrease at least one of: a movement vector for one of the predicted movements or a frequency with which the one of the predicted movements occurs ([0042] reciting “Namely, the skeletal rig of the selected second 3D avatar is modified based on the joint movements, speed, direction, positions and acceleration specified in the movement vector. This way, the second user can apply movements or motion recorded by a first user to an avatar selected by the second user. The second user can also modify the motion or movement vector to increase or decrease the amount or duration of movements specified in the movement vector.”; [0109] reciting “In some embodiments, the augmentation system 208 is trained using a machine learning technique to predict or estimate a position on the real-world object that is associated with the least movement or noise.”).
42 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Liu) to incorporate the teachings of Assouline to provide access to a second user that can be communicated with the first users, the communications being animation data provided by Liu, as well as to determine an increase or decrease of the predicted movement vector based on the second user, which can utilize the predicted movement taught by Liu. Doing so would allow users to record a movement vector and to send them to another user as stated by Assouline ([0017] recited).
43 Regarding claim 20, Liu in view of Assouline teaches the computer-readable storage device of claim 19, further comprising instructions executable by the processor for (see claim 15 and 19 rejections above):
44 Assouline from claim 19 can further teach the limitations, specifically in response to the second input, generating an updated animation of the avatar of the user ([0017] reciting “The second user can then use the movement vector recorded by the first user to animate the 3D avatar selected by the second user. The second user can update or modify the movement vector based on recorded movement of the second user.”); and
communicating the updated animation of the avatar to the second user device ([0125] reciting “The second user can capture an image or video of a selected avatar performing movements according to the modified or updated first movement vector. The second user can then send a communication back to the first user or to a third user that includes the updated first movement vector or the video that depicts the selected avatar (and optionally one or more persons) performing the movements of the modified first movement vector.”; [0157] reciting “Specifically, the first user can select a second virtual object to be displayed in a real-time feed and to mimic movement of a person depicted in the real-time feed. The second virtual object can represent motion of the person (e.g., is updated based on tracking of the skeletal joints of the person) while the record motion option is activated.”).
45 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Liu) to incorporate the teachings of Assouline to provide a second user that can communicate its update to and from the first user, utilizing the movement methods and/or changes that are provided by Liu. Doing so would allow users to record a movement vector and to send them to another user as stated by Assouline ([0017] recited).
46 Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al. (WO 2022103877 A1) in view of Wedig et al. (US 20190362529 A1).
47 Regarding claim 8, Liu teaches the method according to claim 1 (see claim 1 rejection above), but does not explicitly teach receiving information from the user device editing an appearance of the avatar; updating the avatar based on the received information from the user device editing the appearance of the avatar.
48 Wedig teaches receiving information from the user device editing an appearance of the avatar; updating the avatar based on the received information from the user device editing the appearance of the avatar ([0104] reciting “The 3D model processing system 680 can be configured to animate and cause the display 220 to render a virtual avatar 670. The 3D model processing system 680 can include a virtual character processing system 682 and a movement processing system 684. The virtual character processing system 682 can be configured to generate and update a 3D model of a user (for creating and animating the virtual avatar). The movement processing system 684 can be configured to animate the avatar, such as, e.g., by changing the avatar's pose, by moving the avatar around in a user's environment, or by animating the avatar's facial expressions, etc. As will further be described herein, the virtual avatar can be animated using rigging techniques. In some embodiments, an avatar is represented in two parts: a surface representation (e.g., a deformable mesh) that is used to render the outward appearance of the virtual avatar and a hierarchical set of interconnected joints (e.g., a core skeleton) for animating the mesh. In some implementations, the virtual character processing system 682 can be configured to edit or generate surface representations, while the movement processing system 684 can be used to animate the avatar by moving the avatar…”).
49 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Liu) to incorporate the teachings of Wedig to provide a method that can give the ability to edit or update a 3d avatar model, utilizing the models generated by Liu. Doing so would allow specific changes like changing the avatar's pose, by moving the avatar around in a user's environment, or by animating the avatar's facial expressions as stated by Wedig ([0104] recited).
50 Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al. (WO 2022103877 A1) in view of Cao et al. (US 20220358719 A1).
51 Regarding claim 13, Liu teaches the method according to claim 1 (see claim 1 rejection above), but does not teach wherein training the first machine learning model based on the video data comprises using at least one few-shot learning technique.
52 Cao teaches wherein training the first machine learning model based on the video data comprises using at least one few-shot learning technique ([0046] reciting “A head pose 644 may include position, r, and tilt, t, of the rigid head of a subject (no expression). In a real-time facial animation stage, encoder 600 takes test image 611 from a user's new input video and combines a coarse mesh tracking algorithm with the encoder obtained from the offline training step (e.g., encoder 500) to achieve pixel-precise facial animation in real-time. The environments and lighting of the testing scenario (e.g., raw images 511) may be different from those in the training data (e.g., test image 611). In some embodiments, encoder 600 applies a few-shot learning strategy to include face parameters 642 including rigid (e.g., rotation, r and translation, t) and non-rigid parameters (e.g., facial expression Z) to accurately recover the facial motions of test image 611 with new environments and lighting.”).
53 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Liu) to incorporate the teachings of Cao to provide a handful of few-shot techniques for the facial animations from specific videos that are taught by Liu. Doing so would include various face parameters including rigid (e.g., rotation) and non-rigid parameters (e.g., facial expression) as stated by Cao ([0046] recited).
Conclusion
54 Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
55 Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHNNY TRAN LE whose telephone number is (571)272-5680. The examiner can normally be reached Mon-Thu: 7:30am-5pm; First Fridays Off; Second Fridays: 7:30am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHNNY T LE/Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights