DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
1. A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on March 4th, 2026 has been entered.
Response to Amendment
2. This action is in response to the amendment filed on March 4th, 2026. Claims 1, 4-6, 8, 11-13, 15, and 18-20 have been amended. Claims 2-3, 9-10, and 16-17 have been cancelled. Claims 1, 4-8, 11-15, and 18-20 remain rejected in the application. Applicant’s amendments to the claims have overcome each and every objection previously set forth in the final office action mailed December 31st, 2025.
Response to Arguments
3. Applicant’s arguments with respect to claim 1, and similarly claims 8 and 15, with respect to the rejection under 35 U.S.C. 103 regarding that the prior art Richard, Marks, and Gu does not teach the limitation(s): “randomly pairing a first training silhouette image with a second training silhouette image from a plurality of different training silhouette images of training data, wherein the first training silhouette image is filtered to remove an upper half of the human face and the second training silhouette image is filtered to remove a lower half of the human face” and “an audio encoder configured to generate encoded data by encoding visemes that correspond to keypoints of lower portions of the merged silhouette images” have been considered but are moot because of the new ground of rejection. In addition, regarding arguments that "Richard does not disclose or suggest training such a unimodal model," that "Gu ... does not appear to be a multimodal model," and that "the combination of the various models of Richard, Marks, and Gu ... would change the principle of operation of the cited references" have also been considered but are moot because of the new ground of rejection. The claims are now disclosed by Jeong, Malafeev, Xiong, and Zhang.
4. Regarding arguments to claims 4-7, 11-14, and 18-20, they are dependent on independent claims 1, 8 and 15 respectively. Applicant does not argue anything other than independent claim 1, and similarly claims 8 and 15. The limitations in those claims, in conjunction with their combination, has previously been established and explained.
Claim Rejections - 35 USC § 103
5. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
6. Claims 1, 8, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Jeong et al. (US-2022/0207262-A1, hereinafter "Jeong") in view of Malafeev et al. (US-2023/0144458-A1, hereinafter "Malafeev"), further in view of Xiong et al. (CN-113450436-A, hereinafter "Xiong"), and further in view of Zhang et al. (CN-112785671-A, hereinafter "Zhang").
7. As per claim 1, Jeong discloses: A computer-implemented method, comprising: (Jeong, [0007], “Therefore, an object of the present invention is to provide a mouth shape synthesis device and a method using an artificial neural network in which the precision of the synthesis of a mouth shape and a reality are improved through an audio input.”)
generating a set of merged [[silhouette]] images corresponding to a human face by, for each merged [[silhouette]] image: (Jeong, [0132], “According to the upper and lower alignment discriminator 90 according to the modified example of the present invention, the effect of harmoniously synthesizing visual features such as facial expression, color, and influence of lighting of the upper and lower halves of the synthesized face is generated.”)
randomly pairing a first training [[silhouette]] image with a second training silhouette image (Jeong, Fig. 19-20; [0063], “The landmark detection module 111 is a module for outputting landmark position information and confidence information for defining the lower half of the target face, such as the nose or the mouth. ... The target face lower half mask module 113 is a module that defines the lower half of the target face in the target face bounding box based on the landmark position information output from the landmark detection module 111 and corrected by the landmark position correction module 112.”; Examiner’s note: The generated facial landmarks are a silhouette of the lower half of the face.) from a plurality of different training [[silhouette]] images of training data, wherein the first training [[silhouette]] image is filtered to remove an upper half of the human face and the second training silhouette image is filtered to remove a lower half of the human face (See Jeong, Fig. 19-20; [0063] above regarding “second training silhouette image.”); and (Jeong, Fig. 18-20; [0124], “The angle discriminator 80 may be configured to include a face upper half angle detection module 81 and a face lower half angle detection module 82. … Generation of the upper half video and the lower half video of the target face from the synthesized video data may be performed by the face detection module 11.” and [0129], “The upper and lower alignment discriminator 90 is a pre-learned artificial neural network module including a plurality of Convolution Layers that embed the upper half face video and the lower half face video of the synthesized video data to be received as input data, and output upper and lower alignment vectors (probability values between 0 and 1) as output data, which are indicators of how well the visual features (facial expression, color, influence of lighting, and the like) of the upper and lower halves of the face in the Output Layer configured of the activation functions such as Sigmoid and ReLU are aligned. The generation of the upper half video and the generation of the lower half video of the target face from the synthesized video data may be performed by the face detection module 11.” and [0130], “As illustrated in FIG. 18, in the learning session of the upper and lower alignment discriminator 90, the learning session may be configured such that [face upper half video and face lower half video] labeled with the upper and lower alignment and [face upper half video and face lower half video] labeled with the up and down non-alignment are input to the learning data, and the parameters of the upper and lower alignment discriminator 90 are updated in a direction in which the upper and lower alignment loss configured of a difference between the upper and lower alignment vector output from the Output Layer of the upper and lower alignment discriminator 90 and the actual label is reduced. The upper and lower alignment loss of the upper and lower alignment discriminator 90 may be configured of a Mean square loss, a Cross entropy loss, and the like.”; Examiner’s note: The upper and lower face portions of the face are separated and then used as learning data. The upper and lower face portions are paired into an alignment discriminator. A upper and lower face input can be “non-aligned” and thus are not correlated (meaning that the pairing can be random).)
generating a merged [[silhouette]] image of the set of [[silhouette]] images by merging the first training [[silhouette]] image as filtered and the second training silhouette image as filtered (See Jeong, Fig. 19-20; [0063] above regarding “second training silhouette image.”) such that the merged [[silhouette]] image generated for the set of merged [[silhouette]] images has an upper portion of a human face uncorrelated with a lower portion of the human face; (Jeong, [0132], “According to the upper and lower alignment discriminator 90 according to the modified example of the present invention, the effect of harmoniously synthesizing visual features such as facial expression, color, and influence of lighting of the upper and lower halves of the synthesized face is generated.” and [0131], “With respect to the operational relationship of the upper and lower registration discriminator 90, the face upper half video and the face lower half video of the synthesized video data are embedded in the upper and lower alignment discriminator 90 to be received as input data, and the upper and lower registration vectors are output as output data. The output upper and lower alignment vector may be applied in the learning session so that the parameters of the original video encoder 10, the audio encoder 20, and the synthesized video decoder 30 are updated. That is, in the original video encoder 10, the audio encoder 20, and the synthesized video decoder 30 for each epoch, learning session may be configured such that the parameters of the original video encoder 10, the audio encoder 20, and the synthesized video decoder 30 are updated in a direction in which a sum of the reconstruction loss calculated based on the synthesized video data and the original video data, the synthesized video loss in the pre-learned synthesized video discriminator 31, the sync loss calculated based on the synthesized video data and the audio data in the pre-learned sync discriminator 40, and the top and bottom alignment vectors output from the pre-learned top and bottom alignment discriminator 90 is reduced.” and [0130], “As illustrated in FIG. 18, in the learning session of the upper and lower alignment discriminator 90, the learning session may be configured such that [face upper half video and face lower half video] labeled with the upper and lower alignment and [face upper half video and face lower half video] labeled with the up and down non-alignment are input to the learning data, and the parameters of the upper and lower alignment discriminator 90 are updated in a direction in which the upper and lower alignment loss configured of a difference between the upper and lower alignment vector output from the Output Layer of the upper and lower alignment discriminator 90 and the actual label is reduced.”; Examiner’s note: Since Jeong discloses generates a synthesized video, this produces a plurality of generated faces. Also, the upper and lower face portions are paired into an alignment discriminator. A upper and lower face input can be “non-aligned” and thus are not correlated.)
training a unimodal machine learning model with the set of merged [[silhouette]] images to generate synthetic images of the human face; and (Jeong, [0132], “According to the upper and lower alignment discriminator 90 according to the modified example of the present invention, the effect of harmoniously synthesizing visual features such as facial expression, color, and influence of lighting of the upper and lower halves of the synthesized face is generated.” and [0130], “As illustrated in FIG. 18, in the learning session of the upper and lower alignment discriminator 90, the learning session may be configured such that [face upper half video and face lower half video] labeled with the upper and lower alignment and [face upper half video and face lower half video] labeled with the up and down non-alignment are input to the learning data, and the parameters of the upper and lower alignment discriminator 90 are updated in a direction in which the upper and lower alignment loss configured of a difference between the upper and lower alignment vector output from the Output Layer of the upper and lower alignment discriminator 90 and the actual label is reduced. The upper and lower alignment loss of the upper and lower alignment discriminator 90 may be configured of a Mean square loss, a Cross entropy loss, and the like.”)
[[training a multimodal rendering network based on minimizing differences between the synthetic images output from the unimodal machine learning model and images generated by the multimodal rendering network;]]
[[wherein the multimodal rendering network includes a pre-processor configured to generate partial silhouettes from image modality data of the training data, wherein the partial silhouettes omit a mouth portion, an audio encoder configured to generate encoded data by encoding visemes that correspond to keypoints of lower portions of the merged silhouette images of the set of merged silhouette images used to generate the synthetic images into a same latent space as the image modality data, a silhouette encoder configured to generate encoded data by compressing the partial silhouette, and a decoder configured to generate a voice-animated digital human from the encoded data.]]
8. Jeong doesn't explicitly disclose but Malafeev discloses: [[generating a set of merged]] silhouette [[images corresponding to a human face by, for each merged]] silhouette [[image:]] (Malafeev, See Fig. 2, [0042], [0099], [0073]-[0074], [0083], [0078], and [0044] below.)
[[randomly pairing a first training]] silhouette [[image with a second training silhouette image from a plurality of different training]] silhouette [[images of training data, wherein the first training]] silhouette [[image is filtered to remove an upper half of the human face and the second training silhouette image is filtered to remove a lower half of the human face; and]] (Malafeev, Fig. 2; [0042], “In one or more embodiments, the one or more landmarks include fiducial facial landmarks pertaining to the eyes, mouth, nose, face boundary and/or other facial features for each detected face.” and [0099], “For example, the landmark detector(s) 102 may analyze video data representative of one or more sequences of images, including the image(s) 122 depicting the face(s) 132 to determine one or more locations of one or more facial landmarks corresponding to the face(s) 132.” and [0073]- [0074], “Thus, the set may include a portion of the image data corresponding to the region(s) or location(s). By way of example, and not limitation, where the input data represents or indicates locations of landmarks for the face 132, one data set may include location data corresponding to one or more locations on the left upper half of the face 132 (e.g., including left eye and eyebrow and a center of the nose), another data set may include location data corresponding to one or more locations on right upper half of the face 132 (e.g., including right eye and eyebrow and the center of the nose), and another data set may include location data corresponding to one or more locations on a bottom half of the face 132 (e.g., including the nose below the bridge, the mouth, and the jawline). Additionally, or alternatively, where the input data sets 320 correspond to image data, one data set may include image data capturing an image region corresponding to the left upper half of the face 132, another data set may include image data corresponding to an image region including the right upper half of the face 132, and another data set may include image data capturing an image region corresponding to a bottom half of the face 132.” and [0083], “By way of example, and not limitation, where the subset analyzer 304A analyzes a set of the input data sets 320 representing the locations of the landmarks pertaining to the left half of the face 132 and the subset analyzer 304B analyzes a set of the input data sets 320 representing the locations of the landmarks pertaining to the right half of the face 132, the sets may share in common at least some of the landmarks that run along the nose bridge (e.g., along a border of the regions). Providing overlap in the input data sets 320 may help regularize the data by making the data that is analyzed by a particular subset analyzer 304 less localized.” and [0078], “The object information 344 may correspond to a plurality of the sets included in the input data sets 320, such as each set analyzed by the subset analyzers 304. For example, the object information 344 may include information corresponding to the entire face 132 when the subset analyzers 304 analyze input data sets for respective regions of the face 132.” and [0044], “Next, fiducial facial landmarks may be detected for the detected face using the object location(s).”; Examiner’s note: Malafeev discloses in [0073]-[0074] generating landmarks from image data which would correspond to silhouettes of aspects of the face (eyes, nose, mouth, etc.). In addition, although Malafeev discloses left upper half and right upper half sections of the face, Malafeev also discloses in [0083] these sets can be treated together with shared data (and thus can be treated as the entire upper face.)
[[generating a merged]] silhouette [[image of the set of]] silhouette [[images by merging the first training]] silhouette [[image as filtered and the second training silhouette image as filtered such that the merged]] silhouette [[image generated for the set of merged]] silhouette [[images has an upper portion of a human face uncorrelated with a lower portion of the human face;]] (Malafeev, See Fig. 2, [0042], [0099], [0073]-[0074], [0083], [0078], and [0044] above.)
[[training a unimodal machine learning model with the set of merged]] silhouette [[images to generate synthetic images of the human face; and]] (Malafeev, See Fig. 2, [0042], [0099], [0073]-[0074], [0083], [0078], and [0044] above.)
9. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of Jeong to include the disclosure of generating silhouettes of a human face, of Malafeev. The motivation for this modification could have been to provide a method to quickly identify landmark areas that make up the profile (“silhouette”) of a human’s face. This can be used to train and encode specific parts of the face to generate, when combined, realistic synthetic 2D facial images that could be used instead of 3D meshes to ultimately create a human face animated by speech. By identifying landmark regions of the face, like the mouth, the machine learning model can generate more accurate results as other facial regions are not influenced by each other.
10. Jeong in view of Malafeev doesn't explicitly disclose but Xiong discloses: training a multimodal rendering network based on minimizing differences between the synthetic images output from the unimodal machine learning model and images generated by the multimodal rendering network; (Xiong, page 6, [0020]-[0021], “Module 1 is used to build a speech-to-expression mapping network based on multimodal correlation. After preprocessing the 2D speaker video dataset, a self-supervised method is used to train the speech-to-expression mapping network. The loss function is used to guide the training of the speech-to-expression mapping network based on multimodal correlation, and a trained speech-to-expression mapping network is obtained. The multimodal correlation speech-to-expression mapping network includes a speech Transformer network Taudio, an expression Transformer network Texp, and a cross-modal Transformer network Tcross.” and page 9, [0046], “In this embodiment, the loss function is calculated by training a speech-to-expression mapping network based on multimodal correlation using the backpropagation method; the mean squared error (MSE) loss is calculated between the predicted expression sequence and βpred, that is, the speech Transformer, expression Transformer and cross-modal Transformer are trained by the backpropagation algorithm, and the loss function of the training results is observed. Training is stopped when the loss value does not decrease for 5 consecutive rounds.”)
wherein the multimodal rendering network includes a pre-processor configured to generate partial silhouettes from image modality data of the training data, (Xiong, page 6, [0020], “Module 1 is used to build a speech-to-expression mapping network based on multimodal correlation. After preprocessing the 2D speaker video dataset, a self-supervised method is used to train the speech-to-expression mapping network.” and page 6, [0011], “Step 3: Preprocess the speaking video and the driving speech to obtain the facial expression feature sequence and the source speech feature sequence from the speaking video, and obtain the driving speech feature sequence from the driving speech.” and page 7, [0031], “... based on the facial key point information obtained from the preprocessing in Module 1 ...”; Examiner’s note: The partial silhouettes are from the facial key point data.) [[wherein the partial silhouettes omit a mouth portion, an audio encoder configured to generate encoded data by encoding visemes that correspond to keypoints of lower portions of the merged silhouette images of the set of merged silhouette images used to generate the synthetic images into a same latent space as the image modality data, a silhouette encoder configured to generate encoded data by compressing the partial silhouette, and a decoder configured to]] generate a voice-animated digital human from the [[encoded]] data. (Xiong, page 7, [0033], “This invention proposes a novel Transformer-based architecture for voice-driven face animation generation. This architecture can provide temporal stability during the feature mapping stage, ensuring the diversity and smoothness of the generated results. In addition, thanks to the parallel computing capabilities of the Transformer architecture, the speech-to-expression mapping of this invention can achieve real-time performance compared to previous methods.”)
11. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of Jeong in view of Malafeev to include the disclosure of training a multimodal rendering network based on minimizing differences between the synthetic images output from the unimodal machine learning model and images generated by the multimodal rendering network, pre-processing to generate partial silhouettes, and finally generating a voice-animated digital human, of Xiong. The motivation for this modification could have been to utilize a multimodal machine learning model to simultaneously process different types of data, such as images/video and audio. In this instance, the model can be trained to correlate different types of data, such as video and voice. This can generate more accurate output for voice-animation as the multimodal model preserves relationships between audio and video.
12. Jeong in view of Malafeev, and further in view of Xiong doesn't explicitly disclose but Zhang discloses: [[wherein the multimodal rendering network includes a pre-processor configured to generate partial silhouettes from image modality data of the training data,]] wherein the partial silhouettes omit a mouth portion, (Zhang, Fig. 2; page 4, [0001], “This disclosure relates to the field of artificial intelligence/multimodal technology, and more specifically, to a method for synthesizing fake human face animations.” and page 9, [0067], “Meanwhile, to improve the synthesis of mouth region textures, a mouth synthesis branch was added to the video generation network structure, and a mouth mask was added as a parameter of the model to achieve intra-frame consistency.” and page 9, [0068]-[0070], “According to another embodiment of this disclosure, the video generation network is trained using a mouth-mask loss function, the calculation formula of which is as follows: ... In the formula, T represents the number of frames of the sample image; xt represents the sample image of frame t; represents the output sample image generated by inputting the synthesized face sketch of frame t into the initial network model; and mt represents the mouth mask corresponding to the sample image of frame t.” and page 9, [0073], “In the formula, represents the synthetic face sketch sequence and the mouth mask, st represents the synthetic face sketch of frame t, mt represents the mouth mask corresponding to the synthetic face sketch of frame t; represents the generated L-frame image; represents the occlusion mask corresponding to the synthetic face sketch of frame t; represents the generated hallucinated image of frame t, used to synthesize the occluded background region; represents the generated image of the mouth region of frame t; represents the estimated optical flow between and, which can be estimated from.”) an audio encoder configured to generate encoded data by encoding visemes that correspond to keypoints (Zhang, Fig. 2; page 8, [0056], “According to another embodiment of this disclosure, based on the angle and shape information of the face in the multi-frame face sketch 209, the mouth inverse normalization method 210 is used to update the position of each key point in the multiple sets of key point sequences 207 one by one, so as to obtain multiple sets of predicted key point sequences that are consistent with the face angle and shape information in the multi-frame face sketch. The angle and shape information includes the rotation angle of the face image in the face sketch 209, the maximum width of the chin in the multiple key point sequences 207 and the face sketch 209, and the position of the center point of the mouth in the multiple key point sequences 207 and the face sketch 209.” and p. 4-5, [0009], “According to embodiments of this disclosure, generating a multi-frame synthetic face sketch based on the second target video and the multiple sets of keypoint sequences includes: decoding the second target video frame by frame and extracting multiple frames of face sketches; updating the position of each keypoint in the multiple sets of keypoint sequences one by one using a mouth inverse normalization method based on the face angle and shape information ...” and page 7, [0036]-[0037], “In operation S120, multiple frames of audio features and multiple frames of prosodic features are input into the trained encoder-decoder network, which outputs multiple keypoint sequences composed of multiple keypoints corresponding to each frame of audio features and prosodic features. According to embodiments of this disclosure, the keypoint sequence includes multiple keypoints in the lip region and chin region. For each keypoint, its position changes according to a certain pattern in different image frames of the first target video. These patterns of change are highly correlated with audio features and prosodic features.” and page 9, [0067]-[0068], “Meanwhile, to improve the synthesis of mouth region textures, a mouth synthesis branch was added to the video generation network structure, and a mouth mask was added as a parameter of the model to achieve intra-frame consistency. According to another embodiment of this disclosure, the video generation network is trained using a mouth-mask loss function ...”; Examiner’s note: Visemes are visible mouth shapes. Figure 2 also shows the process of generating mouth shape keypoints at 207.) of lower portions of the merged silhouette images of the set of merged silhouette images used to generate the synthetic images into a same latent space as the image modality data, (Zhang, Fig. 2; page 4-5, [0009], “According to embodiments of this disclosure, generating a multi-frame synthetic face sketch based on the second target video and the multiple sets of keypoint sequences includes: decoding the second target video frame by frame and extracting multiple frames of face sketches; updating the position of each keypoint in the multiple sets of keypoint sequences one by one using a mouth inverse normalization method based on the face angle and shape information in the multiple frames of face sketches to obtain multiple sets of predicted keypoint sequences consistent with the face angle and shape information in the multiple frames of face sketches; and sequentially synthesizing the multiple sets of predicted keypoint sequences and the multiple frames of face sketches to obtain the multi-frame synthetic face sketch.” and page 9, [0067], “Meanwhile, to improve the synthesis of mouth region textures, a mouth synthesis branch was added to the video generation network structure, and a mouth mask was added as a parameter of the model to achieve intra-frame consistency.” and page 8, [0059], “In the formula, represents a keypoint in a set of keypoint sequences 207; represents a keypoint in a set of predicted keypoint sequences corresponding to and ; θ represents the rotation angle of the face image in the face sketch; d1 represents the maximum width between keypoints representing the chin in the set of keypoint sequences; d2 represents the maximum width of the chin in the face sketch; c′(x, y) represents the center point of keypoints representing the mouth in the set of keypoint sequences; c″(x, y) represents the center point of the mouth in the face sketch.” and page 9, [0068]-[0070], “According to another embodiment of this disclosure, the video generation network is trained using a mouth-mask loss function, the calculation formula of which is as follows: ... In the formula, T represents the number of frames of the sample image; xt represents the sample image of frame t; represents the output sample image generated by inputting the synthesized face sketch of frame t into the initial network model; and mt represents the mouth mask corresponding to the sample image of frame t.” and page 9, [0073], “In the formula, represents the synthetic face sketch sequence and the mouth mask, st represents the synthetic face sketch of frame t, mt represents the mouth mask corresponding to the synthetic face sketch of frame t; represents the generated L-frame image; represents the occlusion mask corresponding to the synthetic face sketch of frame t; represents the generated hallucinated image of frame t, used to synthesize the occluded background region; represents the generated image of the mouth region of frame t; represents the estimated optical flow between and, which can be estimated from.” and page 4, [0008], “According to embodiments of this disclosure, the initial encoder-decoder network structure includes: a speech encoder for acquiring and encoding the multi-frame audio features or the multi-frame sample audio features; a text encoder for acquiring and encoding the multi-frame prosodic features or the multi-frame sample prosodic features; and a decoder for decoding the features acquired by the speech encoder and the text encoder, and outputting the multi-set keypoint sequences or the multi-set sample keypoint sequences.”) a silhouette encoder configured to generate encoded data by compressing the partial silhouette, and a decoder configured to [[generate a voice-animated digital human from the]] encoded [[data.]] (Zhang, Fig. 2; page 4, [0008], “According to embodiments of this disclosure, the initial encoder-decoder network structure includes: a speech encoder for acquiring and encoding the multi-frame audio features or the multi-frame sample audio features; a text encoder for acquiring and encoding the multi-frame prosodic features or the multi-frame sample prosodic features; and a decoder for decoding the features acquired by the speech encoder and the text encoder, and outputting the multi-set keypoint sequences or the multi-set sample keypoint sequences.” and page 4, [0006], “The method for synthesizing fake face animation according to embodiments of this disclosure includes: extracting speech information and text information from a first target video, and extracting multi-frame audio features from the speech information and multi-frame prosodic features from the text information, respectively; inputting the multi-frame audio features and multi-frame prosodic features into a trained encoder-decoder network, and outputting multiple sets of keypoint sequences composed of multiple keypoints corresponding to each frame of the audio features and prosodic features; generating multi-frame synthetic face sketches based on a second target video and the multi-set keypoint sequences; and synthesizing fake face animations using a video generation network based on the multi-frame synthetic face sketches.” and page 7-8, [0046], “According to another embodiment of this disclosure, the encoder-decoder network 206 includes a speech encoder 2061, a text encoder 2062, and a decoder 2063. The speech encoder 2061 and text encoder 2062 are the input parts of the encoder-decoder network structure 206. Both use the TCN network as their model architecture and are used to encode audio features 204 and prosodic features 205, respectively, and input them into the encoder-decoder network 206.” and page 4, [0001], “This disclosure relates to the field of artificial intelligence/multimodal technology, and more specifically, to a method for synthesizing fake human face animations.” and page 8, [0055], “According to another embodiment of this disclosure, the target video is decoded frame by frame, and multiple frames of face sketches 209 are extracted from the decoded multi-frame target images 208. The face sketch 209 contains complete spatial and temporal information about the lip and chin regions.” and page 9-10, [0074], “The proposed method for synthesizing fake face animations in this embodiment integrates the complementarity of multimodal information. The method ensures the synchronization of lip movements with speech/text information and the synchronization of chin movements with speech/text, thereby ensuring the consistency of lip region and chin movement and improving the prediction accuracy of key points. Meanwhile, optical flow and mouth synthesis branches were used to model the correlation between video frames and within frames, respectively, and the synthesis of fake face animation with lip-sync and temporal continuity was achieved. The synthesized animation has a high degree of realism.”)
13. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of Jeong in view of Malafeev, and further in view of Xiong to include the disclosure of partial silhouettes omitting a mouth portion, an audio encoder configured to generate encoded data by encoding visemes that correspond to keypoints of lower portions of the merged silhouette images of the set of merged silhouette images used to generate the synthetic images into a same latent space as the image modality data, a silhouette encoder configured to generate encoded data by compressing the partial silhouette, and a decoder to generate output, of Zhang. The motivation for this modification could have been separate the mouth from the machine learning model so it is not influenced by other data in order to improve its accuracy. In addition, an audio encoder is used as a means to accurately map keypoints of visemes (mouth shapes) to voice audio. By performing this mapping, the synthesized representation of the mouth with more accurately represent the voice audio being played. This adds more realism to the overall voice-animated face. Lastly, the images and audio are encoded and decoded into latent space for the machine learning model. This allows for a correlation between images/video and audio in latent space to preserve certain data relationships. This can then be used for quick generation of an audio/video output such as a voice-animation.
14. Claim 8 is similar in scope to claim 1 except for additional limitations that Jeong in view of Malafeev, further in view of Xiong, and further in view of Zhang discloses: A system, comprising: one or more processors configured to initiate operations including: (Malafeev, [0108], “The CPU(s) 706 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 700 to perform one or more of the methods and/or processes described herein.”)
15. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the system of Jeong in view of Xiong, and further in view of Zhang to include the disclosure of one or more processors configured to initiate operations, of Malafeev. The motivation for this modification could have been to use a processor to quickly and efficiently perform the process of facial generation rather than a manual, by-hand process. Using a computing system with a processor can generate accurate and timely results that might be difficult to produce manually. In addition, the motivation to combine is the same rationale as claim 1, described above.
16. Claim 15 is similar in scope to claim 1 except for additional limitations that Jeong in view of Malafeev, further in view of Xiong, and further in view of Zhang discloses: A computer program product, comprising: one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, wherein the program instructions are executable by one or more processors to initiate operations including: (Malafeev, [0106], “The computer-storage media may include both volatile and nonvolatile media and/or removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, and/or other data types. For example, the memory 704 may store computer-readable instructions (e.g., that represent a program(s) and/or a program element(s), such as an operating system.” and [0108], “The CPU(s) 706 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 700 to perform one or more of the methods and/or processes described herein.”)
The motivation to combine is the same rationale as claim 8, described above.
17. Claims 4-6, 11-13, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Jeong et al. (US-2022/0207262-A1, hereinafter "Jeong") in view of Malafeev et al. (US-2023/0144458-A1, hereinafter "Malafeev"), further in view of Xiong et al. (CN-113450436-A, hereinafter "Xiong"), further in view of Zhang et al. (CN-112785671-A, hereinafter "Zhang"), and further in view of Masi et al. (NPL: "Do We Really Need to Collect Millions of Faces for Effective Face Recognition?," Computer Vision – ECCV 2016, hereinafter "Masi").
18. As per claim 4, Jeong in view of Malafeev, further in view of Xiong, and further in view of Zhang discloses: The method of claim 1, wherein the merging the first training silhouette image and the second training silhouette image comprises: (See rejection of claim 1.)
extracting associated sets of keypoints from each of the first training silhouette image and the second training silhouette image; (Malafeev, [0042], “In one or more embodiments, the one or more landmarks include fiducial facial landmarks pertaining to the eyes, mouth, nose, face boundary and/or other facial features for each detected face.” and Malafeev, [0073]- [0074], “Thus, the set may include a portion of the image data corresponding to the region(s) or location(s). By way of example, and not limitation, where the input data represents or indicates locations of landmarks for the face 132, one data set may include location data corresponding to one or more locations on the left upper half of the face 132 (e.g., including left eye and eyebrow and a center of the nose), another data set may include location data corresponding to one or more locations on right upper half of the face 132 (e.g., including right eye and eyebrow and the center of the nose), and another data set may include location data corresponding to one or more locations on a bottom half of the face 132 (e.g., including the nose below the bridge, the mouth, and the jawline). Additionally, or alternatively, where the input data sets 320 correspond to image data, one data set may include image data capturing an image region corresponding to the left upper half of the face 132, another data set may include image data corresponding to an image region including the right upper half of the face 132, and another data set may include image data capturing an image region corresponding to a bottom half of the face 132.” and Malafeev, [0083], “By way of example, and not limitation, where the subset analyzer 304A analyzes a set of the input data sets 320 representing the locations of the landmarks pertaining to the left half of the face 132 and the subset analyzer 304B analyzes a set of the input data sets 320 representing the locations of the landmarks pertaining to the right half of the face 132, the sets may share in common at least some of the landmarks that run along the nose bridge (e.g., along a border of the regions). Providing overlap in the input data sets 320 may help regularize the data by making the data that is analyzed by a particular subset analyzer 304 less localized.” and Jeong, Fig. 19-20; [0063], “The landmark detection module 111 is a module for outputting landmark position information and confidence information for defining the lower half of the target face, such as the nose or the mouth. ... The target face lower half mask module 113 is a module that defines the lower half of the target face in the target face bounding box based on the landmark position information output from the landmark detection module 111 and corrected by the landmark position correction module 112.”)
frontalizing the sets of keypoints of each of the first training silhouette image and the second training silhouette image; (Malafeev, [0047]-[0048], “The normalizer(s) 104 may be configured to analyze the one or more locations of the one or more landmarks to determine normalized version of the one or more locations of the one or more landmarks, which may be represented and/or indicated by the normalized location data 126. In at least one embodiment, the locations of all (e.g., 126) landmarks are normalized. Normalization may include, in accordance with one or more embodiments, one or more of rotating, translating, or scaling of one or more locations of one or more landmarks, such that values of the landmarks all lie in a predetermined range (e.g., −1 to 1). In at least one embodiment, the rotating may include rotating one or more of the locations of the one or more landmarks in-plane. The normalizer 104 may be configured to perform the rotation to normalize the orientation of the location(s) with respect to a coordinate system. For example, the rotation may rectify the face 132 so that it is straight or otherwise facing a consistent direction across inputs. This may be accomplished, for example, by performing the rotation such that the line joining the two eyes are along the horizontal.”)
19. Richard in view of Jeong in view of Malafeev, further in view of Xiong, and further in view of Zhang doesn't explicitly disclose but Masi discloses: generating a frontalized silhouette by adding the sets of keypoints of the first training silhouette image and the second training silhouette image as frontalized; and (Masi, Figure 2; page 5, ¶ 2-¶ 4, “In order to generate unseen viewpoints given a face image I, we use a technique similar to the frontalization proposed by [11]. ... Given the corresponding landmarks pi ↔ Pi we use PnP [9] to estimate extrinsic camera parameters, assuming the principal point is in the image center and then refining the focal length by minimizing landmark re-projection errors. This process gives us a perspective camera model mapping the generic 3D shape S on the image such as pi ∼ M Pi where M=K [R t] is the camera matrix.”)
de-frontalizing the frontalized silhouette, thereby generating the merged silhouette image. (Masi, Figure 2; page 5, ¶ 5, “Given the estimated pose M, we decompose it to obtain a rotation matrix R ∈ R3x3 containing rotation angles for the 3D head shape with respect to the image. We then create new rotation matrices R’Θ ∈ R3x3 for unseen (novel) viewpoints by sampling different yaw angles θ.”)
20. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 1 of Jeong in view of Malafeev, further in view of Xiong, and further in view of Zhang to include the disclosure of generating a frontalized silhouette and de-frontalizing the frontalized silhouette to generate a silhouette image, of Masi. The motivation for this modification could have been to allow for facial images of any pose to be normalized and then re-posed in other orientations, including a direct frontal view or even the original pose. The generated facial images can then be used to match the speech audio for voice animation.
21. As per claim 5, Jeong in view of Malafeev, further in view of Xiong, further in view of Zhang, and further in view of Masi discloses: The method of claim 4, wherein the frontalizing the sets of keypoints comprises:
multiplying the sets of keypoints of the first training silhouette image and the second training silhouette image by an inverse of a head pose matrix of the first training silhouette image and an inverse of a head pose matrix of the second training silhouette image, respectively. (Masi, Figure 2; page 5, ¶ 2, “We begin by applying the facial landmark detector from [2]. Given these detected landmarks we estimate the six degrees of freedom pose for the face in I using correspondences between the detected landmarks pi ∈ R2 and points Pi ≐ S(i) ∈ R3, labeled on a 3D generic face model S.” and page 5, ¶ 2-¶ 4, “In order to generate unseen viewpoints given a face image I, we use a technique similar to the frontalization proposed by [11]. ... Given the corresponding landmarks pi ↔ Pi we use PnP [9] to estimate extrinsic camera parameters, assuming the principal point is in the image center and then refining the focal length by minimizing landmark re-projection errors. This process gives us a perspective camera model mapping the generic 3D shape S on the image such as pi ∼ M Pi where M=K [R t] is the camera matrix.”)
See claim 4 rejection for reason to combine.
22. As per claim 6, Jeong in view of Malafeev, further in view of Xiong, further in view of Zhang, and further in view of Masi discloses: The method of claim 4, wherein the de-frontalizing the frontalized silhouette comprises:
multiplying the frontalized silhouette by either a head pose matrix of the first training silhouette image, or a head pose matrix of the second training silhouette image. (Masi, Figure 2; page 5, ¶ 5, “Given the estimated pose M, we decompose it to obtain a rotation matrix R ∈ R3x3 containing rotation angles for the 3D head shape with respect to the image. We then create new rotation matrices R’Θ ∈ R3x3 for unseen (novel) viewpoints by sampling different yaw angles θ. … Figure 2 shows viewpoint (pose) synthesis results for a training subject in CASIA, illustrating the 3D pose estimation process.”)
See claim 4 rejection for reason to combine.
23. Claim 11, which is similar in scope to dependent claim 4 and independent claim 8, is thus rejected under the same rationale as described above. The motivation for this modification is the same as claim 4.
24. Claim 12, which is similar in scope to dependent claim 5 and independent claim 8, is thus rejected under the same rationale as described above. The motivation for this modification is the same as claim 5.
25. Claim 13, which is similar in scope to dependent claim 6 and independent claim 8, is thus rejected under the same rationale as described above. The motivation for this modification is the same as claim 6.
26. Claim 18, which is similar in scope to dependent claim 4 and independent claim 15, is thus rejected under the same rationale as described above. The motivation for this modification is the same as claim 4.
27. Claim 19, which is similar in scope to dependent claim 5 and independent claim 15, is thus rejected under the same rationale as described above. The motivation for this modification is the same as claim 5.
28. Claim 20, which is similar in scope to dependent claim 6 and independent claim 15, is thus rejected under the same rationale as described above. The motivation for this modification is the same as claim 6.
29. Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Jeong et al. (US-2022/0207262-A1, hereinafter "Jeong") in view of Malafeev et al. (US-2023/0144458-A1, hereinafter "Malafeev"), further in view of Xiong et al. (CN-113450436-A, hereinafter "Xiong"), further in view of Zhang et al. (CN-112785671-A, hereinafter "Zhang"), and further in view of Yang et al. (CN-113610058-A, hereinafter "Yang").
30. As per claim 7, Jeong in view of Malafeev, further in view of Xiong, and further in view of Zhang discloses: The method of claim 1, wherein the generating each synthetic image comprises: (See rejection of claim 1.)
31. Jeong in view of Malafeev, further in view of Xiong, and further in view of Zhang doesn't explicitly disclose but Yang discloses: performing an image-to-image transformation of a corresponding merged silhouette image. (Yang, Claim 4, “The facial pose enhanced interaction method of human faces migration according to claim 3, wherein training enhanced model by Pix2Pix method; Pix2Pix method is as follows: i) the x is the edge image of the input image, y is the source image of the input image, Pix2Pix method training is supervised, so the pair of images (x, y) for training; generator G converts the input x into the generated image G (x); the x and G (x) are merged together as the input of the discriminator D through the channel dimension ...”)
32. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 1 of Jeong in view of Malafeev, further in view of Xiong, and further in view of Zhang to include the disclosure of performing an image-to-image transformation of a corresponding merged silhouette image, of Yang. The motivation for this modification could have been to allow for an image-to-image training to occur with generated silhouette images and provide for an automatic, machine learned method to join silhouette images together.
33. Claim 14, which is similar in scope to dependent claim 7 and independent claim 8, is thus rejected under the same rationale as described above. The motivation for this modification is the same as claim 7.
Conclusion
34. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. These are as follows: Aoyama et al. (US-2010/0332229-A1), Matthews et al. (US-2014/0035929-A1), and Cohen-or et al. (US-2025/0140257-A1).
35. Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW CLOTHIER whose telephone number is (571)272-4667. The examiner can normally be reached Mon-Fri 8:00am-4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW CLOTHIER/Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights