DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/24/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 6, 15, and 19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claims 6, 15, and 19 cite “wherein each of the multiple convolution blocks comprises one or more residual units, containing one or more dilated convolutions with one or more dilation rates, and at least one down-sampling layer comprising a strided convolution.” It’s not clear if it indicates each of the multiple convolution blocks comprises
a) one or more residual units
b) containing one or more dilated convolutions with one or more dilation rates
c) at least one down-sampling layer comprising a strided convolution
or
each of the multiple convolution blocks comprises
a) one or more residual units which contain one or more dilated convolutions with one or more dilation rates
b) at least one down-sampling layer comprising a strided convolution.
For an examination purpose, the latter is applied.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 9-13, and 17 are rejected under 35 U.S.C. 102(a)(1) as
being anticipated by Zhao et al. ("Media2face: Co-speech facial animation generation with multi-modality guidance." ACM SIGGRAPH 2024 conference papers. 2024) (hereinafter Zhao).
Regarding claim 1, Zhao disclose A computer-implemented method comprising: (Figure 1 "We introduce Media2Face, which is capable of generating highly realistic and expressive 3D facial animations from diverse multi-media inputs - audio, text, and image. "; 2.1 3D Facial Animation Representations "Our technique not only aligns better with practical applications but also ensures compatibility with traditional computer graphics (CG) pipelines.")
encoding one or more image-related features by processing one or more portions of input image data using at least one multi-channel image encoder; (Figure 2; .3. 2 Image Facial Expression Extraction “In addition to the geometry VAE, we train two vision encoders, Eexp and Epose, to extract unified expression latent code and head pose from RGB images.”; Figure 3)
PNG
media_image1.png
564
994
media_image1.png
Greyscale
PNG
media_image2.png
421
825
media_image2.png
Greyscale
encoding one or more audio-related features by processing one or more portions of input audio data using at least one audio encoder;
encoding one or more text-related features by processing one or more portions of input text data using at least one text encoder; and (Figure 3)
generating at least one image-based avatar by processing, using at least one vision encoder, at least a portion of the one or more image-related features and one or more of at least a portion of the one or more audio-related features and at least a portion of the one or more text-related features; (Figure 3; Figure 1 "We introduce Media2Face, which is capable of generating highly realistic and expressive 3D facial animations from diverse multi-media inputs—audio, text, and image. With such multi-modality ability, it enables producing accurate and vivid facial expressions while conforming to complex emotional states given diverse media style prompts. The teaser demonstrates our method’s ability to produce vibrant emotional expressions for different avatars, set against stylistic backgrounds that illustrate the diverse media inputs it can process." ;3.2 Image Facial Expression, “In addition to the geometry VAE, we train two vision encoders, Eexp and Epose, to extract unified expression latent code and head pose from RGB images.”
wherein the method is performed by at least one processing device comprising a processor coupled to a memory. (3.2 Image Facial Expression Extraction, “Owing to the rapid inference speed of GNPFA (about 500 fps on Nvidia RTX 3090 GPU), we can efficiently extract high-quality and diverse expressions and head poses from in-the-wild videos.”)
Regarding claim 9, Zhao discloses The computer-implemented method of claim 1, further comprising:
concatenating, into at least one matrix, at least a portion of the one or more image-related features, at least a portion of the one or more audio-related features and at least a portion of the one or more text-related features; and
wherein generating at least one image-based avatar comprises processing the at least one matrix using the at least one vision encoder. (Figure 1 "We introduce Media2Face, which is capable of generating highly realistic and expressive 3D facial animations from diverse multi-media inputs—audio, text, and image. With such multi-modality ability, it enables producing accurate and vivid facial expressions while conforming to complex emotional states given diverse media style prompts. The teaser demonstrates our method’s ability to produce vibrant emotional expressions for different avatars, set against stylistic backgrounds that illustrate the diverse media inputs it can process." Figure 3; Figure 2;
PNG
media_image3.png
393
728
media_image3.png
Greyscale
PNG
media_image4.png
267
335
media_image4.png
Greyscale
)
Regarding claim 10, Zhao discloses The computer-implemented method of claim 1, further comprising: performing one or more automated actions based at least in part on the at least one image- based avatar. (ABSTRACT, “We address this challenge through a trilogy. We first introduce Generalized Neural Parametric Facial Asset (GNPFA), an efficient variational auto-encoder mapping facial geometry and images to a highly generalized expression latent space, decoupling expressions and identities.”; 3 RESHAPE FACIAL ANIMATION DATA, “To address these constraints, we propose Generalized Neural Parametric Facial Asset (GNPFA), which is in essence a Variational Auto Encoder, mapping facial geometry and video footprints to the same latent space. We train GNPFA on large-scale 4D facial scanning, including high-resolution images and artists’ refined geometries, enabling it to produce nuanced facial animation from videos with diverse identities, languages, emotions, and head poses.”; Figure 2; Figure 3; 3.1 Expression Latent Space Learning; 3.2 Image Facial Expression Extraction; 4.1 Facial Animation Latent Diffusion Models; Figure 4; 5.4 User Study; Figure 9)
Regarding claim 11, Zhao discloses The computer-implemented method of claim 10, wherein performing one or more automated actions comprises automatically training, using feedback related to the at least one image-based avatar, at least a portion of one or more of the at least one multi-channel image encoder, the at least one audio encoder, the at least one text encoder, and the at least one vision encoder. (ABSTRACT, “We address this challenge through a trilogy. We first introduce Generalized Neural Parametric Facial Asset (GNPFA), an efficient variational auto-encoder mapping facial geometry and images to a highly generalized expression latent space, decoupling expressions and identities.”; 3 RESHAPE FACIAL ANIMATION DATA, “To address these constraints, we propose Generalized Neural Parametric Facial Asset (GNPFA), which is in essence a Variational Auto Encoder, mapping facial geometry and video footprints to the same latent space. We train GNPFA on large-scale 4D facial scanning, including high-resolution images and artists’ refined geometries, enabling it to produce nuanced facial animation from videos with diverse identities, languages, emotions, and head poses.”; Figure 2; Figure 3; 3.1 Expression Latent Space Learning; 3.2 Image Facial Expression Extraction; 4.1 Facial Animation Latent Diffusion Models)
Regarding claim 12, Zhao discloses The computer-implemented method of claim 10, wherein performing one or more automated actions comprises automatically transmitting the at least one image-based avatar to at least one user device associated with one or more of the input image data, the input audio data, and the input text data. (Figure 2; Figure 4;
PNG
media_image5.png
682
533
media_image5.png
Greyscale
;
5.4 User Study, “Participants ensure side-by-side animations with other methods assessing Media2Face with three conditions: with a specific style prompt for each audio, with a neutral prompt, and without any prompts and head pose animation.”; Figure 9)
Regarding claim 13, claim 13 is a non-transitory processor-readable storage medium claim of method claim 1, except program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device (5.1 Implementation Details) and is accordingly rejected under similar rationale as claim 1. (Media2Face is an image processing model being used to process program code, so the form of a non-transitory processor-readable storage medium is implicit.)
Regarding claim 17, claim 17 is an apparatus claim of method claim 1 except at least one processing device comprising a processor coupled to a memory; the at least one processing device being configured (5.1 Implementation Details) and is accordingly rejected under similar rationale as claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-5, 14, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Zhao et al. ("Media2face: Co-speech facial animation generation with multi-modality guidance." ACM SIGGRAPH 2024 conference papers. 2024) (hereinafter Zhao) in view of Yang et al. ("Constrained R-CNN: A general image manipulation detection model." 2020 IEEE International conference on multimedia and expo (ICME). IEEE, 2020.; IDS REF) (hereinafter Yang).
Regarding claim 2, Zhao does not disclose The computer-implemented method of claim 1, wherein encoding one or more image-related features comprises processing one or more portions of input image data using, as part of the at least one multi-channel image encoder, one or more red-green-blue (RGB) modality (RGBM) techniques, one or more high-frequency image modality (HFIM) techniques, and one or more attention-guided feature learning techniques.
Yang teaches The computer-implemented method of claim 1, wherein encoding one or more image-related features comprises processing one or more portions of input image data using, as part of the at least one multi-channel image encoder, one or more red-green-blue (RGB) modality (RGBM) techniques, (2.1. Overview,
PNG
media_image6.png
321
439
media_image6.png
Greyscale
;
2.2. Learnable manipulation feature extractor,
PNG
media_image7.png
220
458
media_image7.png
Greyscale
;
3.2. Pre-trained Model,
PNG
media_image8.png
159
438
media_image8.png
Greyscale
)
one or more high-frequency image modality (HFIM) techniques, and (1. INTRODUCTION,
PNG
media_image9.png
349
454
media_image9.png
Greyscale
;
2.1. Overview,
PNG
media_image10.png
246
446
media_image10.png
Greyscale
;
2.3. Coarse Manipulation Detection (Stage-1),
PNG
media_image11.png
299
439
media_image11.png
Greyscale
;
PNG
media_image12.png
246
444
media_image12.png
Greyscale
)
one or more attention-guided feature learning techniques. (2.3. Coarse Manipulation Detection (Stage-1),
PNG
media_image13.png
247
466
media_image13.png
Greyscale
)
As both Zhao and Yang are from the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhao to include wherein encoding one or more image-related features comprises processing one or more portions of input image data using, as part of the at least one multi-channel image encoder, one or more red-green-blue (RGB) modality (RGBM) techniques, one or more high-frequency image modality (HFIM) techniques, and one or more attention-guided feature learning techniques, in the context of deep learning, by Zhao according to the teaching of Yang, in order to learn a unified feature representation directly from image data (ABSTRACT of Yang).
Regarding claim 3, Zhao does not disclose The computer-implemented method of claim 2, wherein processing one or more portions of input image data using one or more RGBM techniques comprises learning one or more content-related features of the one or more portions of input image data using at least one convolutional neural network (CNN).
Yang teaches The computer-implemented method of claim 2, wherein processing one or more portions of input image data using one or more RGBM techniques comprises learning one or more content-related features of the one or more portions of input image data using at least one convolutional neural network (CNN). (2.1. Overview,
PNG
media_image6.png
321
439
media_image6.png
Greyscale
;
2.2. Learnable manipulation feature extractor,
PNG
media_image7.png
220
458
media_image7.png
Greyscale
;
3.2. Pre-trained Model,
PNG
media_image8.png
159
438
media_image8.png
Greyscale
)
As both Zhao and Yang are from the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhao to include wherein processing one or more portions of input image data using one or more RGBM techniques comprises learning one or more content-related features of the one or more portions of input image data using at least one convolutional neural network (CNN), in the context of deep learning, by Zhao according to the teaching of Yang, in order to simulate the coarse to fine process (2.1. Overview of Yang).
Regarding claim 4, Zhao does not disclose The computer-implemented method of claim 2, wherein processing one or more portions of input image data using one or more HFIM techniques comprises learning one or more high-frequency features of the one or more portions of input image data by filtering the one or more portions of input image data using at least one constraint convolutional layer and extracting one or more prediction errors based at least in part on the filtering of the one or more portions of input image data.
Yang teaches The computer-implemented method of claim 2, wherein processing one or more portions of input image data using one or more HFIM techniques comprises learning one or more high-frequency features of the one or more portions of input image data by filtering the one or more portions of input image data using at least one constraint convolutional layer and extracting one or more prediction errors based at least in part on the filtering of the one or more portions of input image data. (1. INTRODUCTION,
PNG
media_image9.png
349
454
media_image9.png
Greyscale
;
2.1. Overview,
PNG
media_image10.png
246
446
media_image10.png
Greyscale
;
2.3. Coarse Manipulation Detection (Stage-1),
PNG
media_image11.png
299
439
media_image11.png
Greyscale
;
PNG
media_image12.png
246
444
media_image12.png
Greyscale
)
As both Zhao and Yang are from the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhao to include wherein processing one or more portions of input image data using one or more RGBM techniques comprises learning one or more content-related features of the one or more portions of input image data using at least one convolutional neural network (CNN), in the context of deep learning, by Zhao according to the teaching of Yang, in order to create unified feature representation of various content manipulation directly from data (1. INTRODUCTION of Yang).
Regarding claim 5, Zhao does not disclose The computer-implemented method of claim 2, wherein processing one or more portions of input image data using one or more attention-guided feature learning techniques comprises learning one or more tampering artifacts in the one or more portions of input image data by processing the one or more portions of input image data using at least one convolutional block attention module (CBAM) in conjunction with one or more spatial attention processes and one or more channel attention processes.
Yang teaches The computer-implemented method of claim 2, wherein processing one or more portions of input image data using one or more attention-guided feature learning techniques comprises learning one or more tampering artifacts in the one or more portions of input image data by processing the one or more portions of input image data using at least one convolutional block attention module (CBAM) in conjunction with one or more spatial attention processes and one or more channel attention processes. (2.3. Coarse Manipulation Detection (Stage-1),
PNG
media_image13.png
247
466
media_image13.png
Greyscale
)
As both Zhao and Yang are from the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhao to include wherein processing one or more portions of input image data using one or more attention-guided feature learning techniques comprises learning one or more tampering artifacts in the one or more portions of input image data by processing the one or more portions of input image data using at least one convolutional block attention module (CBAM) in conjunction with one or more spatial attention processes and one or more channel attention processes, in the context of deep learning, by Zhao according to the teaching of Yang, in order to avoid weakening the inter-class discrimination of global features and ultimately impairing manipulation segmentation (2.3. Coarse Manipulation Detection (Stage-1) of Yang).
Regarding claims 14 and 18, similar reasoning as discussed claim 2 is applied.
Claims 6, 15, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Zhao et al. ("Media2face: Co-speech facial animation generation with multi-modality guidance." ACM SIGGRAPH 2024 conference papers. 2024) (hereinafter Zhao) in view of Zeghidour et al. ("Soundstream: An end-to-end neural audio codec." IEEE/ACM Transactions on Audio, Speech, and Language Processing 30 (2021): 495-507.; IDS REF) (hereinafter Zeghidour).
Regarding claim 6, Zhao does not disclose The computer-implemented method of claim 1, wherein encoding one or more audio-related features comprises processing the one or more portions of input audio data using at least one one-dimensional (1D) convolution layer and multiple convolution blocks, wherein each of the multiple convolution blocks comprises one or more residual units, containing one or more dilated convolutions with one or more dilation rates, and at least one down-sampling layer comprising a strided convolution.
Zeghidour teaches The computer-implemented method of claim 1, wherein encoding one or more audio-related features comprises processing the one or more portions of input audio data using at least one one-dimensional (1D) convolution layer and multiple convolution blocks, wherein each of the multiple convolution blocks comprises one or more residual units, containing one or more dilated convolutions with one or more dilation rates, and at least one down-sampling layer comprising a strided convolution. (Fig. 2; Fig. 3;
PNG
media_image14.png
320
715
media_image14.png
Greyscale
PNG
media_image15.png
714
365
media_image15.png
Greyscale
PNG
media_image16.png
382
715
media_image16.png
Greyscale
)
As both Zhao and Zeghidour are from the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhao to include wherein encoding one or more audio-related features comprises processing the one or more portions of input audio data using at least one one-dimensional (1D) convolution layer and multiple convolution blocks, wherein each of the multiple convolution blocks comprises one or more residual units, containing one or more dilated convolutions with one or more dilation rates, and at least one down-sampling layer comprising a strided convolution, in the context of audio enhancement, by Zhao according to the teaching of Zeghidour, in order efficiently to compress speech, music and general audio at bitrates normally targeted by speech-tailored codecs (Abstract of Zeghidour).
Regarding claims 15 and 19, similar reasoning as discussed claim 6 is applied.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Zhao et al. ("Media2face: Co-speech facial animation generation with multi-modality guidance." ACM SIGGRAPH 2024 conference papers. 2024) (hereinafter Zhao) in view of Zia et al. ("Text-to-image generation with attention based recurrent neural networks." arXiv preprint arXiv:2001.06658 (2020).) (hereinafter Zia).
Regarding claim 7, Zhao does not disclose The computer-implemented method of claim 1, wherein encoding one or more text-related features comprises converting at least part of the one or more portions of input text data into one or more numeric representations using at least one recurrent neural network-based (RNN- based) encoder.
Zia teaches The computer-implemented method of claim 1, wherein encoding one or more text-related features comprises converting at least part of the one or more portions of input text data into one or more numeric representations using at least one recurrent neural network-based (RNN- based) encoder. (
PNG
media_image17.png
174
663
media_image17.png
Greyscale
)
As both Zhao and Zia are from the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhao to include wherein encoding one or more text-related features comprises converting at least part of the one or more portions of input text data into one or more numeric representations using at least one recurrent neural network-based (RNN- based) encoder, in the context of text-to-image generation, by Zhao according to the teaching of Zia, in order to map input text description into annotation vectors (4 Proposed Approach of Zia).
Claims 8, 16, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhao et al. ("Media2face: Co-speech facial animation generation with multi-modality guidance." ACM SIGGRAPH 2024 conference papers. 2024) (hereinafter Zhao) in view of Liu et al. ("OPT: Omni-perception pre-trainer for cross-modal understanding and generation." arXiv preprint arXiv:2107.00249 (2021).; IDS REF) (hereinafter Liu).
Regarding claim 8, Zhao disclose The computer-implemented method of claim 1, wherein generating at least one image-based avatar comprises:
processing, at least a portion of the one or more image-related features and one or more of at least a portion of the one or more audio-related features and at least a portion of the one or more text-related features, using at least one transformer decoder to generate at least one image code; and (Figure 3; 5.1 Implementation Details, "For Media2Face, we employ an eight-layer transformer decoder for the denoiser, utilizing four attention heads.")
generating at least one output image comprising at least a portion of the at least one image- based avatar (Figure 2)
Zhao does not disclose
Liu further teaches The computer-implemented method of claim 1, wherein generating at least one image-based avatar comprises:
processing, at least a portion of the one or more image-related features and one or more of at least a portion of the one or more audio-related features and at least a portion of the one or more text-related features, using at least one transformer decoder to generate at least one image code; and (Figure 1, Figure 2)
PNG
media_image18.png
399
881
media_image18.png
Greyscale
generating at least one output image comprising at least a portion of the at least one image- based avatar by processing the at least one image code using at least one discrete variational autoencoder (DVAE) decoder. (
PNG
media_image19.png
275
361
media_image19.png
Greyscale
PNG
media_image20.png
266
416
media_image20.png
Greyscale
)
As both Zhao and Liu are from the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhao to include generating at least one output image comprising at least a portion of the at least one image- based avatar by processing the at least one image code using at least one discrete variational autoencoder (DVAE) decoder, in the context of multi-modal representation or generation, by Zhao according to the teaching of Liu, in order to endow the pre-trained model with the generation capability and also further benefit the representation learning (4.2. Modality-Level Modeling of Liu).
Regarding claims 16 and 20, similar reasoning as discussed claim 8 is applied.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Hyorim Park whose telephone number is (571)272-3859. The examiner can normally be reached Monday - Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at (571) 272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Hyorim Park/Examiner, Art Unit 2615
/JAMES A THOMPSON/Primary Examiner, Art Unit 2615
Prosecution Insights