Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The use of the terms “LINUX” and “ANDROID” in [0068] of the specification, which is a trade name(s) or a mark(s) used in commerce, has been noted in this application. The term should be accompanied by the generic terminology; furthermore the term should be capitalized wherever it appears or, where appropriate, include a proper symbol indicating use in commerce such as ™, SM , or ® following the term.
Although the use of trade names and marks used in commerce (i.e., trademarks, service marks, certification marks, and collective marks) are permissible in patent applications, the proprietary nature of the marks should be respected and every effort made to prevent their use in any manner which might adversely affect their validity as commercial marks.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 10, and 16 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation "the masked images" in line 10. There is insufficient antecedent basis for this limitation in the claim. In line 8, a singular masked image is introduced, therefore, for the sake of further prosecution, Examiner will interpret this limitation to read “deriving the pose or visage based on the driving image and the masked image by the second sub model;”.
Independent Claims 10 and 16 recite identical limitations and are rejected under the same rationale.
Dependent Claims 2-9, 11-15, and 17-20 are rejected due to dependency on Claims 1, 10, and 16 respectively.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 8, 10, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Rochow et al., “FSRT: Facial Scene Representation Transformer for Face Reenactment from Factorized Appearance, Head-post, and Facial Expression Features”, 15 April 2024, hereinafter referenced as Rochow, in view of Knaan et al. (US 20260228944 A1, effectively filed 9 May 2023), hereinafter referenced as Knaan.
Regarding Claim 1, Rochow discloses
A method of generating images using a machine learning model (Rochow: [1. Introduction, pg. 1], discloses a method for face reenactment; [3.2. Augmentation and Regularization, pg. 5], discloses the generated output is an image; see [Fig. 2] below, illustrating the method’s transformer-based architecture <machine learning model>), comprising:
PNG
media_image1.png
770
1318
media_image1.png
Greyscale
receiving a source image and a driving image by the machine learning model, wherein the source image comprises a portrait of a first subject, the driving image comprises a second subject that is different from the first subject, the driving image depicts a pose or a visage, and the machine learning model comprises a first sub-model and a second sub-model (Rochow: [Fig. 1, pg. 1], illustrates an overview of the method, where a source image and a driving image are received by the transformer-based architecture <machine learning model, see [6. Conclusion, pg. 8]>, wherein the source image comprises a portrait of a first subject and the driving image comprises a second subject that is different than the first subject and depicts a post and a visage; [3.1. Input and Query Representation, Fig. 2, pg. 3-4], illustrates the transformer-based architecture which comprises a keypoint detector, a fully convolutional hourglass network <interpreted as a first sub-model> and a Patch CNN and a Transformer <interpreted together as a multi-modal second sub-model>);
PNG
media_image2.png
396
608
media_image2.png
Greyscale
extracting appearance features of the first subject from the source image by the first sub-model (Rochow: [3.1. Input and Query Representation, pg. 4], discloses extracting facial keypoints <appearance features> of the first subject from the source image using the keypoint detector <first sub-model>);
and generating an image by the machine learning model, wherein the generated image preserves the appearance features of the first subject and follows the pose or the visage depicted in the driving image (Rochow: [Fig. 1, pg. 1], illustrates a generated output image by the method <a transformer-based architecture which is a machine learning model, see [Fig. 2] and [6. Conclusion, pg. 8]> wherein the output <generated image> preserves the appearance features of the first subject and follows the pose and visage depicted in the driving frame <image>).
driving image (Rochow: [Fig. 1], illustrates the driving image)
Rochow fails to explicitly disclose
generating a masked image based on the driving image, wherein the masked image comprises at least one of a mouth region or eye regions in the driving image;
deriving the pose or the visage based on the driving image and the masked images by the second sub-model;
However, Knaan discloses
generating a masked image based on the (Knaan: [0010], discloses generating a preserving mask based on the initial image <masked image>, wherein the preserving mask <masked image> corresponds to the face of the subject from the initial image <a face comprises at least one of a mouth or eye region>);
deriving the pose or the visage based on the (Knaan: [0120], discloses outputting from a second CNN <second sub-model of the system’s architecture>, based on the preserving mask and the processed initial image, a generated image with the original face preserved <derived visage>; see [Fig. 9] below);
PNG
media_image3.png
552
984
media_image3.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply and/or modify the method disclosed by Rochow by preserving facial features through a mask when generating an image with a second sub-model as taught by Knaan. One of ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to make this modification to ensure continuity and prevent artifacts in facial reenactment.
Regarding Claim 10, it discloses limitations similar to Claim 1 but as a system. As shown in the rejection, the combination of Rochow and Knaan discloses the method of Claim 1. The combination of Rochow and Knaan further disclose(s)
A system of generating images using a machine learning model, comprising: at least one processor; and at least one memory communicatively coupled to the at least one processor and comprising computer-readable instructions that upon execution by the at least one processor cause the at least one processor to perform operations (Knaan: [0010], discloses a system for generating images using a machine learning model that includes a processor and a memory coupled to the processor, with instructions stored thereon that, when executed by the processor, cause the processor to perform operations of a computer-implemented method) comprising: …
It would have been obvious to one of ordinary skill in the art before the effective filing date to apply and/or modify the method disclosed by the combination of Rochow and Knaan by using a system, device with a memory and at least one processor to execute operations as further taught by Knaan. One of ordinary skill in the art would have been motivated to make this modification consolidate the performed method to one device with real-time data processing and instruction execution.
Regarding Claim 16, it discloses limitations similar to Claims 1 and 10 but as a non-transitory computer-readable storage medium. As shown in the rejection, the combination of Rochow and Knaan disclose the method and system of Claims 1 and 10 respectively. The combination of Rochow and Knaan further disclose(s)
A non-transitory computer-readable storage medium, storing computer-readable instructions that upon execution by a processor cause the processor to implement operations (Knaan: [0008], discloses a non-transitory computer-readable medium includes instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations of a computer-implemented method) comprising: …
It would have been obvious to one of ordinary skill in the art before the effective filing date to apply and/or modify the method or system disclosed by the combination of Rochow and Knaan by using a non-transitory computer-readable storage medium as further taught by Knaan. One of ordinary skill in the art would have been motivated to make this modification because a non-transitory computer-readable storage medium is highly reliable for long-term data archiving and will not be affected by power outages.
PNG
media_image4.png
515
556
media_image4.png
Greyscale
Regarding Claim 3, the combination of Rochow and Knaan disclose(s) the method of Claim 2. The combination of Rochow and Knaan further disclose(s) wherein the applying a cross-identity training scheme comprises:
generating cross-identity image pairs each of which comprises different subjects (Rochow: [Fig. 3, pg. 7], illustrates a pair of images generated using cross-reenactment relative motion transfer, where the generated images comprise different subjects);
and training the second sub-model on the cross-identity image pairs to mitigate appearance leakage from driving signals (Rochow: [3.2. Augmentation and Regularization, pg. 5], discloses the network <including the interpretation of the second-sub model> learning to decouple appearance, pose, and expression using loss functions to mitigate leakage of colors in the driving image; [3.2. Augmentation and Regularization, pg. 5-6; Fig. 3, pg. 7], discloses three phases of training where loss functions optimize the network, the Phase 1 training outputs illustrated in Fig. 3, interpreted as the gross identity image pairs, are used for the subsequent phases of training; [8.7. Statistical Regularization], discloses the statistical regularization mitigates identity information leaks from the driving frame <leakage from driving signals, wherein driving signals include landmarks, segmentations and dense poses of the driving image>; see [Fig. 16, pg. 23] for further examples with relative motion transfer mitigating leakage).
Regarding Claims 11 and 17, the combination of Rochow and Knaan disclose(s) the system and non-transitory computer-readable storage medium of Claims 10 and 16 respectively. The combination of Rochow and Knaan further disclose(s) wherein
the second sub-model is trained by applying a cross-identity training scheme, wherein the cross- identity training scheme is configured to instruct the second sub-model to derive identity- disentangled poses or visages (Rochow: [3.3. Training, pg. 6; 4.3. Cross-reenactment, pg. 8], discloses training the network <including the Patch CNN and transformer, interpreted as the second sub model> by using a relative motion transfer for cross reenactment <interpreted as a cross-identity training scheme>; [3.2. Augmentation and Regularization], discloses the network is configured to learn <instruct the models> decouple appearance, pose, and expression information <interpreted as deriving identity-disentangled poses and visages>),
and wherein the applying a cross-identity training scheme comprises:
generating cross-identity image pairs each of which comprises different subjects (Rochow: [Fig. 3, pg. 7], illustrates a pair of images generated using cross-reenactment relative motion transfer, where the generated images comprise different subjects);
and training the second sub-model on the cross-identity image pairs to mitigate appearance leakage from driving signals (Rochow: [3.2. Augmentation and Regularization, pg. 5], discloses the network <including the interpretation of the second-sub model> learning to decouple appearance, pose, and expression using loss functions to mitigate leakage of colors in the driving image; [3.2. Augmentation and Regularization, pg. 5-6; Fig. 3, pg. 7], discloses three phases of training where loss functions optimize the network, the Phase 1 training outputs illustrated in Fig. 3, interpreted as the gross identity image pairs, are used for the subsequent phases of training; [8.7. Statistical Regularization], discloses the statistical regularization mitigates identity information leaks from the driving frame <leakage from driving signals, wherein driving signals include landmarks, segmentations and dense poses of the driving image>; see [Fig. 16, pg. 23] for further examples with relative motion transfer mitigating leakage).
Regarding Claim 8, the combination of Rochow and Knaan disclose(s) the method of Claim 1. The combination of Rochow and Knaan further disclose(s)
PNG
media_image5.png
160
128
media_image5.png
Greyscale
wherein the pose comprises a head pose, and the visage comprises a facial visage (Rochow: [Fig. 1], illustrates the driving frame, with a person’s head pose and their facial expression <visage is the literal term for a human face>; see snippet of the input Driving frame of Fig. 1 on the left, illustrating a head pose and facial visage).
Regarding Claim 2, the combination of Rochow and Knaan disclose(s) the method of Claim 1. The combination of Rochow and Knaan further disclose(s) wherein
the second sub-model is trained by applying a cross-identity training scheme, and the cross-identity training scheme is configured to instruct the second sub-model to derive identity-disentangled poses or visages (Rochow: [3.3. Training, pg. 6; 4.3. Cross-reenactment, pg. 8], discloses training the network <including the Patch CNN and transformer, interpreted as the second sub model> by using a relative motion transfer for cross reenactment <interpreted as a cross-identity training scheme>; [3.2. Augmentation and Regularization], discloses the network is configured to learn <instruct the models> decouple appearance, pose, and expression information <interpreted as deriving identity-disentangled poses and visages>).
Claims 4, 9, 12, 15, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Rochow and Knaan in view of Yin et al., “StyleHEAT: One-Shot High-Resolution Editable Talking Face generation via Pre-trained StyleGAN, 2022.
Regarding Claims 4, 12, and 18, the combination of Rochow and Knaan disclose(s) the method, system, and non-transitory computer-readable storage medium of Claims 3, 11, and 17 respectively. the combination of Rochow and Knaan teach generating each cross-identity image pair (Rochow: [3.2. Augmentation and Regularization]) but fail to explicitly disclose the limitations of Claims 4, 12, and 18, however, Yin disclose(s) wherein generating each cross-identity image pair comprises:
selecting an appearance reference image and a reconstruction target image that feature a same subject (Yin: [4.1 Video-Driven Motion Generator, pg. 8; Fig. 3], illustrates a framework pipeline of inputting <selecting> a driving video of a subject including multiple frames, a first frame interpreted as a reconstruction target image, and a second frame interpreted as an appearance reference image);
PNG
media_image6.png
796
1364
media_image6.png
Greyscale
and generating a control image by a pre-trained image reenactment generator, wherein the control image features a subject different from the subject in the appearance reference image and the reconstruction target image, and wherein the control image shares motion information with the reconstruction target image (Yin: [Fig. 3], illustrates generating an output frame <control image> by the pre-trained, disclosed in [4.1 Video-Driven Motion Generator, pg. 8], image reenactment generation framework, wherein the output frame features a different subject from the frames of the driving video, and wherein the output image shares motion information with the first input frame of the driving image).
PNG
media_image7.png
118
126
media_image7.png
Greyscale
PNG
media_image8.png
194
204
media_image8.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply and/or modify the method, system, or non-transitory computer-readable storage medium disclosed by the combination of Rochow and Knaan by using multiple frames of a single subject to output a frame of a different subject with the motion information of the first subject as taught by Yin. One of ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to make this modification for identity preservation and high accuracy image generation.
Regarding Claims 9 and 15, the combination of Rochow and Knaan disclose(s) the method and system of Claims 1 and 10 respectively. The combination of Rochow and Knaan teach facial reenactment using a source image and a driving video, with inputs including a driving frame to the multi-modal model but fails to explicitly disclose the limitations of Claims 9 and 15, however, Yin disclose(s)
receiving the source image and a driving video by the machine learning model, wherein the driving video comprises a sequence of frames and features the second subject with motions associated with a head or a face (Yin: [Fig. 4, pg. 5], discloses a Video-based Motion Generator, wherein a the framework comprising a Calibration Net <interpreted as a machine learning model, see [4.3 Feature Calibration and Joint Training, pg. 7]> and a GAN <also a machine learning model> receives a source image and a processed driving video, wherein the driving video comprises a sequence of frames and features a second subject with motions associated with the head and face, as pictured below; see [4.1 Video-Driven Motion Generator, pg. 6] also);
PNG
media_image6.png
796
1364
media_image6.png
Greyscale
and generating a video by the machine learning model, wherein the generated video preserves the appearance features of the first subject and follows the motions depicted in the driving video (Yin: [Fig. 4, pg. 5], illustrates a generated output video by the framework, wherein the generated video preserves the appearance features of the subject of the source image and follows the motions of the driving video <in reference to the Video-based Motion Generator embodiment>; see also [pg. 1]; see [Fig. 1] below with Video Driven Talking Face Generation).
PNG
media_image9.png
628
1350
media_image9.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply and/or modify the method or system disclosed by the combination of Rochow and Knaan by using a driving video generate a video as taught by Yin. One of ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to make this modification to add dynamic motion and expressiveness to the generate face reenactments; further, this method of more efficient that 3D modelling.
Claims 5, 13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Rochow, Knaan, and Yin in view of Hong et al., “Depth-Aware Generative Adversarial Network for Talking Head Video Generation”, 2022, hereinafter referenced as Hong.
Regarding Claims 5, 13, and 19, the combination of Rochow, Knaan, and Yin disclose(s) the method, system, and non-transitory computer-readable storage medium of Claims 4, 12, and 18 respectively. The combination of Rochow, Knaan, and Yin fail to explicitly disclose the limitations of Claims 5, 13, and 19, however, Hong disclose(s)
generating local control images based on control images in the cross-identity image pairs, wherein each of the local control images comprises at least one of a mouth region or eye regions (Hong: [Hong: [3. The proposed DaGAN Approach, pg. 2-3; Fig. 2], discloses generating a depth map and a feature warped appearance representation <each a local control image> based on the input images of different subjects <control images in the cross identity image pair>, wherein each of the local control region comprises a representation of the subject’s face <at least one of a mouth or eye regions>; see [Fig. 2] below);
PNG
media_image10.png
720
1340
media_image10.png
Greyscale
and guiding the second sub-model to enhance attention to local facial movements using the local control images (Hong: [3. The proposed DaGAN Approach, pg. 2-3], discloses a cross-model attention learning sub-network to use facial landmarks <guiding/enhancing attention to local facial movements> using the depth maps and warped appearance representations <local control images>).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply and/or modify the method disclosed by the combination of Rochow, Knaan, and Yin by using the local attention mechanism as taught by Hong. One of ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to make this modification of using local attention for higher accuracy or realistic motion transfer between two faces for facial reenactment.
Claims 6-7, 14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Rochow, Knaan, and Yin in view of Zheng et al., “General Facial Representation Learning in a Visual-Linguistic Manner”, 2022.
Regarding Claims 6, 14, and 20, the combination of Rochow, Knaan, and Yin disclose(s) the method, system, and non-transitory computer-readable storage medium of Claims 4, 12, and 18 respectively. The combination of Rochow, Knaan, and Yin further disclose(s)
performing a random pixel offset during training to estimate loss (Rochow: [3.3. Training, pg. 6], discloses applying a random pixel offset during training to estimate loss)
The combination of Rochow, Knaan, and Yin fail to explicitly disclose
performing random heterogeneous scaling operations on control images and local control images during training to force the machine learning model to derive appearance features from appearance reference images.
However, Zheng discloses
performing random heterogeneous scaling operations on control images and local control images during preprocessing to force the machine learning model to derive appearance features from appearance reference images (Zheng: [G. More Details], discloses performing random rescaling to augment a matrix, then transforming the image <control image> and ground truth label maps <interpreted as a local control image because it includes: “including not only facial components but also body parts and accessories like eyeglasses, earrings and necklaces” in [3.5. Downstream Face Tasks pg. 4]> using the augmented matrix <performs random scaling operations, these operations are interpreted as heterogeneous because they are being applied to various, in this case two different, images> during preprocessing to ensure the network can segment the whole face and as well as focus on the face region from the LaPa dataset <interpreted as appearance reference images>).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply and/or modify the method disclosed by the combination of Rochow, Knaan, and Yin by performing random heterogeneous scaling operations to derive appearance features as taught by Zheng. One of ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to make this modification to improve identity preservation.
Regarding Claim 7, the combination of Rochow, Knaan, Yin, and Zheng disclose(s) the method of Claim 6. The combination of Rochow, Knaan, Yin, and Zheng further disclose(s)
where a random scaling factor of the random heterogeneous scaling operations is greater than or equal to 0.9 and less than or equal to 1.1 (Zheng: [G. More Details, pg. 10], discloses a rescaling factor of random rescaling <interpreted as heterogeneous because it is applied to various, in this case two different, images> of [0.9, 1.1] <greater than or equal to 0.9 and less than or equal to 1.1>).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply and/or modify the method disclosed by the combination of Rochow, Knaan, Yin, and Zheng by random scaling by a factor within [0.9, 1.1] as further taught by Zheng. One of ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to make this modification to strike a balance between identity preservation and motion expressiveness.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Kim et al., “Deep Video Portraits”, 2018 discloses a method of receiving a source video and driving video by the machine learning model, wherein the driving video comprises a sequence of frames and features the second subject with motions associated with the head or face; and generating a video by the machine learning model, wherein the generated video preserves the appearance features of the first subject and follows the motions depicted in the driving video.
PNG
media_image11.png
448
1260
media_image11.png
Greyscale
Wang et al., “One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing”, 2022 discloses a cross-identity training and reenactment.
PNG
media_image12.png
416
624
media_image12.png
Greyscale
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ISABELLA OCHSNER whose telephone number is (571)272-9322. The examiner can normally be reached 9:30 - 6:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached at (571) 272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/I.O./Examiner, Art Unit 2618
/DEVONA E FAULK/Supervisory Patent Examiner, Art Unit 2618
Prosecution Insights