Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-7, 9, and 13-19 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Siarohin, “First Order Motion Model for Image Animation”. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.
Regarding claim 1, Siarohin discloses: a method of decoding a bitstream to output one or more pictures for a video stream, the method comprising:
receiving a bitstream comprising one or more types of facial representation parameters (See ); and
decoding, using coded information of the bitstream, one or more pictures, wherein the decoding comprises (See Section 3, “Method”, 3rd para., “The locations of the keypoints in D and S are separately predicted by an encoder-decoder network.”):
decoding the one or more types of facial representation parameters (See page 4, “In the first step, we approximate both transformations from sets of sparse trajectories, obtained by using keypoints learned in a self-supervised way. Note in particular in Section 4: Metrics: “For the VoxCeleb and Nemo datasets we use the facial landmark detector of Bulat et al. [5].” Siarohin’s general motion is tested in at least one instance on facial data, deriving keypoints therefore.);
converting the one or more types of facial representation parameters into one or more dense motion flows having a common format (See description in Section 3, 2nd para. about modeling a dense motion field by a function T-s←D: R2 →R2 that maps each pixel location in D with its corresponding location S.); and
generating a facial picture based on the one or more dense motion flows and a key reference picture of the one or more pictures (See section 3, final paragraph, “Finally, the generation module renders an image of the source object moving as provided in the driving video.”).
Regarding claim 2, Siarohin discloses the method according to claim 1, wherein converting the one or more types of facial representation parameters into one or more dense motion flows having the common format further comprises:
converting the one or more types of facial representation parameters by one or more dense motion flow translators into the one or more dense motion flows (See Section 3.4: At this stage our goal is to animate an object in a source frame S1 using the driving video D1, ... DT.”), wherein one dense motion flow translator corresponds to one type of the one or more types of facial representation parameters (See mapping from driving frame D to source frame S, as disclosed in Section 3, “Method”: The motion field is modeled by a function T←D : IR2 ➔ IR2 that maps each pixel location in D with its corresponding location in S. T←D.).
Regarding claim 3, Siarohin discloses the method according to claim 2, wherein the one or more types of facial representation parameters comprise: 2-dimesnional (2D) key point, 3-dimensional (3D) key point, compact feature, or facial semantics (See Section 3, 2nd para. “The keypoint representation acts as a bottleneck resulting in a compact motion representation.” The keypoints are 2-dimensional keypoints.).
Regarding claim 4, Siarohin discloses: the method according to claim 1, wherein generating the facial picture based on the one or more dense motion flows further comprises:
generating the facial picture by a generator (See Equation 7.), and the common format of the one or more dense motion flows satisfies a requirement of the generator (Siarohin discloses in Section 3, 3rd para. that the self-supervised learning of keypoints in a source image, which are used by motion estimation module to anchor a backward optical flow estimation from a driving frame to a source frame, as disclosed in Section 3.1 1st para.).
Regarding claim 5, Siarohin discloses: the method according to claim 4, wherein the common format is a first common format, and the method further comprises:
converting the one or more types of facial representation parameters to obtain one or more occlusion maps having a second common format (As shown in figure 2, unsupervised keypoint detector extracts first order motion representation consisting of sparse keypoints and local affine transformations by learning a source image S, and a dense motion network uses the motion representation to generate dense optical flow TS←D from D to S and occlusion map OS←D.); and
generating the facial picture based on the one or more dense motion flows, the key reference picture, and the one or more occlusion maps (Section 3, 1st paragraph, “perform image animation of the source object.”).
Regarding claim 6, Siarohin discloses: the method according to claim 5, wherein generating the facial picture based on the one or more dense motion flows, the key reference picture, and the one or more occlusion maps further comprises:
warping the key reference picture according to the one or more dense motion flows (Section 3.1, equation 4, and in “Combining Local Motions” section, “In order to provide inputs already roughly aligned with Ts+--D, we we warp the source frame S according to local transformations estimated in Eq. (4).”); and
generating the facial picture by masking out feature map region using the one or more occlusion maps (See Section 3.2, where Siarohin discloses introducing an occlusion map to mask out the feature map regions that need to be inpainted for subsequent optical flow generation (facial picture) generation.).
Regarding claim 7, Siarohin discloses: the method according to claim 6, wherein the generator is trained with the key reference picture (See Section 3, Method: “Our model is trained to reconstruct the training videos by combining a single frame and a learned latent representation of the motion in the video.”).
Regarding claim 9, Siarohin discloses: the method according to claim 2, wherein the one or more dense motion flow translators are trained using staged single-module, and under supervision of using an L1 reconstruction loss (See Section 4, “Evaluation Protocol: Metrics”, where Siarohin discloses: “We report the average L1 distance between the generated and the ground-truth videos.”).
Claims 10-12 are rejected under 35 U.S.C. 102(a)(2) as being unpatentable over Chen et al., US 2024/0251098 A1.
Non-transitory computer-readable storage medium claims 10-12 are directed to a non-transitory computer-readable storage medium storing a bitstream produced according to a method. This claim is a product-by-process claim, where the product (non-transitory computer-readable storage medium storing a bitstream) is produced by the claimed method. Such a bitstream is disclosed in Chen et al., which discloses in the The bitstream includes an encoded reference picture; and encoded facial semantics of a plurality of inter frames, wherein the facial semantics are determined based on the reference frame and the plurality of inter frames. Chen thus discloses a bitstream that anticipates the bitstream of claim 10. See MPEP 2113.I. “"[E]ven though product-by-process claims are limited by and defined by the process, determination of patentability is based on the product itself. The patentability of a product does not depend on its method of production. If the product in the product-by-process claim is the same as or obvious from a product of the prior art, the claim is unpatentable even though the prior product was made by a different process." In re Thorpe, 777 F.2d 695, 698, 227 USPQ 964, 966 (Fed. Cir. 1985)
Decoder claim 13, 16, 17, 18, 19 are rejected for the same reason of anticipation as given above for claim 1, 4, 5, 6, 7 respectively.
Regarding claim 14, Siarohin discloses: the decoder according to claim 13, further comprising a general decoder configured to decode the bitstream to obtain the key reference picture (See mapping from driving frame D to source frame S, as disclosed in Section 3, “Method”: The motion field is modeled by a function T←D : IR2 ➔ IR2 that maps each pixel location in D with its corresponding location in S. T←D. See also ).
Regarding claim 15, Siarohin discloses: the decoder according to claim 13, wherein one dense motion flow translator corresponds to one type of the one or more types of facial representation parameters (See Section 3.4: At this stage our goal is to animate an object in a source frame S1 using the driving video D1, ... DT.).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 8 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Siarohin, in view of Tulyakov, et al. MoCoGAN:Decomposing Motion and Content for Video Generation. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Regarding claim 8, Siarohin discloses: the method according to claim 7, wherein the generator is trained under supervision of using a perceptual loss (See Section 3.3, “First, we use the reconstruction loss based on the perceptual loss of Johnson et al. [19] using the pre-trained VGG-19 network as our main driving loss.”)
Siarohin does not disclose training the generator also using an adversarial loss
However, in Section I, “Introduction” Siarohin discloses: In particular, Generative Adversarial Networks (GANs) [14] and Variational Auto-Encoders (VAEs) [20] have been used to transfer facial expressions [37] or motion patterns [3] between human subjects in videos. Nevertheless, these approaches usually rely on pre-trained models in order to extract object-specific representations such as keypoint locations.”
However, in an analogous art, Tulyakov discloses MoCoGAN framework for video generation, that “map[s] a sequence of random vectors to a sequence of video frames. Each random vector consists of a content part and a motion part. While the content part is kept fixed, the motion part is realized as a stochastic process. To learn motion and content decomposition in an unsupervised manner, we introduce a novel adversarial learning scheme utiliz ing both image and video discriminators.” See Tulyakov abstract.
It would have been obvious to one having ordinary skill in the art before the time of the applicant’s effective filing date to combine a perceptual loss training, as disclosed in Siarohin, with an adversarial loss training used to learn a latent representation of motion and disambiguate motion from object content, as disclosed in Tulyakov, for the generator, in order to complement the perceptual loss training already disclosed in Siarohin with a training method that can use unsupervised learning to learn keypoints in a source frame and separate them from motion information. See Tulyakov, “Conclusion”. Incorporating an adversarial network alongside the perceptual loss network for training the motion generator would have entailed simply combining the prior art elements respectively disclosed in Siarohin and in Tulyakov without changing their respective functions, and the combination would have yielded nothing more than predictable results for one of ordinary skill in the art. KSR Int'l Co. v. Teleflex Inc. See 2143.1.A. 550 U.S. at 416, 82 USPQ2d at 1395.
Decoder claim 20 is directed to a decoder for decoding a bitstream that performs a decoding method corresponding to the decoding of claim 8. Therefore, decoder claim 20 is rejected for the same reasons of obviousness as given above with respect to decoding method claim 8.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYLE M LOTFI whose telephone number is (571)272-8762. The examiner can normally be reached 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Brian Pendleton can be reached at 571-272-7527. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KYLE M LOTFI/ Examiner, Art Unit 2425
Prosecution Insights