DETAILED ACTION
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 2-5, 8, 19-21 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Singer ("Make-A-Video: Text-to-Video Generation without Text-Video Data", cited from IDS).
Regarding clam 2, Singer teaches a method performed by one or more computers, the method comprising:
receiving a text prompt describing a scene(Input Text x in Figure 2);
processing the text prompt, using a text encoder neural network, to generate a contextual embedding of the text prompt(P prior in Figure 2);
processing the contextual embedding, using a sequence of generative neural networks, to generate a final output representation of a video depicting the scene(
y
^
in Figure 2)
wherein the sequence of generative neural networks comprises:
an initial generative neural network configured to:
receive the contextual embedding(P prior fps in Figure 2); and
process the contextual embedding to generate, as output, an initial output representation of the video having an initial dimensionality(P prior in Figure 2); and
one or more subsequent generative neural networks each configured to:
receive a respective input comprising an input representation of the video generated as output by a preceding generative neural network in the sequence( Frame Interpolation in Figure 2); and
process the respective input to generate, as output, a respective output representation of the video having higher dimensionality than the input representation(
S
R
l
t
in Figure 2); and
processing the final output representation of the video, using an output neural network, to generate the video depicting the scene(
y
⏞
in Figure 2).
Regarding clam 3, Singer teaches the method of claim 2, wherein the final output representation of the video is the respective output representation of a final generative neural network in the sequence(
S
R
h
in Figure. 2).
Regarding clam 4, Singer teaches the method of claim 2, wherein the respective input of each subsequent generative neural network in the sequence further comprises the contextual embedding of the text prompt( Prior P in Figure 2).
Regarding clam 5, Singer teaches the method of claim 2, wherein the video has higher dimensionality than each output representation of the video(
S
R
h
in Figure. 2) .
Regarding clam 8, Singer teaches the method of claim 2, wherein the output neural network is a post-processor neural network, and each output representation of the video is a respective output pixel representation of the video(
y
⏞
in Figure 2).
Regarding clam 19, Singer teaches the method of claim 2, wherein the one or more subsequent generative neural networks are a plurality of subsequent generative neural networks (
S
R
l
t
,
S
R
h
in Figure 2) .
Claims 19 and 20 recite the system and medium for the method in claim 1. Since Singer inherently teaches a computer implemented method, claims 19 and 20 are also rejected
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Singer.
Regarding clam 6, Singer teaches the method of claim 2, wherein the output neural network is a decoder neural network(
D
t
in Figure 2).
Singer does not expressly teach each output representation of the video is a respective output compressed representation of the video.
However, it is well known in the art to compress video.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to compress the video generated by Singer, with motivation to save storage.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Singer in view of CHAKRABORTY ( US 20180225808).
Regarding clam 9, Singer teaches the method of claim 8.
Singer does not expressly teach wherein the post-processor neural network is configured to: receive the final output pixel representation of the video; and apply one or more video enhancement transformations to the final output pixel representation of the video to generate the video depicting the scene.
However, CHAKRABORTY teaches receive the final output pixel representation of the video(340 in Fig. 3); and apply one or more video enhancement transformations to the final output pixel representation of the video to generate the video depicting the scene( 250 in Fig. 3).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Singer and CHAKRABORTY , by recoding final images from Singer following the teaching of CHAKRABORTY, with motivation “to enhance the visual quality of the encoded images and/or videos”( CHAKRABORTY, [0011]).
Claim(s) 10-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Singer in view of Meng (“On Distillation of Guided Diffusion Models”, 2022).
Regarding clam 10, Singer teaches the method of claim 2.
Singer does not expressly teach wherein each generative neural network in the sequence is a diffusion-based generative neural network.
However, Meng teaches each generative neural network in the sequence is a diffusion-based generative neural network((3, our approach for distilling a classifier-free guided diffusion model into a student model that requires fewer steps to sample from).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Singer and Meng, by generating subsequent videos as taught by Meng following the initial video generated by Singer, with motivation to “capture a wide range of classifier-free guidance levels, allowing for the trade-off between sample quality and diversity efficiently” (Meng, 3.)
Regarding clam 11, Singer in view of Meng teaches the method of claim 10, wherein each diffusion-based generative neural network in the sequence is a denoising diffusion-based generative neural network( Meng, 4.1, use 1024 DDIM denoising steps).
Regarding clam 12, Singer teaches the method of claim 11, wherein each denoising diffusion-based generative neural network in the sequence implements a continuous-time parametrization to generate the respective output representation of the video(Meng, 3.3, one deterministic sampling step with two-times the original step-length(i.e., the same as a N/2-stepdeterministicsampler)and then perform one stochastic step backward(i.e., perturb with noise) using the original step-length).
Regarding clam 13, Singer in view of Meng teaches the method of claim 12, wherein each denoising diffusion-based generative neural network in the sequence implements a v-prediction parametrization to generate the respective output representation of the video( Meng, 4.2, we fine-tune the open-sourced ϵ-prediction models into v-prediction teacher models).
Regarding clam 14, Singer in view of Meng teaches the method of claim 13, wherein each denoising diffusion-based generative neural network in the sequence further implements progressive distillation to generate the respective output representation of the video( Meng, Figure 5).
Claim(s) 15-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Singer in view of Hong ( CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers, 2022).
Regarding clam 15, Singer teaches the method of claim 2, wherein:
the generative neural networks in the sequence have been jointly trained on a training dataset comprising a plurality of training examples( Singer, 4.1, datasets);
Singer does not express
each training example comprises: (i) a respective input text prompt describing a respective scene, and (ii) a respective target video depicting the respective scene; and
the text encoder neural network is pre-trained and was held frozen during the joint training of the generative neural networks.
However, Hong teaches
each training example comprises: (i) a respective input text prompt describing a respective scene, and (ii) a respective target video depicting the respective scene( Fig. 1); and
the text encoder neural network is pre-trained and was held frozen during the joint training of the generative neural networks ( 3.2, The proposed technique is dual-channel attention, where we only add a new spatial-temporal attention channel to the pretrained CogView2 [6] at each transformer layer. All the parameters in the CogView2 are frozen in the training, and only the parameters in the newly added attention layer (See the attention plus in Figure 3) are trainable).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Singer and Hong , by substituting the training dataset and generative neural networks in Singer with those taught by Hong, with motivation “to better align text and video semantics”( Hong, 3).
Regarding clam 16, Singer in view of Hong teaches the method of claim 15, wherein training the initial generative neural network on the training dataset comprised:
for each training example in the training dataset:
processing the respective input text prompt, using the text encoder neural network, to generate a contextual embedding of the respective input text prompt;
processing the respective target video to generate an initial output representation of the respective target video having the initial dimensionality( Hong, Image Tokenizer in Figure 2); and
returning a respective initial training pair comprising:
the contextual embedding of the respective input text prompt( Hong, Text tokenization in Figure 2); and
the initial output representation of the respective target video (Hong, Flatten in Figure 2) ; and
training the initial generative neural network on training data comprising the respective initial training pair for each training example (Hong, page 4, in each training sample, we want the text and the frames to match as possible) .
Regarding clam 17, Singer in view of Hong teaches the method of claim 16, wherein training each subsequent generative neural network in the sequence on the training dataset comprised:
for each training example in the training dataset
processing the respective target video to generate an output representation of the respective target video for the subsequent generative neural network( Hong, Stage 1 in Figure 2);
receiving the output representation of the respective target video for the preceding generative neural network in the sequence( Hong, interpolate frames in Figure 2); and
returning a respective subsequent training pair comprising:
a respective training input comprising the output representation of the respective target video for the preceding generative neural network in the sequence(Hong, Sequence 1 in Figure 2); and
(ii) the output representation of the respective target video for the subsequent generative neural network( Hong, Text tokenization in Figure 2); and
training the subsequent generative neural network on training data comprising the respective subsequent training pair for each training example ( Hong, Page 4, we adopt Cross-Modal General Language Model(CogLM) proposed in[6] which unites bidirectional context-aware mask prediction and autoregressive generation by dividing tokens into unidirectional and bidirectional attention regions).
Regarding clam 18, Singer in view of Hong teaches the method of claim 17, wherein for each subsequent generative neural network in the sequence and each training example in the training dataset, the respective training input of the respective subsequent training pair further comprises the contextual embedding of the respective input text prompt ( Hong,
x
i
n
x
o
u
t
in Figure 3).
Allowable Subject Matter
Claims 7 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANGENG SUN whose telephone number is (571)272-3712. The examiner can normally be reached 8am to 5pm, EST, M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Randolph Vincent can be reached at 571 272 8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JIANGENG SUN
Examiner
Art Unit 2661
/Jiangeng Sun/Examiner, Art Unit 2671