DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This office action is responsive to the amendment received 04/06/2026.
In the response to the Non-Final Office Action 10/01/2025, the applicant states that claims 1-24 are pending. Claims 1, 23, and 24 are independent claims.
Claims 1, 17, 23, and 24 have been amended. In summary, claims 1-24 are pending in current application.
Response to Arguments
Applicant's arguments filed 04/06/2026 have been fully considered.
Regarding to claim 1, the applicant argues that the cited references do not teach or suggest "a diffusion neural network that generates each of the video frames from a corresponding noisy video frame by performing a sequence of denoising iterations conditioned on the input image and the one or more point trajectories”. The arguments have been fully considered, and are persuasive. Therefore, the 35 U.S.C 103 rejection has been withdrawn. However, upon further consideration, new grounds of rejection are made in newly applied art.
Claims 17, 23 and 24 are not allowable due to a newly applied art as discussed above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6, 9-13, 17, and 21-24 are rejected under 35 U.S.C. 103 as being unpatentable over Walker (An Uncertain Future: Forecasting from Static Images Using Variational Autoencoders) in view of Wang (US 20200342646 A1), and further in view of Pakhomov (US 20240135514 A1).
Regarding to claim 1 (Currently Amended), Walker discloses a method performed by one or more computers and for generating a video that animates an input image across a plurality of time steps, the method (page 835; 1. Introduction: develop a generative framework which, given a static input image, outputs the space of possible future actions; page 836; Fig. 1: generated videos; page 837: predict dense trajectories at each and every pixel using a feedforward Convolutional Network.) comprising:
receiving the input image (page 835; 1. Introduction: receive a static input image; page 838; 3 Algorithm: receive a static, RGB image; page 839; 3.1 Model: receive and pass an image as input);
processing a first input derived from the input image using a first generative neural network to generate respective point trajectories for each of one or more points in the input image (page 835; 1. Introduction: develop a generative framework which, given a static input image, outputs the space of possible future actions; page 838; 3 Algorithm: Let X be the image, and Y be the full set of trajectories; page 839; 3.1 Model: a deep neural network is capable of encoding dependencies between the output trajectories; page 840; 3.3 The Conditional Variational Autoencoder:
PNG
media_image1.png
52
238
media_image1.png
Greyscale
), wherein each point trajectory comprises, for each of the plurality of time steps in the video, a predicted spatial position of the corresponding point in a video frame at the time step in the video (page 835; 1. Introduction: develop a generative framework which, given a static input image, outputs the space of possible future actions; our model characterizes the whole distribution of future states and are used to sample multiple possible future events; page 836; Fig. 1: predict multiple correct one-second motion trajectories given the scene;
PNG
media_image2.png
726
870
media_image2.png
Greyscale
; predict dense trajectories and capture most of a video’s content; page 838: predict for a relatively long period of time: one second; page 839; 3.1 Model: if there are multiple possible futures given an image, then for each possible future, there will be a different set of z values which map to that future; page 840; 3.3 The Conditional Variational Autoencoder:
PNG
media_image1.png
52
238
media_image1.png
Greyscale
); and
generating each of the video frames in the video that animates the input image (page 836; Fig. 1: generated videos; page 838: our algorithm predicts from a single image—which may enable graphics applications that involve animating still photographs;
PNG
media_image3.png
228
906
media_image3.png
Greyscale
; page 844; Fig. 3: a full view of two predicted motions in 3D space-time; on the left is the projection of the trajectories onto the image plane; Best seen in our generated videos from single image; page 844; Fig. 3: Best seen in our videos;
PNG
media_image4.png
198
832
media_image4.png
Greyscale
;
PNG
media_image5.png
206
838
media_image5.png
Greyscale
).
Walker fails to explicitly disclose:
using a second generative neural network and based on the input image and the one or more point trajectories, wherein the second generative neural network comprises a diffusion neural network that generates each of the video frames from a corresponding noisy video frame by performing a sequence of denoising iterations conditioned on the input image and the one or more point trajectories.
In same field of endeavor, Wang teaches:
generate video using a second generative neural network and based on the input image and the one or more point trajectories (Fig. 1; [0030]: the pose-to-dance NN applies a third RNN to the waveform latent vector L1, the points P, and images 160 of a person in a plurality of different poses to generate a video 170 of the person dancing;
PNG
media_image6.png
392
588
media_image6.png
Greyscale
; Fig. 2; [0038-0039]; Fig. 7; [0056]: apply an output of the second RNN to a second CNN to generate the video).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Walker to include generate video using a second generative neural network and based on the input image and the one or more point trajectories as taught by Wang. The motivation for doing so would have been to apply 3D rendering to visualize a predicted motion sequence; to provide better prediction of human body motion under a generalized scenario, provides more subjective dancing motion, and provides human body movements with a higher degree of freedom relative to conventional systems; to generate a video 170 of the person dancing and to apply an output of the second RNN to a second CNN to generate the video as taught by Wang in Fig. 7 and paragraphs [0004], [0023], [0030] and [0056].
Walker in view of Wang fails to explicitly disclose:
wherein the second generative neural network comprises a diffusion neural network that generates each of the video frames from a corresponding noisy video frame by performing a sequence of denoising iterations conditioned on the input image and the one or more point trajectories.
In same field of endeavor, Pakhomov teaches:
wherein the second generative neural network comprises a diffusion neural network that generates each of the video frames from a corresponding noisy video frame by performing a sequence of denoising iterations conditioned on the input image and the one or more point trajectories ([0559]: the scene-based image editing system 106 uses another generative neural network as the mask completion neural network 4510; [0562]: the scene-based image editing system 106 uses a generative neural network 4516 to generate completed objects 4518; the scene-based image editing system 106 uses various generative neural networks; Fig. 46A; [0592]: the diffusion neural network 4602 executes an iterative denoising process to generate the at least partially denoised image 4612; the incrementally denoised image 4610 represents the output of a given iteration before the final iteration of the denoising process, and the completed object 4618 represents the output of the final iteration of the denoising process).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Walker and Wang to include “wherein the second generative neural network comprises a diffusion neural network that generates each of the video frames from a corresponding noisy video frame by performing a sequence of denoising iterations conditioned on the input image and the one or more point trajectories” as taught by Pakhomov. The motivation for doing so would have been to improve the image quality; to offer improved efficiency; to provide improved efficiency by reducing the user interactions required in determining these characteristics; to use various generative neural networks; to execute an iterative denoising process to generate the at least partially denoised image 4612 by the diffusion neural network 4602 as taught by Pakhomov in paragraphs [0165], [0228], [0232], [0562], and [0592].
Regarding to claim 2 (Original), Walker in view of Wang and Pakhomov discloses the method of claim 1, wherein each point trajectory comprises, for each of the time steps, (i) the predicted spatial position of the corresponding point in the video frame at the time step (Walker; page 835; 1. Introduction: develop a generative framework which, given a static input image, outputs the space of possible future actions; our model characterizes the whole distribution of future states and are used to sample multiple possible future events; page 836; Fig. 1: predict multiple correct one-second motion trajectories given the scene) and (ii) an occlusion score that estimates a likelihood that the corresponding point will be occluded in the video frame at the time step (page 839; 3.1. Model: estimate the likelihood of sampling each possible future; page 839; 3.2 Training by Autoencoding: estimate the overall likelihood of Y; page 844; Fig. 3: the likelihood of occlusion is estimated to be very low as illustrated in Fig. 3; Best seen in our videos.
PNG
media_image4.png
198
832
media_image4.png
Greyscale
;
PNG
media_image5.png
206
838
media_image5.png
Greyscale
).
Walker in view of Wang further discloses (ii) an occlusion score that estimates a likelihood that the corresponding point will be occluded in the video frame at the time step (Wang; [0051]: the occlusion between people is handled by a layer depth map).
Regarding to claim 3 (Original), Walker in view of Wang and Pakhomov discloses the method of claim 1, further comprising:
processing the input image using an image encoder neural network to generate an encoded representation of the input image (Walker; page 837: a conditional variational auto-encoder; page 840; Fig. 2: the decoder produces trajectories depending both on the image information as well as output from the encoder; page 840; 3.3 The Conditional Variational Autoencoder; page 841: 3.4 Architecture), wherein the first input comprises the encoded representation of the input image (Walker; page 840; Fig. 2: the decoder produces trajectories depending both on the image information as well as output from the encoder; page 840; 3.3 The Conditional Variational Autoencoder; page 841: 3.4 Architecture).
Walker in view of Wang and Pakhomov further discloses:
processing the input image using an image encoder neural network to generate an encoded representation of the input image (Wang; Fig. 2; [0041]: the appearance encoder 152 includes a convolutional neural network, CNN, that operates on the input images 160 of a target person in various poses), wherein the first input comprises the encoded representation of the input image (Wang; Fig. 2; [0041]: the appearance encoder 152 includes a convolutional neural network, CNN, that operates on the input images 160 of a target person in various poses; the function of the appearance encoder 152 is to let the neural network know which person is to be animated; Fig. 7; [0056]: apply a first CNN to the images of the target person and apply an output of the second RNN to a second CNN to generate the video).
Same motivation of claim 1 is applied here.
Regarding to claim 4 (Original), Walker in view of Wang and Pakhomov discloses the method of claim 1, wherein the first generative neural network is a diffusion neural network that generates each point trajectory from a corresponding noisy trajectory conditioned on the first input (Walker; page 839; 3.1 Model: z is random Gaussian noise: passing an image as input and sampling from the noise variable allows us to sample from the model’s posterior given the image; page 840; 3.3 The Conditional Variational Autoencoder: Gaussian noise; page 841: neural networks; neural network layers).
Regarding to claim 5 (Original), Walker in view of Wang and Pakhomov discloses the method of claim 4, wherein the first generative neural network has been trained on a training data set that comprises (i) a plurality of video sequences (Walker; page 843; 4 Experiments: train our network on videos from the UCF101 dataset) and (ii) for each of the video sequences, a respective point trajectory for each of one or more points in a first frame within the video sequence (Walker; page 843; 4 Experiments: we utilized as much training data as possible, i.e., all the videos except for a small hold out set for every action; randomly sample 2800 frames and their corresponding trajectories for our testing data; page 844; Fig. 3: Best seen in our videos
PNG
media_image4.png
198
832
media_image4.png
Greyscale
).
Regarding to claim 6 (Original), Walker in view of Wang and Pakhomov discloses the method of claim 4, wherein the first generative neural network comprises a two-dimensional convolutional neural network (Wang; [0041]: the appearance encoder 152 includes a convolutional neural network, i.e., CNN; Fig. 5A; [0046]).
Same motivation of claim 1 is applied here.
Regarding to claim 9 (Original), Walker in view of Wang and Pakhomov discloses the method of claim 4, wherein each corresponding noisy trajectory comprises a concatenation of noisy coordinates (Walker; page 839; 3.1 Model: z is random Gaussian noise: passing an image as input and sampling from the noise variable allows us to sample from the model’s posterior given the image; page 839: sample from the noise variable; the likelihood of sampling each possible future will be proportional to the likelihood of sampling a z value, i.e. a noise, that maps to it; page 840; 3.3 The Conditional Variational Autoencoder: Gaussian noise; a distribution of z values which are likely to give rise to Yi given Xi; page 841; 3.4 Architecture: compute three separate functions: μ(X, z); page 843: the total loss function) and
Walker in view of Wang and Pakhomov further discloses a noisy occlusion estimate (Wang; [0051]: the occlusion between people is handled by a layer depth map).
Same motivation of claim 1 is applied here.
Regarding to claim 10 (Original), Walker in view of Wang and Pakhomov discloses the method of claim 9, wherein the noisy coordinates are augmented with a positional encoding (Walker; page 837: our conditional variational auto-encoder outputs a mapping from noise variables; page 839: sample from the noise variable; the likelihood of sampling each possible future will be proportional to the likelihood of sampling a z value, i.e. a noise, that maps to it; page 840: Gaussian noise.).
Regarding to claim 11 (Original), Walker in view of Wang and Pakhomov discloses the method of claim 5, wherein, for at least a subset of the video sequences, one or more of the point trajectories have been generated by processing an input comprising the corresponding point in the first frame using a point tracking neural network (Walker; page 843: the amount of absolute motion varies considerably; total loss function; where Y represents trajectories, X is the image, Mi are the global magnitudes, and ˆ Y , ˆMi are the corresponding estimates by our network; page 844; Fig. 3: Best seen in our videos;
PNG
media_image4.png
198
832
media_image4.png
Greyscale
;
PNG
media_image5.png
206
838
media_image5.png
Greyscale
).
Regarding to claim 12 (Original), Walker in view of Wang and Pakhomov discloses the method of claim 11, wherein the point tracking neural network is configured to, for each video sequence in the subset and for a corresponding point in the first frame in the video (same as rejected in claim 11):
generate a query feature for the corresponding point (Walker; page 836: Fig. 1; page 844: Fig. 3 );
generate, using the query feature, a cost volume that comprises a respective cost map for each of a plurality of frames in the video sequence (Wang; [0051]: the occlusion between people is handled by a layer depth map); and
generate, for each of the plurality of frames, an initial position of the corresponding point in the frame and an initial occlusion estimate for the corresponding point in the frame (Wang; page 839; 3.1. Model: estimate the likelihood of sampling each possible future; page 839; 3.2 Training by Autoencoding: estimate the overall likelihood of Y; page 844; Fig. 3: the likelihood of occlusion is estimated to be very low as illustrated in Fig. 3; Best seen in our videos.
PNG
media_image4.png
198
832
media_image4.png
Greyscale
;
PNG
media_image5.png
206
838
media_image5.png
Greyscale
).
Walker in view of Wang and Pakhomov further discloses using the cost map for the frame (Wang; [0051]: the occlusion between people is handled by a layer depth map).
Regarding to claim 13 (Original), Walker in view of Wang and Pakhomov discloses the method of claim 12, wherein the point tracking neural network is further configured to:
generate the point trajectory for the corresponding point by refining the initial positions, refining the initial occlusion estimates for the plurality of frames, or both, using an initial point trajectory that comprises the initial positions and the initial occlusion estimates for the plurality of frames (Walker; page 839: 3.2 Training by “Autoencoding”; page 840; Fig. 2: during training, the inputs to the network include both the image and the ground truth trajectories; page 841; page 843: Coarse-to-Fine).
Regarding to claim 17 (Currently Amended), Walker in view of Wang and Pakhomov discloses the method of claim 1, wherein the second generative neural network generates each frame from the corresponding noisy frame conditioned on a conditioning input derived from the input image and the one or more point trajectories (Pakhomov; [0559]: the scene-based image editing system 106 uses another generative neural network as the mask completion neural network 4510; [0562]: the scene-based image editing system 106 uses a generative neural network 4516 to generate completed objects 4518; the scene-based image editing system 106 uses various generative neural networks; Fig. 46A; [0592]: the diffusion neural network 4602 executes an iterative denoising process to generate the at least partially denoised image 4612; the incrementally denoised image 4610 represents the output of a given iteration before the final iteration of the denoising process, and the completed object 4618 represents the output of the final iteration of the denoising process).
Same motivation of claim 1 is applied here.
Regarding to claim 21 (Original), Walker in view of Wang and Pakhomov discloses the method of claim 1, wherein the input image depicts a real-world environment at a particular time and each of the video frames is a prediction of the real-world environment at a corresponding time after the particular time (Walker; page 836; Fig. 1: predict multiple correct one-second motion trajectories given the scene;
PNG
media_image2.png
726
870
media_image2.png
Greyscale
; predict dense trajectories and capture most of a video’s content).
Regarding to claim 22 (Original), Walker in view of Wang and Pakhomov discloses the method of claim 1, wherein the input image depicts a person having a pose at a particular time and each of the video frames is a prediction of a pose that the person will adopt at a corresponding time after the particular time (Walker; Walker; page 836; Fig. 1: predict multiple correct one-second motion trajectories given the scene;
PNG
media_image2.png
726
870
media_image2.png
Greyscale
; predict dense trajectories and capture most of a video’s content; page 844; Fig. 3).
Regarding to claim 23 (Currently Amended), Walker discloses a system (page 835; 1. Introduction: develop a generative framework which, given a static input image, outputs the space of possible future actions; page 836; Fig. 1: generated videos; page 837: predict dense trajectories at each and every pixel using a feedforward Convolutional Network.) comprising:
In same field of endeavor, Wang teaches:
one or more computers ([0068]: one or more processors and system memory); and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations ([0068]: a processor, e.g., a microprocessor, receives instructions, from a non-transitory computer-readable medium, and executes those instructions, thereby performing one or more processes) comprising:
The rest claim limitations are similar to claim limitations recited in claim 1. Therefore, same rational used to reject claim 1 is also used to reject claim 23.
Regarding to claim 24 (Currently Amended), Walker discloses a method (page 835; 1. Introduction: develop a generative framework which, given a static input image, outputs the space of possible future actions; page 836; Fig. 1: generated videos; page 837: predict dense trajectories at each and every pixel using a feedforward Convolutional Network.):
In same field of endeavor, Wang teaches: one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising ([0068]: one or more processors and system memory; a processor, e.g., a microprocessor, receives instructions, from a non-transitory computer-readable medium, and executes those instructions, thereby performing one or more processes).
The rest claim limitations are similar to claim limitations recited in claim 1. Therefore, same rational used to reject claim 1 is also used to reject claim 24.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Walker (An Uncertain Future: Forecasting from Static Images Using Variational Autoencoders) in view of Wang (US 20200342646 A1), in view of Pakhomov (US 20240135514 A1), and further in view of Mallya (US 20240095989 A1).
Regarding to claim 7 (Original), Walker in view of Wang and Pakhomov discloses the method of claim 6,
Walker in view of Wang and Pakhomov fails to explicitly disclose wherein the two-dimensional convolutional neural network is a U-Net.
In same field of endeavor, Mallya teaches:
wherein the two-dimensional convolutional neural network is a U-Net (Fig. 2; [0069]: U-nets; one or more circuits execute an U-net; two U-nets take inputs at ¼ of image resolution; [0083]: a similar U-net is used to encode a source image 408).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Walker in view of Wang and Pakhomov to include wherein the two-dimensional convolutional neural network is a U-Net as taught by Mallya. The motivation for doing so would have been to improve accuracy when animating images; to use a similar U-net to encode a source image; to improve output quality as taught by Mallya in paragraphs [0056], [0083], and [0085].
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Walker (An Uncertain Future: Forecasting from Static Images Using Variational Autoencoders) in view of Wang (US 20200342646 A1), in view of Pakhomov (US 20240135514 A1), and further in view of Bakunov (US 20240297957 A1).
Regarding to claim 8 (Original), Walker in view of Wang and Pakhomov discloses the method of claim 6,
Walker in view of Wang and Pakhomov fails to explicitly disclose:
wherein the two-dimensional convolutional neural network comprises one or more self-attention layers.
In same field of endeavor, Bakunov teaches:
wherein the two-dimensional convolutional neural network comprises one or more self-attention layers ([0098]: a cross-attention layer is inserted between a self-attention layer and a feed-forward network; [0099]: It uses causal self-attention layers).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Walker in view of Wang to include wherein the two-dimensional convolutional neural network comprises one or more self-attention layers as taught by Bakunov. The motivation for doing so would have been to display overlays on the video during presentation to a recipient user; to insert a cross-attention layer between a self-attention layer and a feed-forward network; to use causal self-attention layers; to improve training efficiency while benefiting from multi-task learning as taught by Bakunov in paragraphs [0068] and [0098-0100].
Allowable Subject Matter
Claims 14-16 and 18-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Hai Tao Sun whose telephone number is (571)272-5630. The examiner can normally be reached 9:00AM-6:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at 5712727642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAI TAO SUN/Primary Examiner, Art Unit 2616
Prosecution Insights