DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on November 11, 2024 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is considered by examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 8-12, 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Petersen et al (US 2024/0378698) in view of Chu et al (Video ControlNet: Towards Temporally Consistent Synthetic-to-Real Video Translation using Conditional Image Diffusion Models).
Regarding Claim 1, Petersen et al teach a computer-implemented method for generating data (video model 600 for video enhancement; Fig 6 and ¶ [0063]), the method comprising:
determining a first set of flow vectors between a first input frame and a second input frame (a previous (first frame) low-resolution frame 605 and an input (second frame) low-resolution frame 604 are input to a flow estimate engine 614 to determine a first set of flow vectors; Fig 6 and [0063]-[0064]);
generating, based on the first set of flow vectors and a first noise sample associated with the first input frame, a second noise sample associated with the second input frame (the noise warping engine 624 can adjust the previous (first noise) noise input 622 (interpreted as associated with previous low-resolution frame 605) by an amount indicated by flow vector for each pixel of input frame 604 (which flow vector accounts for relationship between frames 604-605) to generate warped (second noise) noise 619 (thereby associated with frame 604 via flow vectors); Fig 6 and ¶ [0065]); and
converting, via execution of the diffusion model, the second input frame into a second output frame based on the second noise sample (the input (second frame) low-resolution frame 604 with warped noise 619 (replacement of the input noise 606) (along with frame 617) into the video enhancement diffusion model 608 to generate the output (second output) upsampled frame 610; Fig 6 and ¶ [0066]-[0067]).
Petersen et al does not teach converting, via execution of a diffusion model, the first input frame into a first output frame based on the first noise sample.
Chu et al is analogous art pertinent to the technological problem addressed in the current application and teaches converting, via execution of a diffusion model, the first input frame into a first output frame based on the first noise sample (the stable diffusion model receives the input video frame with corresponding optimal input noise to output a denoised (synthesized) frame, with the process recursively performed with corresponding optimized noise samples for the input image; Fig 2 and 3 Video ControlNet, 3.1 Flow warping, 3.2 Crafting optimal noise, 3.3 Frame Interpolation).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Petersen et al with Chu et al including converting, via execution of a diffusion model, the first input frame into a first output frame based on the first noise sample. By using an input noise tunable to the specific frame along with the optical flow correspondences between adjacent video frames, the output video frame from the diffusion model creates temporal consistency among the pixels in each frame, thereby improving the image generation consistency between consecutive frames of a video, as recognized by Chu et al (1. Introduction ¶ 2-3).
Regarding Claim 2, Petersen et al in view of Chu et al teach the computer-implemented method of claim 1 (as described above), further comprising: determining a second set of flow vectors between a third input frame and the second input frame (Petersen et al, optical flow (motion) vectors are generated between the video frames (including a second and third frame based on same rationale described in [0063]-[0064]); Fig 6 and ¶ [0079]); and further generating the second noise sample based on the second set of flow vectors and a third noise sample associated with the third input frame (Petersen et al, an updated input noise can be generated for a given frame based on the (motion) vector (based on same rationale described in [0065]); ¶ [0082]).
Regarding Claim 3, Petersen et al in view of Chu et al teach the computer-implemented method of claim 2 (as described above), wherein the third input frame temporally precedes the second input frame within a video (Chu et al, frame interpolation is imposed for noisy latents (understood as when larger transitions occur between frames) and could be between a frame that temporally precedes the second video frame input; 3.3 Frame interpolation).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Petersen et al with Chu et al including wherein the third input frame temporally precedes the second input frame within a video. By using the preceding frame for determining the flow vectors, a consistent optimization process is performed resulting in a continuous evaluation for the pixels between the frames, resulting in an optimized process with seamless variations between the textures and colors in the generated video, as recognized by Chu et al (1. Introduction ¶ 2-3), 3.3 Frame interpolation).
Regarding Claim 8, Petersen et al in view of Chu et al teach the computer-implemented method of claim 1 (as described above), further comprising: determining a second set of flow vectors between the first input frame and a third input frame (Petersen et al, optical flow (motion) vectors are generated between the video frames (including a second and third frame based on same rationale described in [0063]-[0064]); Fig 6 and ¶ [0079]); generating, based on the second set of flow vectors and the first noise sample associated with the first input frame, a third noise sample associated with the third input frame (Petersen et al, an updated input noise can be generated for a given frame based on the flow vector (based on same rationale described in [0065]); ¶ [0082]); and converting, via execution of the diffusion model, the third input frame into a third output frame based on the third noise sample (Petersen et al, the video enhancement diffusion model will generate the output frame for the given frame and input noise based on the flow vector (as on the same rationale described in¶ [0066]-[0067]); ¶ [0082]).
Regarding Claim 9, Petersen et al in view of Chu et al teach the computer-implemented method of claim 1 (as described above), wherein the first input frame comprises a starting frame within a video and the second input frame temporally follows the first input frame within the video (Petersen et al, a first frame of a video sequence can be used as a first input frame with a second frame may be a temporally second frame to the first frame within the video sequence; ¶ [0062]).
Regarding Claim 10, Petersen et al in view of Chu et al teach the computer-implemented method of claim 1 (as described above), wherein the first set of flow vectors comprises at least one of a motion vector or an optical flow (Petersen et al, the flow estimate engine 614 generates an optical flow vector (or motion vector); Fig 6 and [0063], [0065]).
Regarding Claim 11, Petersen et al teach one or more non-transitory computer-readable media storing instructions (SoC 100 includes memory block 118 with instructions stored therein and executed on CPU 102; Fig 1 and ¶ [0032]) that, when executed by one or more processors (CPU 102 executes instructions; Fig 1 and ¶ [0032], [0034], [0042]), cause the one or more processors to perform the steps of: limitations identical to claim 1 (as described above).
Regarding Claim 12, Petersen et al in view of Chu et al teach the one or more non-transitory computer-readable media of claim 11 (as described above), wherein the instructions further cause the one or more processors to perform the steps of: limitations identical to claim 2 (as described above).
Regarding Claim 18, Petersen et al in view of Chu et al teach the one or more non-transitory computer-readable media of claim 11 (as described above), wherein the first input frame temporally precedes the second input frame (a previous (first frame) low-resolution frame 605 temporally precedes the input (second frame) low-resolution frame 604 in a video sequence; Fig 6 and [00623]-[0064]).
Regarding Claim 19, Petersen et al in view of Chu et al teach the one or more non-transitory computer-readable media of claim 11 (as described above), wherein the first output frame and the second output frame comprise at least one of (interpreted as one of the following outputs) edits to the first input frame and the second input frame, restoration of the first input frame and the second input frame, one or more conditions specified in the first input frame and the second input frame, or higher-resolution versions of the first input frame and the second input frame (Petersen et al, each of the output upsampled frame 610 has a higher resolution than the input low-resolution frame; Fig 6 and ¶ [0066]).
Regarding Claim 20, Petersen et al teach a system (system on chip (SoC) 100; Fig 1 and ¶ [0032]), comprising: one or more memories that store instructions (SoC 100 includes memory block 118 with instructions stored therein; Fig 1 and ¶ [0032]) and one or more processors that are coupled to the one or more memories (memory block 118 is coupled with CPU 102; Fig 1 and ¶ [0032]) and, when executing the instructions (CPU 102 executes instructions; Fig 1 and ¶ [0032], [0034], [0042]), are configured to perform the steps of: limitations identical to claim 1 (as described above).
Allowable Subject Matter
Claims 4-7, 13-17 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding Claim 4, the following limitations, in combination with the claims in which it depends, were not readily found to be taught, suggested or found obvious in combination upon the prior art:
Claim 4. The computer-implemented method of claim 1, wherein generating the second noise sample comprises: upsampling a first plurality of noise values included in the first noise sample into a second plurality of noise values; and determining a third plurality of noise values included in the second noise sample based on the second plurality of noise values and the first set of flow vectors.
Claims 5-7 are dependent upon claim 4 and therefore allowable for similar reasons.
Regarding Claim 13, the following limitations, in combination with the claims in which it depends, were not readily found to be taught, suggested or found obvious in combination upon the prior art:
Claim 13. The one or more non-transitory computer-readable media of claim 12, wherein further generating the second noise sample comprises: upsampling a first plurality of noise values included in the third noise sample into a second plurality of noise values; determining a first plurality of locations within the second input frame that are associated with undefined noise values; matching, based on the second set of flow vectors, the first plurality of locations to a second plurality of locations within the third input frame; and aggregating a subset of the second plurality of noise values associated with the second plurality of locations into a first noise value that is (i) associated with the first plurality of locations and (ii) included in the second noise sample.
Regarding Claim 14, the following limitations, in combination with the claims in which it depends, were not readily found to be taught, suggested or found obvious in combination upon the prior art:
Claim 14. The one or more non-transitory computer-readable media of claim 11, wherein generating the second noise sample comprises: upsampling a first plurality of noise values included in the first noise sample into a second plurality of noise values; matching, based on the first set of flow vectors, a first plurality of locations within the second input frame to a second plurality of locations within the first input frame; and aggregating a subset of the second plurality of noise values associated with the second plurality of locations into a second noise value that is (i) associated with the first plurality of locations and (ii) included in the second noise sample.
Claims 15-17 are dependent upon claim 14 and therefore allowable for similar reasons.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Min et al (US 2024/0087179) teach a method and system for training an image generation model that incorporates a diffusion model for generating a video sequence based on an input image and a text condition.
Buades et al (Enhancement of Noisy and Compressed Videos by Optical Flow and Non-Local Denoising) teach a method and system for denoising video sequences based on optical flow analysis between frames and compensation for spatial-temporal comparison to correct noise.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATHLEEN M BROUGHTON whose telephone number is (571)270-7380. The examiner can normally be reached Monday-Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATHLEEN M BROUGHTON/Primary Examiner, Art Unit 2661
Prosecution Insights