DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on November 11, 2024 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are considered by examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5, 8-11, 15-18, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Petersen et al (US 2024/0378698) in view of Joung et al (Learning Canonical 3D Object Representation for Fine-Grained Recognition).
Regarding Claim 1, Petersen et al teach a computer-implemented method for generating data (video model 600 for video enhancement; Fig 6 and ¶ [0063]), the method comprising:
determining a plurality of flow vectors between a plurality of regions within a canonical space (“canonical space” is given its plain meaning for using a common coordinate system, specification ¶ [0009]-[0010]) and a plurality of target spaces (a previous (first frame) low-resolution frame 605 and an input (second frame) low-resolution frame 604 are input to a flow estimate engine 614 to determine a first set of flow vectors; Fig 6 and [0063]-[0064]);
generating, based on the plurality of flow vectors and a first noise sample associated with the canonical space, a plurality of noise samples associated with the plurality of target spaces (the noise warping engine 624 can adjust the previous (first noise) noise input 622 (as associated with previous low-resolution frame 605) by an amount indicated by flow vector for each pixel of input frame 604 (which flow vector accounts for relationship between frames 604-605) to generate warped (second noise) noise 619 (thereby associated with frame 604 via flow vectors) and understood a plurality of similar noise samples are associated with each time step and sequence of frames towards generating the output frame; Fig 3, 6 and ¶ [0044]-[0046], [0065]-[0067]);
generating, via execution of a diffusion model based on the plurality of noise samples, a plurality of denoised intermediate samples associated with the plurality of target spaces (the diffusion model may include a reverse diffusion process that gradually removes noise samples, based on each noise time point from the forward diffusion process, thereby generating denoised intermediate samples based on the time steps during the reverse diffusion process; Fig 3, 6 and ¶ [0044]-[0046], [0066]-[0069]);
blending the plurality of denoised intermediate samples based on the plurality of flow vectors to generate a plurality of blended denoised intermediate samples associated with the plurality of target spaces (the diffusion model 608 can resample the video enhancement at each time point with the one or more previous frames with optical flow to enforce consistent texture on objects; Fig 3, 6 and ¶ [0050], [0066]-[0069]); and
generating an output frame based on the plurality of blended denoised intermediate samples (a output (second output) upsampled frame 610 is generated by the video enhancement diffusion model 608 based on the spatial-temporal modeling; Fig 6 and ¶ [0050], [0066]-[0067], [0072]),
wherein the output frame comprises a projection of a plurality of diffusion outputs that correspond to the plurality of blended denoised intermediate samples from the plurality of target spaces onto the plurality of regions within the canonical space (the output frame 610 is based on spatial and temporal consistency with previous frames based on a combination of warping previous noise and newly sampled noise over time points; Fig 6 and ¶ [0066]-[0072]).
Petersen et al does not explicitly teach a canonical space is used between a plurality of regions and target spaces to determine the flow vectors.
Joung et al is analogous art pertinent to the technological problem addressed in the current application and teaches a canonical space (“canonical space” is given its plain meaning for using a common coordinate system, specification ¶ [0009]-[0010]) is used between a plurality of regions and target spaces to determine the flow vectors (the input image is embedded based on coordinate positions within the image and is used based on appearance flow modeling to transform the input image to a canonical object representation based on the appearance flow at given point coordinates of the object; Fig 2, 3 and 3.1 Preliminaries, 3.3 Disentangling Object Variation Appearance flow, 3.4 Canonical Object Representation Embedding appearance).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Petersen et al with Joung et al including a canonical space used between a plurality of regions and target spaces to determine the flow vectors. By learning canonical space of the input image and an appearance flow, the input may be warped into a canonical configuration with reduced spatial variation, thereby improving the appearance of the generated image appearance, as recognized by Joung et al (1. Introduction ¶ 2).
Regarding Claim 5, Petersen et al in view of Joung et al teach the computer-implemented method of claim 1 (as described above), wherein generating the plurality of denoised intermediate samples (Petersen et al, the diffusion model may include a reverse diffusion process that gradually removes noise samples, based on each noise time point thus generating denoised intermediate samples based on the time step; Fig 3, 6 and ¶ [0044]-[0046], [0066]-[0069]) comprises:
generating, via execution of the diffusion model, a noise prediction associated with a noise sample included in the plurality of noise samples (Petersen et al, the diffusion model forward process gradually adds noise samples over iterations to the image in a (predicted) fixed noise sample, which corresponding noise samples are used in the reverse diffusion; ¶ [0044]-[0046]); and
updating the noise sample based on the noise prediction to generate a denoised intermediate sample that is included in the plurality of denoised intermediate samples (Petersen et al, the input noise can be sampled based on a target distribution of noise at each time stamp; Fig 6 and ¶ [0061]-[0062], [0065]).
Regarding Claim 8, Petersen et al in view of Joung et al teach the computer-implemented method of claim 1 (as described above), wherein generating the output frame (Petersen et al, an output (second output) upsampled frame 610 is generated by the video enhancement diffusion model 608 based on the spatial-temporal modeling; Fig 6 and ¶ [0050], [0066]-[0067]) comprises:
converting the plurality of blended denoised intermediate samples associated with a first diffusion time step into a plurality of noisy intermediate samples associated with a second diffusion time step (Petersen et al, a warped noise 619 can be used in conjunction with a input noise 606 (as a previously sampled noise and current re-sampled noise) with a given time step for the diffusion model 608 to further denoise the input image into a subsequent time step; Fig 6 and ¶ [0050], [0066]-[0069]);
generating the plurality of diffusion outputs based on the plurality of noisy intermediate samples (Petersen et al, the (diffused) output upsampled frame 610 is based on the input noise 606 and the warped noise 619 and occurs for each time step; Fig 6 and ¶ [0065]-[0067]); and
combining the plurality of diffusion outputs into the output frame (Petersen et al, the output upsampled frame 610 is based the compilation of each previous time output; Fig 6 and ¶ [0066]).
Regarding Claim 9, Petersen et al in view of Joung et al teach the computer-implemented method of claim 1 (as described above), wherein each denoised intermediate sample included in the plurality of denoised intermediate samples is further generated based on a set of conditions (Petersen et al, at each intermediate sample as it is denoised, the previous noise input 622 can be adjusted per pixel indicated by a respective optical flow vector for each pixel of the input low-resolution frame 604; Fig 3, 6 and ¶ [0065]).
Regarding Claim 10, Petersen et al in view of Joung et al teach the computer-implemented method of claim 9 (as described above), wherein the set of conditions comprises at least one of a prompt, a pixel value, or a pose (Petersen et al, the influence of adjusting the noise input is based on the pixel value associated with the optical flow vector; Fig 6 and ¶ [0065]).
Regarding Claim 11, Petersen et al teach one or more non-transitory computer-readable media storing instructions (SoC 100 includes memory block 118 with instructions stored therein and executed on CPU 102; Fig 1 and ¶ [0032]) that, when executed by one or more processors (CPU 102 executes instructions; Fig 1 and ¶ [0032], [0034], [0042]), cause the one or more processors to perform the steps of: steps identical to claim 1 (as described above).
Regarding Claim 15, Petersen et al in view of Joung et al teach the one or more non-transitory computer-readable media of claim 11 (as described above), wherein further steps are performed identical to claim 5 (as described above).
Regarding Claim 16, Petersen et al in view of Joung et al teach the one or more non-transitory computer-readable media of claim 11 (as described above), wherein blending the plurality of denoised intermediate samples (Petersen et al, denoised intermediate samples are generated at each time step and blended with the previous output frame 603 via processing by the diffusion model 608; Fig 3, 6 and ¶ [0045]-[0047], [0064]-[0067]) comprises generating a blended denoised intermediate sample included in the plurality of blended denoised intermediate samples based on a first set of pixel values from a corresponding denoised intermediate sample and one or more additional sets of pixel values from one or more additional denoised intermediate samples included in the plurality of denoised intermediate samples (Petersen et al, at each intermediate sample as it is denoised, the previous noise input 622 can be adjusted per pixel indicated by a respective optical flow vector for each pixel of the input low-resolution frame 604 and is combined at each time step (blending denoised intermediate samples iteratively) to result in the generated output upsampled frame 610; Fig 3, 6 and ¶ [0045]-[0047], [0064]-[0067]).
Regarding Claim 17, Petersen et al in view of Joung et al teach the one or more non-transitory computer-readable media of claim 11 (as described above), wherein generating the output frame (Petersen et al, an output (second output) upsampled frame 610 is generated by the video enhancement diffusion model 608 based on the spatial-temporal modeling; Fig 6 and ¶ [0050], [0066]-[0067]) comprises: converting the plurality of blended denoised intermediate samples associated into the plurality of diffusion outputs (Petersen et al, the (diffused) output upsampled frame 610 is based on the input noise 606 and the warped noise 619 (blended intermediate) and occurs for each time step (plurality of blending at each time step); Fig 3, 6 and ¶ [0045]-[0046], [0065]-[0067]); and combining the plurality of diffusion outputs into the output frame (Petersen et al, the output upsampled frame 610 is based the compilation of each previous blended time output; Fig 3, 6 and ¶ [0045], [0066]).
Regarding Claim 18, Petersen et al in view of Joung et al teach the one or more non-transitory computer-readable media of claim 11 (as described above), wherein the plurality of flow vectors comprises a mapping between (i) a first location in the canonical space and (ii) a second location in a target space included in the plurality of target spaces (Petersen et al, flow vectors are determined (mapped) between a current input low-resolution frame 604 and previous low-resolution frame and is performed at the pixel level (corresponding locations); Fig 3, 6 and ¶ [0045], [0050], [0063]-[0064]; noted Joung et al explicitly teaches matching in canonical space (as discussed above in claim 1)).
Regarding Claim 20, Petersen et al teach a system (system on chip (SoC) 100; Fig 1 and ¶ [0032]), comprising: one or more memories that store instructions (SoC 100 includes memory block 118 with instructions stored therein; Fig 1 and ¶ [0032]), and one or more processors that are coupled to the one or more memories (memory block 118 is coupled with CPU 102; Fig 1 and ¶ [0032]) and, when executing the instructions (CPU 102 executes instructions; Fig 1 and ¶ [0032], [0034], [0042]), are configured to perform the steps of: steps identical to claim 1 (as described above).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Petersen et al (US 2024/0378698) in view of Joung et al (Learning Canonical 3D Object Representation for Fine-Grained Recognition) and Vogels et al (US 2020/0184313).
Regarding Claim 6, Petersen et al in view of Joung et al teach the computer-implemented method of claim 1 (as described above), including blending the plurality of denoised intermediate samples (Petersen et al, denoised intermediate samples are generated at each time step and blended with the previous output frame 603 via processing by the diffusion model 608; Fig 3, 6 and ¶ [0045]-[0047], [0064]-[0067]).
Petersen et al in view of Joung et al does not teach wherein blending the plurality of denoised intermediate samples comprises generating a blended denoised intermediate sample included in the plurality of blended denoised intermediate samples based on an overlap of a corresponding denoised intermediate sample with one or more additional denoised intermediate samples included in the plurality of denoised intermediate samples.
Vogels et al is analogous art pertinent to the technological problem addressed in the current application and teaches wherein blending the plurality of denoised intermediate samples comprises generating a blended denoised intermediate sample included in the plurality of blended denoised intermediate samples based on an overlap of a corresponding denoised intermediate sample with one or more additional denoised intermediate samples included in the plurality of denoised intermediate samples (temporal neighborhoods are used for a sequence of frames as it is denoised with overlap in the given temporal neighborhoods of consecutive frames thereby blending the intermediate samples; Fig 5A and ¶ [0113]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Petersen et al in view of Joung et al with Vogels et al including wherein blending the plurality of denoised intermediate samples comprises generating a blended denoised intermediate sample included in the plurality of blended denoised intermediate samples based on an overlap of a corresponding denoised intermediate sample with one or more additional denoised intermediate samples included in the plurality of denoised intermediate samples. By using temporal neighborhoods with frame overlap, residual error in frames are correlated, thereby reducing perspective temporal flicker in the denoised images in the sequence, resulting in reduced temporal artifacts, as recognized by Vogels et al (¶ [0113]).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Petersen et al (US 2024/0378698) in view of Joung et al (Learning Canonical 3D Object Representation for Fine-Grained Recognition) and Chung et al (Diffusion Posterior Sampling for General Noisy Inverse Problems).
Regarding Claim 7, Petersen et al in view of Joung et al teach the computer-implemented method of claim 1 (as described above), including blending the plurality of denoised intermediate samples (Petersen et al, denoised intermediate samples are generated at each time step and blended with the previous output frame 603 via processing by the diffusion model 608; Fig 3, 6 and ¶ [0045]-[0047], [0064]-[0067]).
Petersen et al in view of Joung et al does not teach wherein the plurality of blended denoised intermediate samples is generated via a least squares optimization associated with the plurality of denoised intermediate samples.
Chung et al is analogous art pertinent to the technological problem addressed in the current application and teaches wherein the plurality of blended denoised intermediate samples is generated via a least squares optimization associated with the plurality of denoised intermediate samples (a weighted least squares method is used for solving linear inverse with Gaussian noise; Fig 4 and 4. Experiments Noisy linear inverse problems).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Petersen et al in view of Joung et al with Chung et al including wherein the plurality of blended denoised intermediate samples is generated via a least squares optimization associated with the plurality of denoised intermediate samples. By using a weighted least squares method, the denoised reconstructions most similarly mimic the ground truth, thereby improving the diffusion results, as recognized by Chung et al (4. Experiments Noisy linear inverse problems).
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Petersen et al (US 2024/0378698) in view of Joung et al (Learning Canonical 3D Object Representation for Fine-Grained Recognition) and Oz et al (US 2021/0358193).
Regarding Claim 19, Petersen et al in view of Joung et al teach the one or more non-transitory computer-readable media of claim 11 (as described above), including the output frame (the output upsampled frame 610; Fig 6 and ¶ [0066]).
Petersen et al in view of Joung et al does not teach wherein the output frame comprises at least one of a visual anagram, an anamorphic illusion, a panorama, an infinite zoom video, or a texture for a mesh.
Oz et al is analogous art pertinent to the technological problem addressed in the current application and teaches wherein the output frame comprises at least one of a visual anagram, an anamorphic illusion, a panorama, an infinite zoom video, or a texture for a mesh (random noise is reduced in the 3D model and 2D texture maps using an anisotropic diffusion neural network to generate a high resolution image output as a 3D mesh with 2D texture map; ¶ [0234], [0242]-[0244]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Petersen et al in view of Joung et al with Oz et al including wherein the output frame comprises at least one of a visual anagram, an anamorphic illusion, a panorama, an infinite zoom video, or a texture for a mesh. By outputting a texture map for a mesh the image data is generated to show dimension and texture in a 3D visual representation consistent as a function of time, thereby improving the visual representation, as recognized by Oz et al (¶ [0142]-[0145]).
Allowable Subject Matter
Claims 2-4, 12-14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The prior art was not identified to teach, suggest or provide motivation to combine in an obvious manner the following limitations (recited for claim 2 below with identical limitations in claim 12):
Claim 2.
The computer-implemented method of claim 1, wherein generating the plurality of noise samples comprises: upsampling a first plurality of noise values included in the first noise sample into a second plurality of noise values; matching, based on the plurality of flow vectors, a first plurality of locations within the canonical space to a second plurality of locations within a target space that is included in the plurality of target spaces; and aggregating a subset of the second plurality of noise values associated with the second plurality of locations into a first noise value that is (i) associated with the first plurality of locations and (ii) included in the target space.
Claims 3-4 are dependent on claim 2 and therefore objected to as dependent on claim 2.
Claim 12 recites one or more non-transitory computer-readable media of claim 11, with limitations claimed identical to claim 2 and therefore objected to for similar reasons.
Claims 13-14 are dependent on claim 12 and therefore objected to as dependent on claim 12.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ackermann et al (High-Resolution Image Editing via Multi-Stage Blended Diffusion) teach a method and system for using blended diffusion in first stages, intermediate stages and final stages to upscale the image to a super-resolution image.
Wang et al (Interpolating between Images with Diffusion Models) teach a method and system for generating interpolation intermediate samples between two real input samples that uses a phased approach to transition from one image to the other.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATHLEEN M BROUGHTON whose telephone number is (571)270-7380. The examiner can normally be reached Monday-Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATHLEEN M BROUGHTON/Primary Examiner, Art Unit 2661
Prosecution Insights