Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1-20 are pending.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the arguments are directed towards the newly amended claim limitations that change the scope of the claims as whole and are open to new grounds of rejection/interpretation.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 4, 7, 10-11, 13 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Gu et al. (“NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion”).
Re claim 1, Gu teaches obtaining a first image and an input transformation instruction, wherein the first image depicts a first view of an object having a color and a structural feature (see Fig. 1 wherein an input image is a first view of an object with color and structure of a chair).
and the input transformation instruction indicates a second view of the object different from the first view (see Fig. 1-2, wherein the input transformation instruction indicates a second view different from the first view based on input image and target view (training phase).
generating, using an image generation model, a latent representation by performing a diffusion denoising process based on the first image and the input transformation instruction, wherein the latent representation represents the second view of the object; and generating using the image generation model, a second image depicting the second view of the object based on the latent representation, wherein the second image depicts the object with the color and the structural feature consistent with the first image (see Fig. 1, wherein an input image is used to generate at least a second image depicting a second view different from the input image with color and structure consistent with the first image), (see Abstract: In this work, we propose NerfDiff, which addresses this issue by distilling the knowledge of a 3D-aware conditional diffusion model (CDM) into NeRF through synthesizing and refining a set of virtual views at test-time. We further propose a novel NeRF-guided distillation algorithm that simultaneously generates 3D consistent virtual views from the CDM samples, and finetunes the NeRF based on the improved virtual views), and (see Fig.2, wherein diffusion models and Denoising is performed on the first image input view to generate target view that is a second view of the object with color and structural features consistent with the input image).
Re claim 4, Gu teaches claim 1. Furthermore, Gu teaches generating a model of the object based on the second image (see Fig. 2, wherein a 3d model is generated based on a second image).
Re claim 7, Gu teaches a method of training a machine learning model, the method comprising:
obtaining a training set including a training image depicting a first view of an object (see Fig. 2, wherein an input image is used for a training image based on a target view).
Generating, using an image generation model, an output image depicting a second view of the object based on the training image (see Fig. 2, wherein an input image is used to generate a second view of an object image in a second target view).
generating a three-dimensional (3D) model based on the output image and the training
image (see Fig. 2, target views of the training phase as an output 3d model based on the training image (image-conditioned NerF).
and training, using the 3D model, an image generation model to generate a synthetic
image depicting a third view of the object (see Fig. 2, right side, We use the learned network parameters at test time to predict an initial NeRF representation for finetuning. The NeRF-guided denoised images from the frozen CDM then supervise the NeRF in turn (synthetic images depicting a third view of the object) and (See Figs. 1 and 4, showing based on a initial view, generated third views of the object that are synthetic).
Re claim 10, Gu teaches claim 7. Furthermore, Gu teaches generating a plurality of output images depicting a plurality of views of the object and generating the 3d models based on the plurality of output images (see Fig. 1 and 4 wherein a plurality of 3d models are generated based on output novel views).
Re claim 11, Gu teaches claim 10. Furthermore, Gu teaches generating a target image based on the 3d model and comparing the target image with a second output image other than the plurality of output images used to generate the 3d model (see Fig. 1, wherein Ours method is compared to images outputted by different methods such as VisionNeRF as a comparison).
Re claim 13, Gu teaches claim 7. Gu further teaches the 3d model comprises a NeRF model (see Abstract and Fig. 2, wherein a NeRF based model is used for 3D modeling).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 2, 5, 12, 16-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gu et al. (“NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion”) in view of Ye et al. (“DreamReward: Text-to-3D Generation with Human Preference”).
Re claim 2, Gu teaches claim 1. Gu does not explicitly teach the image generation model is trained using unsupervised learning by generating a three-dimensional (3D) model based on an output image of the image generation model, computing a reward based on the 3D model, and updating parameters of the image generation model based on the reward.
However, Ye teaches the image generation model is trained using unsupervised learning by generating a three-dimensional (3D) model based on an output image of the image generation model computing a reward based on the 3D model (see p. 1, abstract, Reward3D, text-to-3D human preference reward model to effectively encode human preferences), (see p. 2-3, Reward3D: we train the Reward3D scoring model with 3D-aware capabilities, enabling it to effectively evaluate the quality of generated 3D content), and (see p. 6, 4.2., in reference to Fig. 1-2, wherein 3d models are ranked, for each comparison if better or worse to calculate a loss function, as seen in Fig. 2, wherein 3d datasets are scored).
and updating parameters of the image generation model based on the reward (See Fig. 1, bottom, wherein DreamFL utilizes feedback from Reward3D to computer RewardLoss and incorporate it into the SDS loss for simultaneous optimization of NeRF) and (see p. 1, “Building upon this, we further explore an optimization approach to improve 3D generation results—Reward3D Feedback Learning (DreamFL), which is a direct tuning algorithm designed to optimize multi-view diffusion models using a redefined scorer. Based on Reward3D, we carefully design the LossReward and incorporate it into the SDS pipeline for 3D generation. Grounded by our mathematical derivation, we find that the LossReward effectively drives the optimization of 3D models towards higher quality and alignment. Extensive experimental results demonstrate that 3D assets generated by DreamFL not only achieve impressive visualization but also outperform other text-to-3D generation methods in terms of quantitative metrics”).
Gu and Ye teaches claim 2. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gu’s neural radiance fields (NeRF) system of 3d generation to explicitly include learning by generating 3d model based on an output image, computing a reward based on the 3d model, and updating the parameters of the image generation model based on the reward, as taught by Ye, as the references are in the analogous art of NeRF based image generation. An advantage of the modification is that it achieves the result of fine-tuning the generated 3d model to improve quality of 3d models based on an additional reward-based learning and training.
Re claim 5, Gu teaches claim 1. Gu does not explicitly obtaining a prompt and generating the first image based on the text prompt.
However, Ye teaches obtaining a prompt and generating the first image based on the text prompt (see p. 3, 2.3., Text-to-3D Generation Evaluation Metrics) and (see 1-2, wherein a text prompt is received to generate a first image).
Gu and Ye teaches claim 5. For motivation to combine the references, see claim 2.
Re claim 12, Gu teaches claim 10. Gu does not explicitly teach computing a plurality of rewards corresponding to the plurality of output images, wherein the image generation model is trained based on the plurality of rewards.
However, Ye teaches computing a plurality of rewards corresponding to the plurality of output images, wherein the image generation model is trained based on the plurality of rewards (see Fig. 2-3, wherein Reward3D trains the dataset to assign scores) and (see Fig. 1, wherein Reward3D’s data is utilized as feedback to train the system in DreamFL). Gu and Ye teaches claim 12. For motivation, see claim 2.
Re claim 16, Gu teaches an apparatus comprising at least one processor and at least one memory storing instructions executable by the at least one processor, and the apparatus further comprises an image generation model comprising parameters stored in at least one memory (see p. 1, Introduction, wherein the system is directed towards computer vision systems which implicitly provides for a computer system of a processor with memory and instructions)
Wherein the image generation model is trained to generate a synthetic image depicting a second view of an object based on an input image and an input transformation instruction, wherein the first image depicts a first view of the object having color and structural feature and the input transformation indicates a second view of the object different from the first view (see Fig. 1-2, wherein an input image is a first view and training generates a second view of an object based on the input image with the same color/structure).
Generate a latent representation by performing a diffusion denoising process based on the first image and the input transformation instruction, wherein the latent representation represents the second view of the object (see Fig. 2, Training Phase, including input view and a target view for image-conditioned neRF).
Generate a latent representation by performing a diffusion denoising process based on the first image and the input transformation instruction, wherein the latent representation represents the second view of the object (see Fig. 2, Training Phase, diffusion denoising process).
And generate a second image depicting the second view of the object based on the latent representation, wherein the second image depicts the object with the color and structural feature consistent with the first image (see Abstract and Figs. 1 and 2, wherein second images are generated that depicts the object with the same color and structural features consistent with the first image).
Gu does not explicitly teach wherein the image generation model is trained using unsupervised learning by generating a 3d model based on an output image of the image generation model and computing a reward based on the 3d model.
However, Ye teaches using unsupervised learning by generating a 3d model based on an output image of the image generation model and computing a reward based on the 3d model
(see p. 1, abstract, Reward3D, text-to-3D human preference reward model to effectively encode human preferences), (see p. 2-3, Reward3D: we train the Reward3D scoring model with 3D-aware capabilities, enabling it to effectively evaluate the quality of generated 3D content), (see p. 6, 4.2., in reference to Fig. 1-2, wherein 3d models are ranked, for each comparison if better or worse to calculate a loss function, as seen in Fig. 2, wherein 3d datasets are scored), and (see p. 1, “Building upon this, we further explore an optimization approach to improve 3D generation results—Reward3D Feedback Learning (DreamFL), which is a direct tuning algorithm designed to optimize multi-view diffusion models using a redefined scorer. Based on Reward3D, we carefully design the LossReward and incorporate it into the SDS pipeline for 3D generation. Grounded by our mathematical derivation, we find that the LossReward effectively drives the optimization of 3D models towards higher quality and alignment. Extensive experimental results demonstrate that 3D assets generated by DreamFL not only achieve impressive visualization but also outperform other text-to-3D generation methods in terms of quantitative metrics”). Gu and Ye teaches claim 16. For motivation, see claim 2.
Re claim 17, Gu and Ye teaches claim 16. Furthermore, Gu teaches a 3d modeling component to generate the 3d model (see Fig. 1-2, and 4 wherein 3d models are generated).
Re claim 18, Gu and Ye teaches claim 16. Furthermore, Gu teaches a rendering component configured to generate images based on the 3d model (see Fig. 1 and 2 wherein images are rendered into 3d model images).
Re claim 19, Gu and Ye teaches claim 16. Furthermore, Gu teaches wherein the image generation model comprises a diffusion model (see p. 1 Absstract and Fig. 2, wherein a diffusion model is used).
Re claim 20, Gu and Ye teaches claim 16. Furthermore, Ye teaches a reward component configured to compute the reward (see p. 2-3, Reward3D: we train the Reward3D scoring model with 3D-aware capabilities, enabling it to effectively evaluate the quality of generated 3D content), (see p. 6, 4.2., in reference to Fig. 1-2, wherein 3d models are ranked, for each comparison if better or worse to calculate a loss function, as seen in Fig. 2, wherein 3d datasets are scored). For motivation, see claim 2.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gu et al. (“NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion”) in view of Peng et al. (“CageNeRF: Cage-based Neural Radiance Field for Generalized 3D Deformation and Animation”).
Re claim 3, Gu teaches claim 1.
Furthermore, Gu teaches generating, using the image generation model, a third image depicting a third view of the object based on the latent representation, wherein the third image depicts the object the color, and the structural feature consistent with the first image (see abstract in view of Fig. 1-2, wherein at least a third image shows a third view of the object with color and structural features consistent, using the latent representation).
Gu does not explicitly teach combining the second image and the third image to obtain an animation of the object.
However, Peng teaches combining the second image and the third image to obtain an animation of the object (see p 2. Fig. 1, wherein Given multi-view images of an object, our method learns an implicit deformable 3D representation, which is used to deform and animate the object, such as using second and third images to combine object animation of a character, such as Hulk) and (see p. 3, 3. Methodology, wherein CageNeRF is used for editing and animating 3d objects represented by a neural radiance field).
Gu and Peng teaches claim 3. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gu’s neural radiance fields (NeRF) system of 3d generation to explicitly include animation of the neural radiance images generated, such as using a second and third image generated, as taught by Peng as the references are in the analogous art of NeRF based image generation. An advantage of the modification is that it achieves the result of using the novel images created from NeRF, such as second and third images, to animate a 3d object.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gu et al. (“NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion”) in view of Athar et al. (“FLAME-in-NeRF: Neural control of Radiance Fields for Free View Face Animation”).
Re claim 6, Gu teaches claim 1. Gu does not explicitly teach wherein obtaining the first image comprises: obtaining a preliminary image depicting the object and a background region, and masking the background region of the preliminary image to obtain the first image.
However, Athar teaches wherein obtaining the first image comprises: obtaining a preliminary image depicting the object and a background region, and masking the background region of the preliminary image to obtain the first image (see p. 4, Fig. 2, wherein a silhouette that masks the background region from the object of a person’s face is obtained).
Gu and Athar teaches claim 6. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gu’s neural radiance fields (NeRF) system of 3d generation to explicitly include masking a background region from a preliminary image, as taught by Athar as the references are in the analogous art of NeRF based image generation. An advantage of the modification is that it achieves the result of segmenting an object from the background in a preliminary image so that processing only affects a certain region of interest of an image.
Claim(s) 8-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gu et al. (“NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion”) in view of and Mann et al. (US 20230132243).
Re claim 8, Gu teaches claim 7. Gu teaches generating target images based on 3d models (see Fig. 2, wherein target images are generated based on 3d models), but does not explicitly teach wherein training the image generation model comprises: generating a target image based on the 3d model and computing a reward by comparing the target image with the output image, wherein the image generation model is trained based on the reward.
However, Mann teaches wherein training the image generation model comprises: generating the target image based on the 3d model and computing a reward by comparing the target image with the output image, wherein the image generation model is trained based on the reward (see [0069], multiple instances of the object from multiple scenes from a film) and ([0070] The machine learning model may include one or more neural networks. For example, the machine learning model may include a conditional generative adversarial network (GAN) comprising a generator network configured to generate images in dependence on the parameter values of the synthetic model, and a discriminator network configured to predict whether a given image is a genuine instance of the object or was generated by the generator network. The generator network and the discriminator network may be trained alongside each other using an adversarial loss function which rewards the discriminator network for making correct predictions and rewards the generator network for causing the discriminator to make incorrect predictions. This type of training may be referred to as adversarial training. The adversarial loss function may be supplemented with one or more further loss functions, such as a photometric loss function which penalizes differences between pixel values of the isolated instance of the object and pixel values of the image output by the generator network, and/or a perceptual loss function which compares the image output by the generator network with the isolated instance in a feature space of an image encoder (such as a VGG net trained on ImageNet). By combining an adversarial loss function with a photometric and/or perceptual loss function, the generator network may learn to generate sequence of images which are both photometrically alike to the isolated instances of the object and stylistically indistinguishable from the isolated instances of the object. In this way, the generator network may learn to generate photorealistic reconstructions of isolated instances of the object). Mann teaches generating a target image based on the 3d model and computing a reward by comparing the target image with the output image (rewarding system based on whether a given image is genuine or is considered synthetic by comparison of the given image with the object image, including perceptual losses such as differences between pixel values) wherein the image generation model is trained based on the reward (in this way, the generator network may learn to generate photorealistic reconstructions of isolated instances of the object).
Gu and Mann teaches claim 8. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gu’s neural network based learning system for image reconstruction to explicitly include training by computing a reward by comparing the target image with the output image, as taught by Mann, as the references are in the analogous art of image-based neural network learning systems. An advantage of the modification is that it achieves the result of using reward models to help training the images based on perceptual rewards, thus improving the quality of the training of the images/objects.
Re claim 9, Gu and Mann teaches claim 8. Furthermore, Mann teaches wherein the reward is based on perceptual similarity metric ([0070] The machine learning model may include one or more neural networks. For example, the machine learning model may include a conditional generative adversarial network (GAN) comprising a generator network configured to generate images in dependence on the parameter values of the synthetic model, and a discriminator network configured to predict whether a given image is a genuine instance of the object or was generated by the generator network. The generator network and the discriminator network may be trained alongside each other using an adversarial loss function which rewards the discriminator network for making correct predictions and rewards the generator network for causing the discriminator to make incorrect predictions. This type of training may be referred to as adversarial training. The adversarial loss function may be supplemented with one or more further loss functions, such as a photometric loss function which penalizes differences between pixel values of the isolated instance of the object and pixel values of the image output by the generator network, and/or a perceptual loss function which compares the image output by the generator network with the isolated instance in a feature space of an image encoder (such as a VGG net trained on ImageNet). By combining an adversarial loss function with a photometric and/or perceptual loss function, the generator network may learn to generate sequence of images which are both photometrically alike to the isolated instances of the object and stylistically indistinguishable from the isolated instances of the object. In this way, the generator network may learn to generate photorealistic reconstructions of isolated instances of the object). For motivation, see claim 8.
Claim(s) 14-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gu et al. (“NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion”) in view of Black et al. (“Training Diffusion Models with Reinforcement Learning”).
Re claim 14, Gu teaches claim 7. Gu does not explicitly teach the training comprises reinforcement learning (RL), as Gu does not explicitly state reinforcement learning.
However, Black teaches reinforcement learning (RL) (see p. 1, abstract, wherein reinforcement learning methods are used including denoising diffusion policy optimization).
Gu and Black teaches claim 14. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gu’s learning based image generation system to explicitly include reinforcement learning, as taught by Black, as the references are in the analogous art of image-based learning system using diffusion. An advantage of the modification is that it achieves the result of providing for reinforcement learning to provide by providing maximized aesthetic quality by removing background content and applying foreground smoothing to maximize compressibility (see Black, Fig. 3).
Re claim 15, Gu and Black teach claim 14. Furthermore, Black teaches wherein the RL comprises DDPO (see p. 1, abstract, wherein reinforcement learning methods are used including denoising diffusion policy optimization). For motivation, see claim 14.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Peter Hoang whose telephone number is (571)270-1346. The examiner can normally be reached Monday-Friday 8:00 am - 5:00 pm PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hajnik F. Daniel can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PETER HOANG/ Primary Examiner, Art Unit 2616
Prosecution Insights