DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
1. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
2. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
3. Claims 1, 2, and 7-20 are rejected under 35 U.S.C. 103 as being unpatentable over Gao et al. US 20240096017 A1, in view of Shi et al. US 20240355022 A1.
4. As per claim 1, Gao discloses: A computing device comprising:
one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, (Gao, Figure 6) configure the one or more processors to perform operations comprising:
rendering a three-dimensional (3D) model and trainable 3D texture to generate a first two- dimensional (2D) image; (Gao, [0078], “In at least one embodiment, a 3D shape can be generated 406 using a geometry generator branch and a texture can be generated using a texture generator branch of a generative network. In at least one embodiment, this generated shape and texture data can be used 408 to generate a textured 3D mesh as output of this generative neural network being trained. In at least one embodiment, this may correspond to a generator portion of a GAN. In at least one embodiment, a 2D image and a 2D silhouette can be generated 410 for one or more points of view corresponding to this textured 3D mesh, as may be performed using differentiable rendering.”)
adding noise to the first 2D image to generate a first 2D image with added noise; (Gao, [0116], “In at least one embodiment, this can include generating a first image of an object based, at least in part, upon adding noise to, and removing this noise from, a second image of this object.”)
updating the trainable 3D texture based on the loss. (Gao, [0058], “In at least one embodiment, these determinations can be used with a loss function to determine one or more loss values, which can be fed to a weight calculator 270 during a backpropagation pass to calculate updates network weights, which can be fed back to this selected network 262 for further training. In at least one embodiment, such an approach enables selected network 262 to be trained to generate textured 3D meshes using only 2D image data as input and using 2D discriminator determinations.”)
5. Gao doesn’t disclose:
inputting the first 2D image with added noise and input text into a trained neural network to generate a predicted noise of the first 2D image with added noise;
determining a loss between the first 2D image with added noise and the predicted noise;
6. Shi discloses:
inputting the first 2D image with added noise and input text into a trained neural network to generate a predicted noise of the first 2D image with added noise; (Shi, [0151],” In various embodiments, an image generation model can be trained by obtaining a training data set including a plurality of training images and a text description, generating a new image based on an input test image and the text description using the image generation component, comparing the predicted noise and the ground truth noise, and updating parameters of the image generating component based on the comparison.”)
determining a loss between the first 2D image with added noise and the predicted noise;( Shi, [0138], “ where z.sub.t is the latent noisy image at time step t obtained from the ground-truth image, x.sub.t.sup.1, η is the latent noise to predict, c is the textual embedding, X.sub.s is the set of conditioning images, and η.sub.θ is the noise prediction model with parameters θ. The model can be optimized with only the denoising loss of the diffusion model.”) and
7. Shi is analogous art with respect to Gao because they are from the same field of endeavor, namely image processing. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to include:” inputting the first 2D image with added noise and input text into a trained neural network to generate a predicted noise of the first 2D image with added noise; determining a loss between the first 2D image with added noise and the predicted noise; ” as taught by Shi into the teaching of Gao. The suggestion for doing so would depict aspects of the subject and the input description. Therefore, it would have been obvious to combine Shi with Gao.
8. As per claim 2, Gao in view of Shi disclose: The computing device of claim 1, wherein determining the loss further comprises:
determining the loss based on a difference between the first 2D image with added noise and the predicted noise. (Shi, [0148], “At operation 960, the noise added to the test image can be compared to a ground truth noise map for the training image to obtain a similarity value.”, and [0149], “ At operation 965, a loss value can be calculated for the comparison of the ground truth and the predicted noise.”)
9. As per claim 7, Gao in view of Shi discloses: The computing device of claim 1, wherein the trained neural network is trained, using a diffusion model, to generate 2D images based on input texts, and the trained neural network comprises one or more of:
convolutional layers, one or more up sampling layers, one or more down sampling layers, and one or more fully connected layers. (Gao, [0065], “In at least one embodiment, such an approach can first use 3D convolutional layers to generate a feature volume conditioned on w.sub.1.”)
10. As per claim 8, Gao in view of Shi discloses: The computing device of claim 1, wherein the operations further comprise: receiving the input text and the 3D model from a user. (Gao, [0078], “In at least one embodiment, a 3D shape can be generated 406 using a geometry generator branch and a texture can be generated using a texture generator branch of a generative network. In at least one embodiment, this generated shape and texture data can be used 408 to generate a textured 3D mesh as output of this generative neural network being trained. In at least one embodiment, this may correspond to a generator portion of a GAN. In at least one embodiment, a 2D image and a 2D silhouette can be generated 410 for one or more points of view corresponding to this textured 3D mesh, as may be performed using differentiable rendering. ”)
11. As per claim 9, Gao in view of Shi discloses: The computing device of claim 1, wherein rendering the 3D model and trainable 3D texture further comprises: selecting a camera angle; and rendering the 3D model and trainable 3D texture based on the camera angle to generate the first 2D image.(Gao, [0056], “In at least one embodiment, this neural network (which can take form of a modified generative adversarial network (GAN)), can use this 3D shape representation from geometry generator 106 and this texture representation from texture generator 108 to generate a textured 3D mesh 110 that represents an object with a shape and texture corresponding to these initial sampled latent codes, which were encoded from 2D images. In at least one embodiment, one or more viewpoints of one or more virtual cameras 112 can be selected from which to render 2D images of this textured 3D mesh.”)
12. As per claim 10, Gao in view of Shi discloses: The computing device of claim 9, wherein the operations further comprise: selecting one or more lighting sources, wherein the rendering is further based on the one or more lighting sources. (Gao, [0178], “In at least one embodiment, a flashing light may be identified by operating a third deployed neural network over multiple frames, informing a vehicle's path-planning software of a presence (or an absence) of flashing lights.)
13. As per claim 11, Gao in view of Shi discloses: The computing device of claim 9, wherein the inputting the first 2D image further comprises:
modifying the input text in accordance with the camera angle; (Gao, [0076], “In at least one embodiment, during optimization G.sub.t and G.sub.ƒ both render images from 16 random camera views. In at least one embodiment, given a text query, a number (e.g., 500) of pairs of noise vectors z.sub.1 and z.sub.2 can be sampled, and for each sample parameters of G.sub.t can be optimized to minimize a directional CLIP loss (source text labels may be “car”, “animal” and “house” for corresponding categories), and samples with minimal loss selected.”) and
inputting the first 2D image with added noise and the modified input text into the trained neural network to generate the predicted noise. (Shi, [0151],” In various embodiments, an image generation model can be trained by obtaining a training data set including a plurality of training images and a text description, generating a new image based on an input test image and the text description using the image generation component, comparing the predicted noise and the ground truth noise, and updating parameters of the image generating component based on the comparison.”)
14. As per claim 12, Gao in view of Shi discloses: The computing device of claim 10, wherein the inputting the first 2D image further comprises:
modifying the input text in accordance with the one or more lighting sources; (Shi, [0003], “ Embodiments of the present disclosure provide a machine learning model including a generative network that can learn a concept from a set of images, then generate new scenes or styles of the concept from an input prompt. Personalized image synthesis may generate new images of a particular subject (e.g., person, animal, object, etc.) with different poses, backgrounds, locations, positions, orientations, dressing, lighting, styles, all while keeping the same subject's identity.”)
inputting the first 2D image with added noise and the modified input text into the trained neural network to generate the predicted noise. (Shi, [0151],” In various embodiments, an image generation model can be trained by obtaining a training data set including a plurality of training images and a text description, generating a new image based on an input test image and the text description using the image generation component, comparing the predicted noise and the ground truth noise, and updating parameters of the image generating component based on the comparison.”)
15. As per claim 13, Gao in view of Shi discloses: The computing device of claim 1, wherein adding noise further comprises:
determining Gaussian noise for the first 2D image; (Shi, 0064], “ In various embodiments, noise component 350 generates a noise map based on the original image 115 and a mask, where the output image 125 is generated based on the noise map. In some examples, noise component 350 generates an iterative noise map for each of a set of output images with successively reduced noise to produce the output image.”) and adding the Gaussian noise to the first 2D image to generate the first 2D image with the added noise. (Shi, [0066],” Diffusion models are a class of generative models that convert Gaussian noise into images from a learned data distribution using an iterative denoising process.”, and [0098], ” where p(x.sub.T)=(x.sub.T; 0, I) is the pure noise distribution as the reverse process takes the outcome of the forward process, a sample of pure noise, as input and Π.sub.t=1.sup.T p.sub.θ(x.sub.t-1|x.sub.t) represents a sequence of Gaussian transitions corresponding to a sequence of addition of Gaussian noise to the sample..”)”)
16. As per claim 15, Gao in view of Shi discloses: The computing device of claim 14, wherein the inputting the first 2D image further comprises: Inputting the first 2D image with added noise, the input text, and the number of iterations into the trained neural network to generate the predicted noise of the first 2D image with added noise. (Shi, [0151], “ In various embodiments, an image generation model can be trained by obtaining a training data set including a plurality of training images and a text description, generating a new image based on an input test image and the text description using the image generation component, comparing the predicted noise and the ground truth noise, and updating parameters of the image generating component based on the comparison.”, and [0152], “ In a non-limiting exemplary embodiment, the model is trained for 320k iterations for person and 200k iterations for cat, with the learning rate 1e-6 for adapter layers and 1e-4 for the FC layers in the visual encoders, under batch size 16 deployed over 4 A100 GPUs.”)
17. As per claim 14, Gao in view of Shi discloses: The computing device of claim 1, wherein determining Gaussian noise further comprises: sampling a Gaussian distribution to determine the noise, wherein an amount of the noise is based on a number of iterations of updating the trainable 3D texture. (Gao, [0062], “ In at least one embodiment, a 3D generator M, E=G(z) can be learned to map a sample from a Gaussian distribution z∈(0, I) to a mesh M with texture E. In at least one embodiment, since identical geometry can have different textures, and a same texture can be applied to different geometries, two random input vectors Z.sub.1∈.sup.512 and z.sub.2∈.sup.512 can be sampled. In at least one embodiment, non-linear mapping networks ƒ.sub.geo and ƒ.sub.tex can be used to map Z.sub.1 and Z.sub.2 to intermediate latent vectors w.sub.1=ƒ.sub.geo(z.sub.1) and w.sub.2=ƒ.sub.tex (z.sub.2) which are further used to produce style codes that control generation of 3D shapes and texture, respectively.”)
18. As per claim 16, Gao in view of Shi discloses: The computing device of claim 1, wherein the operations further comprise:
repeating the rendering, the adding, the inputting, the determining, and the updating, until the loss transgresses a threshold value. (Shi, [0076], “ In at least one embodiment, a generative network as presented herein can also support text-guided 3D content synthesis by fine-tuning a pre-trained model, such as under guidance of CLIP. In at least one embodiment, it can be noted that a final synthesis result can be a textured 3D mesh. In at least one embodiment, a dual-generator design can be used, where a trainable copy G.sub.t and a frozen copy G.sub.ƒ of a pre-trained generator are adopted. In at least one embodiment, during optimization G.sub.t and G.sub.ƒ both render images from 16 random camera views. In at least one embodiment, given a text query, a number (e.g., 500) of pairs of noise vectors z.sub.1 and z.sub.2 can be sampled, and for each sample parameters of G.sub.t can be optimized to minimize a directional CLIP loss (source text labels may be “car”, “animal” and “house” for corresponding categories), and samples with minimal loss selected. In at least one embodiment, to accelerate this process, a small number of optimization steps can be run for these 500 samples, then a top 50 samples with lowest losses chosen, with optimization run for 300 steps.”)
19. Claims 16, and 19, which is similar in scope to claim 1, thus rejected under the same rationale.
20. Claims 18, and 20, which is similar in scope to claim 2, thus rejected under the same rationale.
21. Claims 3, 4, 15, 19, 20, and 25 are rejected under 35 U.S.C. 103 as being unpatentable over Gao et al. US 20240096017 A1, in view of Shi et al. US 20240355022 A1, and further in view of Laine et la. US 20220051481 A1.
22. As per claim 3, Gao in view of Shi disclose: The computing device of claim 2, wherein the operations further comprise:
determining a gradient based on the loss; (Gao, [0095], “ In at least one embodiment, training framework 804 trains untrained neural network 806 repeatedly while adjust weights to refine an output of untrained neural network 806 using a loss function and adjustment algorithm, such as stochastic gradient descent.”
23. Gao in view of Shi doesn’t expressly disclose:
backpropagating the gradient through the first 2D image with added noise;
backpropagating the gradient through the 3D trainable texture; and
updating the trainable 3D texture based on the gradient.
24. Laine discloses:
backpropagating the gradient through the first 2D image with added noise; (Laine, [0044], “The backwards propagation pipeline 260 receives antialiased image gradients that indicate how the color of each pixel of the antialiased image affects the loss. The backwards propagation pipeline 260 computes (1) gradient of the aliased image that is input to the last stage of the rendering pipeline 205 (analytic antialiasing), and (2) gradient of the vertex positions.”)
backpropagating the gradient through the 3D trainable texture; (Laine, [0053], “Using barycentrics and NDC depth (u, v, z.sub.c/w.sub.c) as a base coordinate system allows easy coupling of shading and interpolation, as well as combining texture gradients with geometry gradients in the backward pass through the backwards propagation pipeline 260.”) and
updating the trainable 3D texture based on the gradient. (Laine, [0068], “The attribute gradients may be used to adjust the texture coordinates for the global surface texture map of the 3D model.”
25. Laine is analogous art with respect to Gao in view of Shi because they are from the same field of endeavor, namely image processing. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to include:” backpropagating the gradient through the first 2D image with added noise; backpropagating the gradient through the 3D trainable texture; and updating the trainable 3D texture based on the gradient. ” as taught by Laine into the teaching of Gao in view of Shi. The suggestion for doing so would render high quality images. Therefore, it would have been obvious to combine Laine with Gao in view of Shi.
26. As per claim 4, Gao in view of Shi, and in view of Laine discloses: The computing device of claim 3, wherein the backpropagating the gradient through the first 2D image with added noise further comprises: subtracting the predicted noise from the first 2D image with added noise. (Shi, [0137] formula 11, and [0138], “ where z.sub.t is the latent noisy image at time step t obtained from the ground-truth image, x.sub.t.sup.1, η is the latent noise to predict, c is the textual embedding, X.sub.s is the set of conditioning images, and η.sub.θ is the noise prediction model with parameters θ. The model can be optimized with only the denoising loss of the diffusion model.”)
27. As per claim 5, Gao in view of Shi, and in view of Laine discloses: The computing device of claim 3, wherein rendering further comprises: rendering, using a differential renderer component, the 3D model and the trainable 3D texture to generate the first 2D image. (Gao, [0058], “In at least one embodiment, during training this selected network 262 can generate a textured 3D mesh that can be provided to a differential renderer 264. In at least one embodiment, this differential renderer can generate one or more 2D representations, which can be provided to a discriminator 268 to determine whether these images are likely real or fake as discussed elsewhere herein.”)
28. As per claim 6, Gao in view of Shi, and in view of Laine discloses: The computing device of claim 5, wherein the operations further comprise: propagating the gradient through the differential renderer component. (Gao, [0061], “In at least one embodiment, with this representation, shapes with arbitrary topology can be generated by predicting different signs of s.sub.i. where g(u) is defined as g(u)=−log(1+exp(−u)), p.sub.x is a distribution of real images, R denotes rendering, and x is a hyperparameter. In at least one embodiment, since R is differentiable, gradients can be backpropagated from 2D images to 3D generators.”
Response to Arguments
29. Applicant's arguments filed 02/03/2026 have been fully considered but they are not persuasive.
Applicant argues Gao (US- 2024/0096017) does not teach “updating the trainable 3D texture based on the loss.” as recited in claim 1. Applicant alleges Obata merely teaches a database which stores the image data information and registrant information.”
The Examiner’s response is supported by Gao, which discloses updating the trainable 3D texture based on the loss. See paragraph 58: “In at least one embodiment, these determinations can be used with a loss function to determine one or more loss values, which can be fed to a weight calculator 270 during a backpropagation pass to calculate updates to network weights, which can be fed back to this selected network 262 for further training. In at least one embodiment, such an approach enables selected network 262 to be trained to generate textured 3D meshes using only 2D image data as input and using 2D discriminator determinations.” This passage clearly teaches “updating the trainable 3D texture based on the loss.” Applicant also did not provide a detailed explanation of why paragraph 58 does not teach “updating the trainable 3D texture based on the loss.”
Conclusion
30. ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ABDERRAHIM MEROUAN whose telephone number is (571)270-5254. The examiner can normally be reached on Monday to Friday 7:30 AM to 5:00 PM.
The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ABDERRAHIM MEROUAN/Supervisory Patent Examiner, Art Unit 2683
Prosecution Insights