DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, regarding the newly recited claim language, filed 09/11/2014, with respect to the rejection(s) of claim(s) 1-20 under Lin et al. as modified by Huang et al. have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Lin et al. in view of Francis’s.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. PGPubs 2024/0161403 to Lin et al. in view of U.S. PGPubs 2024/0185498 to Francis.
PNG
media_image1.png
224
468
media_image1.png
Greyscale
Regarding claim 1, Lin et al. teach one or more processors comprising (par 0017): circuitry to (par 0078):
generate one or more three-dimensional (3D) models of one or more objects (par 0007, “A 3D mesh is determined for a scene model generated with a first resolution, wherein the scene model is generated from an input text prompt describing a 3D content “, par 0031-0032, “The low-resolution scene model refers to a scene model that has been generated from an input text prompt describing a 3D content ….The mesh extractor 202 processes the low-resolution scene model to determine a 3D mesh for the low-resolution scene model”) ;
use one or more neural networks to determine the one or more 3D models represents one or more text descriptions (par 0018-0019, “a 3D mesh is determined for a scene model generated with a first resolution, wherein the scene model is generated from an input text prompt describing a 3D content ….the scene model may be generated by a diffusion model (different from the diffusion model described below with respect to operation 104). In an embodiment, the diffusion model used to generate the scene model from the input text prompt may back-propagate gradients into the scene model via a loss defined on rendered images at the first resolution. In one exemplary embodiment, this diffusion model may be a pre-trained text-to-image diffusion model”, par 0031-0032, “The low-resolution scene model refers to a scene model that has been generated from an input text prompt describing a 3D content ….. the mesh extractor 202 may be a deep 3D conditional generative model that processes the low-resolution scene model to extract the 3D mesh therefrom”, par 0044-0045, generate a low-resolution scene model based on text prompts through a neural network); and
use the one or more neural networks to adjust the one or more 3D models to represent the one or more text descriptions based, at least in part, on the determination by the one or more neural networks (par 0033, “The 3D mesh is output by the mesh extractor 202 to the diffusion model 204. The diffusion model 204 processes the 3D mesh to predict a high-resolution 3D mesh model. While the 3D mesh model is referred to as a “high-resolution” 3D mesh model, it should be noted that this simply refers to a resolution that is higher than a resolution of the input to the system 200, and in particular a resolution that is higher than a resolution of the scene model input to the mesh extractor 202” …..Generate a fine 3D model using second diffusion processing based on a generated coarse 3D model using first diffusion processing based on inputted text).
But Lin et al. keep silent for teaching use one or more neural networks to determine whether the one or more 3D models represents one or more text descriptions.
In related endeavor, Francis teaches use one or more neural networks to determine whether the one or more 3D models represents one or more text descriptions (par 0023, “it can generate new images by starting with random noise and iteratively removing the noise to create a coherent image. In some embodiments, the T2I model is trained over a dataset, including examples of images and their corresponding text descriptions. In some embodiments, the T2I model is trained over a dataset of images and features that correspond to and can be extracted from a 3D scene, e.g., predicted depth maps. The T2I model learns to associate specific words and phrases with visual patterns, shapes, colors, and compositions found in the images. The T2I model is able to understand and interpret text descriptions and then use this understanding to guide the de-noising process to generate an image that matches the text description”, par 0032, “The T2I model 230 is a diffusion model trained to create images based on both an image and a text description. In some embodiments, this model takes the text input (such as a sentence or a phrase) and an image feature input (such as color, depth, normal, diffuse, metallic, edge map, or model features extracted from an image rasterized based on a 3D scene captured by a client device 110), and generates an image that visually represents the content described in the text …. When generating an image, the T2I model 230 performs a learned reverse diffusion process over the original image, conditioned on the text embedding. At each step, the T2I model 230 predicts and substracts a bit of noise, gradually refining the image until a clear image that corresponds to the text description emerges”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Lin et al. to include use one or more neural networks to determine whether the one or more 3D models represents one or more text descriptions as taught by Francis to use a diffusion model to train over a dataset comprising images and corresponding text descriptions to generate an image consistent with a text prompt.
Regarding claim 2, Lin et al. as modified by Francis teach all the limitation of claim 1, and further teach wherein the one or more neural networks comprise one or more first neural networks and one or more second neural networks (Lin et al.: par 0019, “the scene model may be generated by a diffusion model (different from the diffusion model described below with respect to operation 104). In an embodiment, the diffusion model used to generate the scene model from the input text prompt may back-propagate gradients into the scene model via a loss defined on rendered images at the first resolution”, Fig 4, par 0037-0039, “the input text prompt is processed, using a first diffusion model, to generate a scene model with a first resolution. In an embodiment, the first diffusion model may be a pre-trained text-to-image diffusion model. In an embodiment, the scene model may be an Instant-NGP representation of the 3D content ….3D mesh is extracted from the scene model. In operation 308, the 3D mesh is processed, using a second diffusion model, to predict a 3D mesh model with a second resolution that is greater than the first resolution”), and the one or more first neural networks are trained based, at least in part, on an indication generated by the one or more second neural networks of whether the one or more 3D models adjusted by the one or more neural networks 3D model matches the one or more text descriptions (Lin et al.: par 0058-0059, “the 3D mesh model is optimized (or refined) based on the modification to the input text prompt. In the example above where the modification is to a texture and/or a geometry of the 3D content, the corresponding texture and/or geometry of the 3D mesh model may be optimized accordingly. In the example above where the modification is the addition of the reference image, the 3D mesh model may be optimized to better match the object(s) in the reference image”, Francis: par 0023, “it can generate new images by starting with random noise and iteratively removing the noise to create a coherent image. In some embodiments, the T2I model is trained over a dataset, including examples of images and their corresponding text descriptions. In some embodiments, the T2I model is trained over a dataset of images and features that correspond to and can be extracted from a 3D scene, e.g., predicted depth maps. The T2I model learns to associate specific words and phrases with visual patterns, shapes, colors, and compositions found in the images. The T2I model is able to understand and interpret text descriptions and then use this understanding to guide the de-noising process to generate an image that matches the text description”, par 0032, “The T2I model 230 is a diffusion model trained to create images based on both an image and a text description. In some embodiments, this model takes the text input (such as a sentence or a phrase) and an image feature input (such as color, depth, normal, diffuse, metallic, edge map, or model features extracted from an image rasterized based on a 3D scene captured by a client device 110), and generates an image that visually represents the content described in the text …. When generating an image, the T2I model 230 performs a learned reverse diffusion process over the original image, conditioned on the text embedding. At each step, the T2I model 230 predicts and substracts a bit of noise, gradually refining the image until a clear image that corresponds to the text description emerges”).
Regarding claim 3, Lin et al. as modified by Francis teach all the limitation of claim 1, and Lin et al. further teach wherein the one or more neural networks comprise a diffusion model (par 0019, “the scene model may be generated by a diffusion model (different from the diffusion model described below with respect to operation 104). In an embodiment, the diffusion model used to generate the scene model from the input text prompt may back-propagate gradients into the scene model via a loss defined on rendered images at the first resolution”, Fig 4, par 0037-0039, “3D mesh is extracted from the scene model. In operation 308, the 3D mesh is processed, using a second diffusion model, to predict a 3D mesh model with a second resolution that is greater than the first resolution”).
Regarding claim 4, Lin et al. as modified by Francis teach all the limitation of claim 1, and Lin et al. further teach wherein the one or more neural networks comprise a convolutional neural network (par 0021, “ the Instant-NGP may use a hash grid encoding, and includes a first single-layer neural network that predicts albedo and density and a second single-layer neural network that predicts surface normal”, par 0066-0070, “Deep neural networks (DNNs), including deep learning models, developed on processors have been used for diverse use cases, from self-driving cars to faster drug development, from automatic image captioning in online image databases to smart real-time language translation in video chat applications.” par 0080-0081, “FIG. 7 illustrates another embodiment for training and deployment of a deep neural network. In at least one embodiment, untrained neural network 706 is trained using a training dataset 702.”).
Regarding claim 5, Lin et al. as modified by Francis teach all the limitation of claim 1, and Lin et al. further teach wherein the one or more neural networks are trained to identify features of the one or more 3D models that are adjustable such that the one or more 3D model adjusted by the one or mode neural networks matches the one or more text descriptions (Fig 4, par 0050, “an MLP is used to predict the normals. Note that this does not violate geometric properties since volume rendering is used instead of surface rendering; as such, the orientation of particles at continuous positions need not be oriented to the level set surface. This helps to significantly reduce the computational cost of optimizing the coarse model. Accurate normals can be obtained in the fine stage of optimization when using a true surface rendering mode”, par 0058-0059, “The modification may received as a new input text prompt that changes one or more parameters of the prior input text prompt, in an embodiment. For example, the modification may be to a texture and/or a geometry of the 3D content. As another example, the modification may be adding a reference image to the input text prompt …. where the modification is to a texture and/or a geometry of the 3D content, the corresponding texture and/or geometry of the 3D mesh model may be optimized accordingly. In the example above where the modification is the addition of the reference image, the 3D mesh model may be optimized to better match the object(s) in the reference image”).
Regarding claim 6, Lin et al. as modified by Francis teach all the limitation of claim 1, and Lin et al. further teach wherein the one or more neural networks are to adjust one or more textures of the one or more 3D models (Fig 4, par 0050, “an MLP is used to predict the normals. Note that this does not violate geometric properties since volume rendering is used instead of surface rendering; as such, the orientation of particles at continuous positions need not be oriented to the level set surface. This helps to significantly reduce the computational cost of optimizing the coarse model. Accurate normals can be obtained in the fine stage of optimization when using a true surface rendering mode”, par 0058-0059, “The modification may received as a new input text prompt that changes one or more parameters of the prior input text prompt, in an embodiment. For example, the modification may be to a texture and/or a geometry of the 3D content. As another example, the modification may be adding a reference image to the input text prompt …. where the modification is to a texture and/or a geometry of the 3D content, the corresponding texture and/or geometry of the 3D mesh model may be optimized accordingly. In the example above where the modification is the addition of the reference image, the 3D mesh model may be optimized to better match the object(s) in the reference image”).
Regarding claim 7, Lin et al. as modified by Francis teach all the limitation of claim 1, and Lin et al. further teach wherein the one or more first neural networks are to adjust one or more meshes of the one or more 3D models (par 0027, “the method 100 may also include presenting the 3D content on a display device, using the 3D mesh model. For example, the 3D mesh model may be rendered (from a defined camera perspective) to an image that is presented on the display device. In an embodiment, a modification to the input text prompt may be received (from the user), and in turn the 3D mesh model may be optimized, or refined, based on the modification to the input text prompt. For example, the modification may be to a texture and/or a geometry, such that the corresponding texture and/or geometry of the 3D mesh model may be optimized accordingly”, Fig 4, par 0050, “an MLP is used to predict the normals. Note that this does not violate geometric properties since volume rendering is used instead of surface rendering; as such, the orientation of particles at continuous positions need not be oriented to the level set surface. This helps to significantly reduce the computational cost of optimizing the coarse model. Accurate normals can be obtained in the fine stage of optimization when using a true surface rendering mode”, par 0058-0059, “The modification may received as a new input text prompt that changes one or more parameters of the prior input text prompt, in an embodiment. For example, the modification may be to a texture and/or a geometry of the 3D content. As another example, the modification may be adding a reference image to the input text prompt …. where the modification is to a texture and/or a geometry of the 3D content, the corresponding texture and/or geometry of the 3D mesh model may be optimized accordingly. In the example above where the modification is the addition of the reference image, the 3D mesh model may be optimized to better match the object(s) in the reference image”).
Regarding claims 8-14, the method claims 8-14 are similar in scope to claims 1-7 and are rejected under the same rational.
Regarding claims 15-20, the system claims 15-20 are similar in scope to claims 1-4 and 6-7 and are rejected under the same rational (par 0007).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jin Ge whose telephone number is (571)272-5556. The examiner can normally be reached 8:00 to 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at (571)272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JIN . GE
Examiner
Art Unit 2619
/JIN GE/Primary Examiner, Art Unit 2619
