Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This is in response to applicant’s amendments/remarks filed on March 30th, 2026, which have been entered and made of record. Claims 1-20 remain pending. Claims 1, 2, 3, 13, 15 and 19 have been amended. Applicant’s amendments to the drawings, specifications and claims have overcome each and every objection previously set forth in the Non-Final Office Action mailed December 29th, 2025 and therefore, all have been withdrawn.
Response to Arguments
The following is responses to the applicant’s arguments:
Regarding argument A. Office Personnel Burden, see pages 20-22, filed on March 30th, 2026, the examiner agrees that he treated independent claim 10 too broadly and failed to articulate his findings of fact and the grounds of rejection for the claim and has therefore withdrawn the previous rejection. However, after a reevaluation of the claim, the examiner has now articulated his findings and has made a new ground of rejection based on Dave in view of Matthews (see claim 10 below).
Regarding argument B. Pending Original Independent Claim 10, see pages 22-24, applicant's arguments have been fully considered but they are not persuasive. In response to applicant's argument that Dave does not disclose providing a noise vector corresponding to an initial starting point for a diffusion process or generating material maps via the diffusion process, a recitation of the intended use of the claimed invention must result in a structural difference between the claimed invention and the prior art in order to patentably distinguish the claimed invention from the prior art. If the prior art structure is capable of performing the intended use, then it meets the claim. In this case, when viewed in combination with Matthews, Matthews does teach providing a noise vector corresponding to an initial starting point for a diffusion process (Paragraph 65 teaches that the process begins with a noisy input 302, which can be an image that contains a certain amount of noise or, in other cases, may be solely noise (e.g., Gaussian noise). This noisy input 302 serves as a starting point for the image editing process. Additionally, paragraph 39 teaches that the denoising diffusion model can process a noisy input that includes the textual prompt and/or scalar edit value, while being conditioned on the context image and paragraph 58 teaches that the text prompt 34 can be encoded into a format suitable for the model, such as a vector representation obtained from a language model or word embedding model.) and Dave discloses generating material maps via the diffusion process (Col. 12, Lines 32-35 teaches that the set of material maps 310 generated by neural network system 308 are optimized using differentiable renderer 312, which may be an embodiment of material maps optimizer 230 in FIG. 2.). Therefore, when Dave is viewed in combination with Matthews, it appears that Matthews can provide noisy inputs (Gaussian noise may be considered a noisy vector) and images consisting of vector representations for a denoising/diffusion process and Dave can also provide images for a neural network system/machine learning model, such as a denoising/diffusion process, in order to generate material maps, so together they can perform the intended use and function of the claim language and be able to perform the functions of providing a noise vector corresponding to an initial starting point for a diffusion process or generating material maps via the diffusion process.
Regarding argument C. Amended Independent claim 1, see pages 24-25, applicant's arguments have been fully considered but they are not persuasive. For complete reasoning and mapping of the newly amended claim language, see the claim 1 rejection below. In response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In this case, both Matthews and Dave are of analogous art and related to one another via similar concepts related to using machine learning models to help improve training the generation of images by incorporating different features such as material maps, lighting maps and noisy inputs and/or improving the training of generating different types of maps by using different inputs such as different images, thus the concepts for training and improving the generation of images and/or maps are analogous with one another and it would therefore have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined their teachings and functions together.
Regarding argument D. Amended Independent claim 19, see pages 25-26, applicant's arguments have been fully considered but they are not persuasive due to the similar nature and response to the reasoning for independent claim 1 (see previous response to argument for claim 1 above).
Regarding argument E. Dependent claims 2-9,11-18, 20, see pages 26-29, for the virtue of their dependency are moot because the independent claims are not allowable.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-9 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Matthews et al. (Pub. No.: US 2025/0166136 A1), hereinafter Matthews, in view of Dave et al. (U.S. Patent: #11,663,775 B2), hereinafter Dave.
Regarding claim 1, Matthews discloses one or more processors (FIGS. 4A-4C and paragraph 23 teach that the present disclosure provides systems and methods for controlling material attributes in real images. Additionally, paragraph 88 teaches that the user computing device 102 includes one or more processors 112 and a memory 114.) comprising:
one or more processing units (Paragraph 88 teaches that the one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected.) to:
receive at least one of, one or more material or one or more lighting maps, the one or more materials defining one or more properties of a surface of one or more objects in a scene, the one or more lighting maps representing at least one of one or more shading or lighting characteristics associated with the one or more objects (FIG. 1 and paragraph 45 teach that FIG. 1 provides a conceptual illustration of an example process for generating training data, according to an embodiment of the present disclosure. The process begins with a set of objects 12, environments 14, and materials 16. These assets may be stored in a database or other suitable storage medium. The objects 12 represent three-dimensional (3D) models of various objects such as, for example, a chair, a table, or a car. The environments 14 represent different lighting and environmental conditions, and the materials 16 represent different types of materials that have different material properties (e.g., metallic, albedo, roughness, transparency).). However, Matthews fails to disclose that the materials are maps.
Dave discloses that the materials are maps (Col. 3, Lines 20-25 teach that the “Material maps” (e.g., diffuse albedo map, specular albedo map, normal map, roughness map) as described herein are a set of images that encode the reflectance properties of a material (e.g., its color, its geometry, its roughness). Material maps are typically the input to a physically-based rendering or other visual rendering and Col. 9 Line 66 through Col. 10 Line 6 teach that some embodiments of material maps generator 220 input lighting information into the neural network system with the input images and approximate maps. This lighting information corresponds to the input images as it indicates lighting pattern used for each input image. In some embodiments, this lighting information is in the form of an irradiance map that simulates a particular lighting pattern for white paper (rather than the particular real-world material.). Since Matthews teaches a processing device with the capabilities to receive inputs that take into account different types of materials and environment (lighting effects) maps associated with different types of objects and Dave teaches a processing device with the capabilities to receive inputs that are related to different types of maps, such as material maps and lighting maps, it would have been obvious to a person having ordinary skill in the art to combine the teachings together so that the different materials and environments (lighting effects) being received as initial input, could also utilize and receive associated material maps and lighting maps, as well.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Matthews to incorporate the teachings of Dave, so that the combined features together would allow for the input of specific (and/or newly generated) material/lighting maps, which would help improve the overall rendering process by allowing for more realistic and lifelike looking visuals to be able to be produced.
Furthermore, Matthews in view of Dave disclose and in response to the receiving of at least one of one or more material maps or one or more lighting maps (Col. 11, Lines 63-65 of Dave teaches that the material map approximations 306 as well as the input images 302 are then input into neural network system 308 and Col. 10, Lines 24-32 of Dave teaches that the image renderer 232 is configured to render images of the digital material from material maps output from material maps generator 220. Image renderer 232 renders the images by adding lighting to the material maps. The lighting of the rendered images is intended to simulate the lighting used in the input images capturing the real-world material. As such, image renderer 232 may receive lighting and camera information corresponding to the lighting and camera configurations used to capture the images. Additionally, paragraph 45 of Matthews teaches that the environments 14 represent different lighting and environmental conditions, and the materials 16 represent different types of materials that have different material properties (e.g., metallic, albedo, roughness, transparency) and paragraph 49 of Matthews teaches that the renderer 24 is configured to generate rendered images 26. The renderer 24 uses the objects 12, environments 14, camera sampler 18, and shader 22 to generate the rendered images. The renderer 24 applies the selected environment to the scene, places the object in the scene according to the camera parameters generated by the camera sampler 18, and then applies the shader 22 to the object. The resulting rendered images 26 depict the objects with different material properties in different environments and from different camera viewpoints.), generate, via one or more first machine learning models, an output frame of the scene by denoising a first noise vector using a representation of at least one of the one or more material maps or the one or more lighting maps to guide the denoising, the first noise vector corresponding to an initial starting point for the denoising (Paragraph 54 of Matthews teaches that in particular, each pair of training images obtained from the set of rendered images 26 can include a target image 28 and a context image 30. Both images depict the same object but the material property of the object differs between the target image and the context image. This difference can be the result of applying different attribute values for a particular material property in each image and paragraph 56 of Matthews teaches that the target image 28 and context image 30 are then provided as input to a denoising diffusion model 32. The diffusion model 32 is a machine learning model specifically designed for image generation and editing tasks. It can take as input a noisy image and apply a sequence of denoising operations to progressively generate a clean, final image. In the illustrated figure, the denoising diffusion model 32 is conditioned on the context image 30, which provides the model with information about the scene, the object, and its initial material properties. Lastly, FIG. 3 and paragraph 65 of Matthews teaches that FIG. 3 illustrates an example process for using a trained diffusion model to perform material edits according to example embodiments of the present disclosure. The process begins with a noisy input 302, which can be an image that contains a certain amount of noise or, in other cases, may be solely noise (e.g., Gaussian noise). This noisy input 302 serves as a starting point for the image editing process. The noisy input 302 is fed into the diffusion model 310, which is designed to remove the noise and generate a clean image.).
Regarding claim 2, Matthews in view of Dave disclose everything claimed as applied above (see claim 1), in addition, Matthews in view of Dave disclose wherein the denoising the first noise vector using the representation of at least one or more material maps or the one or more lighting maps to guide the denoising is based on performing cross-attention between a representation derived from the first noise vector and the representation of at least one of the one or more material maps or the one or more lighting maps during iterative steps of the denoising. (Paragraph 56 of Matthews teaches that the target image 28 and context image 30 are then provided as input to a denoising diffusion model 32. The diffusion model 32 is a machine learning model specifically designed for image generation and editing tasks. It can take as input a noisy image and apply a sequence of denoising operations to progressively generate a clean, final image. In the illustrated figure, the denoising diffusion model 32 is conditioned on the context image 30, which provides the model with information about the scene, the object, and its initial material properties and paragraph 61 of Matthews teaches that the model's performance is evaluated by computing a loss function that compares the denoising prediction to the set of noise added to the target image. The loss function can be any suitable function that measures the difference or discrepancy between two images, such as the mean squared error or cross-entropy loss. A lower loss value indicates that the model's prediction is closer to the target image, meaning that the model has done a better job of denoising the image. Lastly, paragraph 98 of Matthews teaches that the training computing system 150 can include a model trainer 160 that trains the machine-learned models 120 and/or 140 stored at the user computing device 102 and/or the server computing system 130 using various training or learning techniques, such as, for example, backwards propagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and/or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations.).
Regarding claim 3, Matthews in view of Dave disclose everything claimed as applied above (see claim 1), in addition, Matthews in view of Dave disclose wherein one or more processing units are further to:
receive user input requesting a lighting condition to be incorporated into the output frame (Col. 16 Lines 13-15 of Dave teaches that input images may be received based on input from a user device. For example, a user may select the input images individually or as a set from a data store. Additionally, paragraph 58 of Matthews teaches that in some implementations, in addition to the context image 30, the diffusion model 32 is also conditioned on a text prompt 34. The text prompt 34 can provide high-level instructions or guidelines about the desired edits to the material properties of the object in the image. For example, the prompt can be something like “make the object more metallic” or “increase the roughness of the object by 0.2 units”. The text prompt 34 can be encoded into a format suitable for the model, such as a vector representation obtained from a language model or word embedding model and paragraph 66 of Matthews teaches that the process also involves a context image 304, which is used to condition the diffusion model 310. The context image 304 provides a reference for the diffusion model 310, guiding the model in its task of denoising the noisy input 302. The context image 304 can be the same as the noisy input 302 or it can be a different image that depicts the same or a similar object. The context image 304 can include additional information about the object, such as its shape, size, orientation, lighting conditions, and initial material properties.);
and based at least in part on the user input, generate the one or more lighting maps by visually incorporating the lighting condition into the one or more lighting maps before the one or more first machine learning models process the one or more lighting maps as input (Col. 14, Lines 37-47 of Dave teaches that some embodiments of training data 442 further include lighting information, which may be in the form of an irradiance map that simulates a particular lighting pattern for white paper (rather than the particular real-world material). Training irradiance maps may be generated as described with respect to digital material generator 200 of FIG. 2. Training component 420 uses training input images, training material map approximations, ground truth data, and, in some embodiments, training irradiance maps to train neural network system 444 to predict material maps.), and wherein the output frame is generated based at least in part on the user input (Paragraph 63 of Matthews teaches that at the end of the process, an edited image 36 is generated based on the denoising prediction. The edited image 36 depicts the same object as in the context image 30 but with the material property of the object modified according to the instructions provided in the text prompt 34. The edited image 36 can be viewed as the output of the diffusion model 32, representing the result of the desired edit to the material property of the object in the image.).
Regarding claim 4, Matthews in view of Dave disclose everything claimed as applied above (see claim 1), in addition, Matthews in view of Dave disclose wherein one or more processing units are further to:
receive an input frame and a second noise vector (Paragraph 50 of Matthews teaches that each rendered image 26 can then be labeled with the corresponding attribute value(s) for each material property. This provides a ground truth for the training of the diffusion model. The labeled rendered images 26 can then be used as input to the denoising diffusion model, which is trained to predict the material properties of an object in an image.);
and provide the second noise vector and a representation of the input frame and as input into one or more second machine learning models to generate the one or more first material maps (Col. 6, Lines 43-57 of Dave teach that the digital material generator 106 generates material maps for a real-world material based on input images capturing the real-world material under a variety of lighting patterns. These material maps may be utilized to generate a physically-based material rendering by applying the maps to one or more digital objects. At a high level, digital material generator 106 utilizes input images, which may be in the form of sets of specular and diffuse component images, to approximate one or more material maps, such as a diffuse albedo material map approximation and a normal material map approximation. These approximations may be generated utilizing a photometric stereo utilizing a pseudo-inverse function. Digital material generator 106 may then use a neural network to generate a set of material maps based on the approximations and input images.).
Regarding claim 5, Matthews in view of Dave disclose everything claimed as applied above (see claim 1), in addition, Matthews in view of Dave disclose wherein one or more processing units are further to:
provide an input frame as input into the one or more first machine learning models, and wherein the first noise vector represents a noisy version of the input frame (Paragraph 55 of Matthews teaches that in some implementations, the context image 30 can depict the object having a base material property (e.g., with an attribute value set to zero). In contrast, the target image 28 can depict the object having an altered or edited material property (e.g., with the attribute value set to a non-zero value) and paragraph 56 of Matthews teaches that the target image 28 and context image 30 are then provided as input to a denoising diffusion model 32. The diffusion model 32 is a machine learning model specifically designed for image generation and editing tasks. It can take as input a noisy image and apply a sequence of denoising operations to progressively generate a clean, final image. In the illustrated figure, the denoising diffusion model 32 is conditioned on the context image 30, which provides the model with information about the scene, the object, and its initial material properties.);
and receive a request to enhance the input frame, wherein the output frame is generated based at least in part on the request and the one or more processing units are further to provide the input frame as input into the one or more first machine learning models (Col. 17, Lines 54-59 of Dave teach that at block 608, an updated set of material maps is generated. This updated set is optimized based on a comparison of the input images received at block 602 and images rendered from the initial set of material maps generated at block 606. Block 608 may be performed by an embodiment of material maps optimizer 230 in FIG. 2.),
and wherein the output frame includes one or more features that have been enhanced relative to the input frame (Col. 10, Lines 57-67 of Dave teach that the material maps optimizer 230 may utilize a differentiable renderer to render the images and update the maps. For example, image renderer 232 may create the rendered images in a forward pass through the differentiable renderer and updated maps generator 234 may create updated, optimized material maps by passing the rendered images back through the differentiable renderer. The material maps and rendered images may be passed through the differentiable renderer back and forth multiple times, each times reducing the differences between the rendered images and the input images. Additionally, paragraph 63 of Matthews teaches that at the end of the process, an edited image 36 is generated based on the denoising prediction. The edited image 36 depicts the same object as in the context image 30 but with the material property of the object modified according to the instructions provided in the text prompt 34. The edited image 36 can be viewed as the output of the diffusion model 32, representing the result of the desired edit to the material property of the object in the image.).
Regarding claim 6, Matthews in view of Dave disclose everything claimed as applied above (see claim 1), in addition, Matthews in view of Dave disclose wherein one or more processing units are further to:
provide a two-dimensional input frame and a second noise vector as input into one or more second machine learning models (Paragraph 57 of Matthews teaches that in particular, noise can be added to the target image 28 before it is processed by the denoising diffusion model 32. The noised target image is then processed by the diffusion model 32 to generate a denoising prediction. Additionally, FIG. 4B and paragraph 105 of Matthews teach that FIG. 4B depicts a block diagram of an example computing device 10 that performs according to example embodiments of the present disclosure. The computing device 10 can be a user computing device or a server computing device. Furthermore, paragraph 106 of Matthews teaches that the computing device 10 includes a number of applications (e.g., applications 1 through N). Each application contains its own machine learning library and machine-learned model(s). For example, each application can include a machine-learned model.);
based at least on providing the two-dimensional input frame and the second noise vector as input into the one or more second machine learning models, generate the one or more material maps (Col. 16, Lines 59-67 of Dave teach that the material maps generated at block 506 may represent the digital material corresponding to the real-world material in the input images. The material maps may be utilized to create a visual rendering of an object with the digital material. Therefore, some embodiments of method 500 include applying the generated set of material maps to a digital object so that the digital object has the appearance of the real-world material when rendered and providing the rendered object for display on a graphic user interface.),
wherein the output frame represents the two-dimensional input frame that includes a lighting property which has been modified in the output frame relative to the input frame (Col. 16, Lines 51-58 of Dave teach that in some embodiments, lighting information for each lighting pattern represented in the input images may be input into neural network to generate the material maps. This lighting information may be in the form of an irradiance map that describes a particular lighting pattern. Each set of diffuse component image and specular component image may be input with an irradiance map representing the lighting corresponding to the set. Additionally, Col. 10, Lines 24-32 teaches that the image renderer 232 is configured to render images of the digital material from material maps output from material maps generator 220. Image renderer 232 renders the images by adding lighting to the material maps. The lighting of the rendered images is intended to simulate the lighting used in the input images capturing the real-world material. As such, image renderer 232 may receive lighting and camera information corresponding to the lighting and camera configurations used to capture the images. Lastly, paragraph 62 of Matthews teaches that based on the computed loss, one or more values of one or more parameters of the diffusion model 32 are modified. This modification can be performed using any suitable optimization algorithm, such as stochastic gradient descent or Adam optimization.).
Regarding claim 7, Matthews in view of Dave disclose everything claimed as applied above (see claim 1), in addition, Matthews in view of Dave disclose wherein one or more processing units are further to:
provide a two-dimensional input frame and a second noise vector as input to one or more second machine learning models (Paragraph 57 of Matthews teaches that in particular, noise can be added to the target image 28 before it is processed by the denoising diffusion model 32. The noised target image is then processed by the diffusion model 32 to generate a denoising prediction. Additionally, FIG. 4B and paragraph 105 of Matthews teach that FIG. 4B depicts a block diagram of an example computing device 10 that performs according to example embodiments of the present disclosure. The computing device 10 can be a user computing device or a server computing device. Furthermore, paragraph 106 of Matthews teaches that the computing device 10 includes a number of applications (e.g., applications 1 through N). Each application contains its own machine learning library and machine-learned model(s). For example, each application can include a machine-learned model.);
and based at least in part on the input frame and the second noise vector being provided as input into the one or more second machine learning models, generate one or more second material maps (Col. 10, Lines 46-56 of Dave teaches that after image renderer 232 generates the rendered images, updated maps generator 234 creates new material maps based on a comparison of the rendered images and the input images capturing the real-world material. This comparison indicates errors that may have resulted from any inaccuracies in the material maps initially generated by the material maps generator 220. Where there is a difference between the rendered images from the image renderer 232 and the input images, the updated maps generator 234 modifies the material maps based on the identified differences in order to reduce the differences.).
Regarding claim 8, Matthews in view of Dave disclose everything claimed as applied above (see claim 7), in addition, Matthews in view of Dave disclose wherein one or more processing units are further to:
generate a multidimensional frame based on the one or more second material maps, the multidimensional frame representing the two-dimensional input frame that includes at least one more dimension relative to the two-dimensional input frame (Col. 3, Lines 9-20 of Dave teaches that an “image” as described herein is a visual representation of one or more portions of the real world or a visual representation of one or more documents. For example, an image can be a digital photograph, a digital image among a sequence of video segments, a graphic image file (e.g., JPEG, PNG, etc.), a picture (or sub-element of a picture), and/or a bitmap among other things. A “visual rendering” as described herein refers to another image (e.g., in 2D or 3D) (e.g., an “output image”), a physically-based rendering (PBR) material, a SVBRDF PBR material, an animation, and/or other suitable media, content, or computer object. Additionally, Col. 17, Line 60 through Col. 18, Line 6 of Dave teach that the embodiments of block 608 may include first rendering a set of rendered images depicting the real-world material. These rendered images may be created from the initial set of material maps. The rendered images simulate lighting utilized for input images. As such, lighting information indicating lighting patterns used for the input images may be utilized to generate the rendered images. The rendered images are compared to the input images and, where there is a difference between the rendered images and the input images, an updated set of material maps is created to reduce or eliminate this difference. In other words, images rendered from the updated set of material maps would be closer to the input images than the images rendered from the initial set of material maps.);
and generate the one or more first material maps based at least in part generating the multidimensional frame, and wherein the output frame is generated based at least in part on the multidimensional frame (Col. 18, Lines 19-33 of Dave teaches that block 608 may be performed utilizing a differentiable renderer. The material maps generated at block 606 may be passed through the differentiable renderer to create the rendered images, and the rendered images may be passed back through the differentiable renderer with the input images to identify differences and generate updated material maps. This process of rendering images, comparing the rendered images to the input images, and updating the material maps may be repeated until a threshold maximum difference, which may be zero, has been achieved. Alternatively, or additionally, this process may be repeated for a threshold number of times. The updated material maps generated at block 608 may be stored and/or applied to a digital object in the same manner described with respect to method 500.).
Regarding claim 9, Matthews in view of Dave disclose everything claimed as applied above (see claim 1), in addition, Matthews in view of Dave disclose wherein one or more processers is comprised in at least one of:
a system for performing deep learning operations (Col. 9, Lines 9-17 of Dave teaches that the material maps generator 220 is generally configured to utilize one or more machine learning models to generate a set of material maps for the real-world material captured in the sets of images. The machine learning model(s) used by material maps generator 220 (or any machine learning model described herein) may be or include any suitable type of model, such as a classifier, clustering model, regression model, or any deep learning (e.g., multiple-layered neural network) model.);
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content (Col. 5, Lines 31-35 of Dave teaches that in addition, or instead, application 110 may comprise a dedicated application, such as an application having image processing functionalities, including but not limited to functionalities for 3D design, game development, augmented reality, and/or virtual reality.);
a system for generating synthetic data (Paragraph 24 of Matthews teaches that the technology described in the present disclosure overcomes two main challenges associated with manipulating material properties in pixel space using a pretrained text-to-image model. The first challenge is the scarcity of real-world datasets with precisely labeled material properties. To address this, example implementations of the present disclosure can generate and/or use a synthetic dataset featuring physically-based materials and environment maps. This provides the necessary fine-grained annotations of material properties.);
or a system implemented at least partially using cloud computing resources (Col. 6, Lines 63-67 of Dave teaches that for cloud-based implementations, the instructions on server 108 may implement one or more components of digital material generator 106, and application 110 may be utilized by a user to interface with the functionality implemented on server(s) 108.).
Regarding claim 19, the method steps correspond to and are rejected similarly to the processor steps of claim 1 (see claim 1 above).
Regarding claim 20, the method steps correspond to and are rejected similarly to the processor steps of claim 9 (see claim 9 above).
Claims 10-18 are rejected under 35 U.S.C. 103 as being unpatentable over Dave in view of Matthews.
Regarding claim 10, Dave discloses a system (FIG. 1 and Col. 4, Lines 37-40 teach that each of the components shown in FIG. 1 may be implemented via any type of computing device, such as one or more of computing device 700 described in connection to FIG. 7, for example.) comprising one or more processing units (Col. 18, Lines 47-49 teach that computing device 700 includes bus 710 that directly or indirectly couples the following devices: memory 712, one or more processors 714) to:
receive an input frame (Col. 15, Lines 47-48 teach that at block 502, input images of a real-world material are received.);
and provide a representation of the input frame as input into one or more first machine learning models to generate one or more first material maps (Col. 6, Lines 43-53 teach that the digital material generator 106 generates material maps for a real-world material based on input images capturing the real-world material under a variety of lighting patterns. These material maps may be utilized to generate a physically-based material rendering by applying the maps to one or more digital objects. At a high level, digital material generator 106 utilizes input images, which may be in the form of sets of specular and diffuse component images, to approximate one or more material maps, such as a diffuse albedo material map approximation and a normal material map approximation.). However, Dave fails to disclose a first noise vector.
Matthews discloses a first noise vector (Paragraph 31 teaches that the computing system then adds a set of noise to the target image to obtain a noised target image. The purpose of adding the noise is to provide a challenge for the denoising diffusion model, which must learn to remove the noise while preserving the underlying image.). Since Dave teaches a system for providing images for a neural network system/machine learning model, such as a denoising/diffusion process, in order to generate material maps and Matthews teaches a system for providing noise and/or noisy images for a denoising/diffusion process, it would have been obvious to a person having ordinary skill in the art to combine the teachings together so that a noisy input and/or image could be used to help improve the training accuracy of a machine learning model.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Dave to incorporate the teachings of Matthews, so that the combined features together would allow for noise to be able to be added to challenge the machine learning model and over time, improve the overall accuracy and recognition of being able to remove and clean up any unwanted noise in relation to a requested related output image/frame.
Furthermore, Dave in view of Matthews disclose that the first noise vector corresponding to an initial starting point for a diffusion process performed by the one or more first machine learning models (FIG. 3 and paragraph 65 of Matthews teaches that FIG. 3 illustrates an example process for using a trained diffusion model to perform material edits according to example embodiments of the present disclosure. The process begins with a noisy input 302, which can be an image that contains a certain amount of noise or, in other cases, may be solely noise (e.g., Gaussian noise). This noisy input 302 serves as a starting point for the image editing process. The noisy input 302 is fed into the diffusion model 310, which is designed to remove the noise and generate a clean image.), the one or more first material maps defining one or more properties of a surface of one or more objects in a scene (Col. 8, Lines 4-20 Material map approximation component 210 is generally configured to approximate one or more material maps from input images depicting a real-world material. Example aspects of material map approximation component 210 generates a diffuse albedo material map approximation and a normal material map approximation. The diffuse albedo approximation is a material map with approximate albedo values representing the solid colors, without shadowing, shading, and/or highlights (i.e., without lighting effects), at each pixel. The normal approximation is a material map with approximate normal values representing approximate surface geometry of the object depicted in the captured images. In some embodiments, the normal approximation is saved in a Red-Green-Blue (RGB) format where each pixel value represents a 3D vector indicating the direction in which the surface normal is pointing.).
Regarding claim 11, Dave in view of Matthews disclose everything claims as applied above (see claim 10), in addition, Dave in view of Matthews disclose wherein the one or more first material maps include at least one of: an albedo map, a normal map, a roughness map, a metallic map, or an ambient occlusion map, a displacement map, a specular map, an emissive map, an opacity map, a cavity map, or a subsurface scattering map (Col. 3, Lines 55-62 of Dave teaches that based on the input images, some embodiments approximate one or more material maps. For example, a diffuse albedo material map approximation and a normal material map approximation may be generated from the input images using a photometric stereo technique, such as one utilizing a pseudo-inverse function. These approximations may be input into a neural network, along with the input images, to generate a set of material maps.).
Regarding claim 12, Dave in view of Matthews disclose everything claims as applied above (see claim 10), in addition, Dave in view of Matthews disclose wherein the one or more processing units are further to:
receive user input requesting at least one of a material property or a lighting condition to be incorporated into an output frame (Col. 16 Lines 13-15 teaches that input images may be received based on input from a user device. For example, a user may select the input images individually or as a set from a data store and Col. 9, Line 66 through Col. 10, Line 6 teach that some embodiments of material maps generator 220 input lighting information into the neural network system with the input images and approximate maps. This lighting information corresponds to the input images as it indicates lighting pattern used for each input image. In some embodiments, this lighting information is in the form of an irradiance map that simulates a particular lighting pattern for white paper (rather than the particular real-world material). Additionally, Paragraph 58 of Matthews teaches that in some implementations, in addition to the context image 30, the diffusion model 32 is also conditioned on a text prompt 34. The text prompt 34 can provide high-level instructions or guidelines about the desired edits to the material properties of the object in the image. For example, the prompt can be something like “make the object more metallic” or “increase the roughness of the object by 0.2 units”. The text prompt 34 can be encoded into a format suitable for the model, such as a vector representation obtained from a language model or word embedding model.);
and based at least in part on the user input, generate at least one of the one or more first material maps or the output frame (Col. 9, Lines 25-44 of Dave teach that the material maps generator 220 inputs, into the one or more machine learning models, the sets of specular and diffuse component images captured for a real-world material as well as the approximate maps (e.g., a diffuse albedo material map approximation and a normal material map approximation) generated by material map approximation component 210. Each set of material maps generated for a particular material may include a diffuse albedo material map, a normal material map, a specular albedo material map, and a roughness material map. In example aspects, material maps generator 220 utilizes a deep neural network system that includes multiple neural networks (e.g., U-Nets), where each neural network outputs a different type of material map. For instance, the material maps generator 220 may utilize four neural networks: (i) a diffuse albedo neural network trained to generate a diffuse albedo material map, (ii) a normal neural network trained to generate a normal material map, (iii) a specular albedo neural network trained to generate a specular albedo material map, and (iv) a roughness neural network trained to generate a roughness material map.).
Regarding claim 13, Dave in view of Matthews disclose everything claims as applied above (see claim 10), in addition, Dave in view of Matthews disclose wherein the one or more processing units are further to:
receive one or more lighting maps that represent at least one of one or more shading or lighting characteristics associated with the one or more objects (Col. 11, Line 63 through Col. 12, Line 3 of Dave teach that the material map approximations 306 as well as the input images 302 are then input into neural network system 308. Neural network system 308 generates a set of material maps 310 based on the material map approximations 306 and the input images 302. In some embodiments, lighting information, such as irradiance maps discussed above with respect to material maps generator 220, are also input into neural network system 308.);
and provide a representation of the one or more lighting maps as input into one or more first machine learning models to generate an output frame based at least in part on the first noise vector, the one or more material maps, and the one or more lighting maps (Col. 16, Lines 51-58 of Dave teach that the lighting information for each lighting pattern represented in the input images may be input into neural network to generate the material maps. This lighting information may be in the form of an irradiance map that describes a particular lighting pattern. Each set of diffuse component image and specular component image may be input with an irradiance map representing the lighting corresponding to the set. Additionally, paragraph 66 of Matthews teaches that the process also involves a context image 304, which is used to condition the diffusion model 310. The context image 304 provides a reference for the diffusion model 310, guiding the model in its task of denoising the noisy input 302. The context image 304 can be the same as the noisy input 302 or it can be a different image that depicts the same or a similar object. The context image 304 can include additional information about the object, such as its shape, size, orientation, lighting conditions, and initial material properties.).
Regarding claim 14, Dave in view of Matthews disclose everything claims as applied above (see claim 10), in addition, Dave in view of Matthews disclose wherein the first noise vector represents a noisy version of the input frame (Paragraphs 55 and 56 of Matthews teach that in some implementations, the context image 30 can depict the object having a base material property (e.g., with an attribute value set to zero). In contrast, the target image 28 can depict the object having an altered or edited material property (e.g., with the attribute value set to a non-zero value). The target image 28 and context image 30 are then provided as input to a denoising diffusion model 32. The diffusion model 32 is a machine learning model specifically designed for image generation and editing tasks. It can take as input a noisy image and apply a sequence of denoising operations to progressively generate a clean, final image. In the illustrated figure, the denoising diffusion model 32 is conditioned on the context image 30, which provides the model with information about the scene, the object, and its initial material properties.), and wherein the one or more processing units are further to:
receive a request to enhance the input frame, wherein an output frame is generated using the one or more first machine learning models based at least in part on the request and the input frame (Paragraph 63 of Matthews teaches that at the end of the process, an edited image 36 is generated based on the denoising prediction. The edited image 36 depicts the same object as in the context image 30 but with the material property of the object modified according to the instructions provided in the text prompt 34. The edited image 36 can be viewed as the output of the diffusion model 32, representing the result of the desired edit to the material property of the object in the image.),
and wherein the output frame includes one or more features that have been enhanced relative to the input frame (Col. 10, Lines 15-23 of Dave teaches that the embodiments of digital material generator 200 further include material maps optimizer 230 as depicted in FIG. 2. Material maps optimizer 230 is generally configured to generate optimized or updated material maps for the real-world material based on the maps generated by material maps generator 220. Material maps optimizer 230 includes an image renderer 232 and an updated maps generator 234. Material maps optimizer 230 may be a differentiable renderer and Col. 12, Lines 32-37 of Dave teach that the set of material maps 310 generated by neural network system 308 are optimized using differentiable renderer 312, which may be an embodiment of material maps optimizer 230 in FIG. 2. Differentiable renderer 312 renders images 314 from the material maps 310. Rendered images 314 are the same type as input images 302.).
Regarding claim 15, Dave in view of Matthews disclose everything claims as applied above (see claim 10), in addition, Dave in view of Matthews disclose wherein the input image represents a two-dimensional input frame (Col. 3, Lines 9-20 of Dave teaches that an “image” as described herein is a visual representation of one or more portions of the real world or a visual representation of one or more documents. For example, an image can be a digital photograph, a digital image among a sequence of video segments, a graphic image file (e.g., JPEG, PNG, etc.), a picture (or sub-element of a picture), and/or a bitmap among other things. A “visual rendering” as described herein refers to another image (e.g., in 2D or 3D) (e.g., an “output image”), a physically-based rendering (PBR) material, a SVBRDF PBR material, an animation, and/or other suitable media, content, or computer object.), and wherein the one or more processing units are further to:
provide the one or more first material maps, a user-specified lighting condition, and a second noise vector as input into one or more second machine learning models (Paragraph 71 of Matthews teaches that it should be noted that the components of FIG. 3 and the associated processes described herein are exemplary and that changes and modifications can be made without departing from the scope of the present disclosure. For example, the noisy input 302 and the context image 304 can be different images depicting the same object or different objects with similar material properties. The scalar edit value 306 can be any numerical value that represents a desired change in a material property, and the textual prompt 308 can contain any textual instruction that describes a desired edit to a material property.);
generate an output frame based at least in part the one or more first material maps, the user-specified lighting condition, and the second noise vector as input into the one or more second machine learning models (Paragraph 69 of Matthews teaches that the diffusion model 310 processes the noisy input 302, the context image 304, the scalar edit value 306, and the textual prompt 308 to generate an edited image 312. The edited image 312 depicts the same object as in the context image 304, but with a modified material property. The material property of the object in the edited image 312 is modified according to the instructions provided in the textual prompt 308 and the scalar edit value 306.),
wherein the output frame represents the two-dimensional input frame that includes a lighting property that has been modified in the output frame relative to the input frame (Col. 10, Lines 24-32 of Dave teaches that the image renderer 232 is configured to render images of the digital material from material maps output from material maps generator 220. Image renderer 232 renders the images by adding lighting to the material maps. The lighting of the rendered images is intended to simulate the lighting used in the input images capturing the real-world material. As such, image renderer 232 may receive lighting and camera information corresponding to the lighting and camera configurations used to capture the images).
Regarding claim 16, Dave in view of Matthews disclose everything claims as applied above (see claim 10), in addition, Dave in view of Matthews disclose wherein the input image represents a two-dimensional input frame (), and wherein the one or more processing units are further to:
generate a multidimensional frame based on the one or more first material maps, the multidimensional frame representing the two-dimensional input frame that includes at least one more dimension relative to the two-dimensional input frame (Paragraph [0029] of Matthews teaches that in computer graphics, the technology can be used to change the material properties of a 3D model, such as making a brick wall appear to be made of marble and paragraph [0052] of Matthews teaches that it should be noted that the process depicted in FIG. 1 is merely an example and that variations and modifications are possible. For instance, additional steps can be included in the process, such as steps for preprocessing the 3D objects, steps for postprocessing the rendered images, or steps for augmenting the training data. Additionally, Col. 11, Lines 12-20 of Dave teaches that the set of material maps (either the set initially generated by material maps generator 220 or the set of updated material maps generated by material maps optimizer 230) may represent the digital material corresponding to the real-world material captured in the images. The material maps may be utilized to create a visual rendering of an object or model with the digital material. The visual rendering may include applying the material maps to a 3D shape, such as a sphere.);
and based at least on the multidimensional frame, generate one or more second material maps (Col. 17, Lines 42-57 of Dave teaches that at block 606, a neural network system is utilized to generate an initial set of material maps for the real-world material. Block 606 may be performed by an embodiment of material maps generator 220 and in a similar manner described with respect to block 506 of FIG. 5. The material maps may be generated based on the one or more material map approximations and the input images. In some aspects, neural network system includes a plurality of neural networks that each generate a different type of material map, including a diffuse albedo material map, a specular albedo material map, a roughness material map, and a normal material map. At block 608, an updated set of material maps is generated. This updated set is optimized based on a comparison of the input images received at block 602 and images rendered from the initial set of material maps generated at block 606.).
Regarding claim 17, Dave in view of Matthews disclose everything claims as applied above (see claim 16), in addition, Dave in view of Matthews disclose wherein the one or more processing units are further to:
provide a second noise vector and the one or more second material maps as input into one or more second machine learning models to generate an output frame (Paragraph [0069] of Matthews teaches that the diffusion model 310 processes the noisy input 302, the context image 304, the scalar edit value 306, and the textual prompt 308 to generate an edited image 312. The edited image 312 depicts the same object as in the context image 304, but with a modified material property. Additionally, Col. 10, Lines 57-67 of Dave teaches that the material maps optimizer 230 may utilize a differentiable renderer to render the images and update the maps. For example, image renderer 232 may create the rendered images in a forward pass through the differentiable renderer and updated maps generator 234 may create updated, optimized material maps by passing the rendered images back through the differentiable renderer. The material maps and rendered images may be passed through the differentiable renderer back and forth multiple times, each times reducing the differences between the rendered images and the input images.), and wherein the output frame is generated based at least in part on generating the multidimensional frame ().
Regarding claim 18, the system steps correspond to and are rejected similarly to the processor steps of claim 9 (see claim 9 above).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Fyffe et al. (U.S. Patent: #12,437,470 B1) teaches rendering an avatar using neural maps.
Kim et al. (Pub. No.: US 2025/0239005 A1) teaches methods and systems for generating 2D and 3D images using diffusion models.
Pronovost (Pub. No.: US 2024/0212360 A1) teaches generating object data using a diffusion model.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to George Renze whose telephone number is (703)756-5811. The examiner can normally be reached Monday-Friday 9:00am - 6:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/G.R./Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights