DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This Office Action is in response to Applicant’s amendment filed 03/13/2026 which has
been entered and made of record. Claims 1, 6-8, 10 and 16 have been amended. No
claim has been cancelled or newly added. Claims 1-20 are pending in the application. Applicant’s amendments to the Claims have overcome each and every objection
previously set forth in the Non-Final Office Action mailed November 11th, 2025.
Response to Arguments
Applicant's arguments filed with respect to claims 1-20, filed on 3/13/2026 with respect
to the rejection under 35 USC 103, have been fully considered but they are not persuasive.
In response to applicant’s arguments that “However, the Office also cites to the very same "height-based shadowing system 106" for allegedly disclosing a separate claim element, namely "determining, using…”, examiner respectfully disagrees because in the first citation of "height-based shadowing system 106" the rejection clearly points to one of the many functions that the height-based shadowing system performs such as shadow generation.
In response to applicant’s arguments that “The "determining" step" is when…this is not the same "model" that produces the initial synthetic image” examiner respectfully disagrees because neural network 242 referred to as machine learning model can be a different model since Zhang paragraph 51 mentions "some embodiments, a neural network includes a combination of neural networks or neural network components" which means the model would be made of multiple models.
In response to applicant’s arguments that “Finally, the "final set of lighting parameters" - which has been updated using the "measure" - is provided to the "generative model" for rendering "subsequent synthetic images." In contrast, Zhang's "height-based shadowing system 106" generates various height maps”, examiner notes that the “updating” and the “subsequent synthetic images” are shown from the Sunkavalli reference in the non-final office action mailed on 11/14/25. Thus, in response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., “Zhang's "height-based shadowing system 106" measures various lighting effects, then generates height maps and shadows. Whereas claim 1 discloses "generating [ . ..] a synthetic image" having "one or more lighting effects," then a "machine learning model" determines a…”) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
In response to applicant’s arguments that “Sunkavalli discloses updating lighting parameters in response to changing positions of a light source, and not based on a machine learning model's "measure of at least the one or more lighting effects" "according to a criterion…” examiner respectfully disagrees becauseupdating of light parameters must come once you already have light parameters (since it is updating), the initial lighting parameters create the lighting effects which are measured and thus when you update those lighting parameters, you do it based on (in a step after) the measure from Zhang; and once again, in response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 5, 7-10, 13, 15-16 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (U.S. Patent Application Publication No. 2024/0273813), hereinafter referenced as Zhang, in view of Sunkavalli et al. (U.S. Patent Application Publication No. 2021/0065440), hereinafter referenced as Sunkavalli.
Regarding claim 1, Zhang teaches A computer-implemented method, comprising:
generating, by a generative model, (Zhang, paragraph 51 teaches "a neural network includes a generative adversarial neural network"); this shows generative model which is used for generating images; a synthetic image including a virtual object inserted into a scene represented in an input image, (paragraph 82 teaches "as shown in FIG. 3D, the training digital images 360 include synthetic training images 362 and real training images 364. In one or more embodiments, the synthetic training images 362 include computer-generated digital images. For instance, in some cases, the synthetic training images 362 include computer-generated three-dimensional models of digital objects and/or backgrounds"); since synthetic include computer-generated, this shows generating the synthetic images, the digital objects act as virtual object inserted in scene and as shown in fig. 3D cited, this is an input image; wherein one or more lighting parameters are applied to the virtual object to produce one or more lighting effects of the virtual object (paragraph 63 teaches "the height-based shadowing system 106 receives the light source information 238 from user input via a computing device, such as a client device 240. In some implementations, however, the height-based shadowing system 106 determines the light source information 238 by analyzing the digital image 220" and paragraph 64 teaches "the height-based shadowing system 106 utilizes a shadow stylization neural network 242 to generate a soft object shadow 244 for the digital object 224 of the digital image 220"); shadow generated is lighting effect for digital/virtual object and this is according to light source information which is considered lighting parameters (thus produced by such) since Zhang, paragraph 63 also teaches "For instance, in some cases, the height-based shadowing system 106 identifies a light source within the digital image 220 and determines a position of the light source by determining a coordinate of the light source" [Note: this definition of lighting parameters is consistent with applicant's disclosure paragraph 30 that says “Any of a number of different lighting parameters can be used or determined as may vary for different implementations or embodiments, including parameters such as intensity, color, location, orientation, direction, source type, light category, irradiance budget, irradiance quality, lighting mode, baked occlusion, lighting mode, decay, distance, strength, and/or contribution, among many other possibilities”]; determining, using a machine learning model and according to a criterion, a measure of at least the one or more lighting effects associated with the virtual object in the scene as represented in the synthetic image (paragraph 65 teaches "height-based shadowing system 106 utilizes the shadow stylization neural network 242 to generate the soft object shadow 244 further based on a softness value 246" and paragraph 67 teaches "utilizing geometry-aware buffer channels to generate soft object shadows for digital objects portrayed in digital images, the height-based shadowing system 106 generalizes the shadow rendering process. In particular, the height-based shadowing system 106 generates soft object shadows using data channels that provide guidance on the shapes, geometries, and/or environments of the contents of digital images"); neural network is machine learning (ML) model, criterion is the softness value (shadow is according softness and using ML), and determining the measure here of lighting effects/shadows associated with the virtual/digital object in the scene as represented in the synthetic image/digital image is the same as generating the shadow using guidance on shapes, geometries and environment (because a measure of shape and geometry would need to be determined for the shadow/lighting effect due to guidance for such).
However, Zhang fails to explicitly teach generating, by a generative model, a synthetic image… providing, to the generative model for rendering one or more subsequent synthetic images; a synthetic image updating the lighting parameters based in part on the measure (Although, Zhang, paragraph 103 teaches "the height-based shadowing system 106 generates the shadow stylization neural network 428 by utilizing multiple refinement iterations to refine its parameters and improve its accuracy in generating soft object shadow...generate a predicted soft object shadow, compares the predicted soft object shadow to a ground truth via a loss function, and back propagates the determined loss to the shadow stylization neural network 428 to update its parameters."); this implies lighting parameters/light source information is updated alongside the parameters and is based on the aforementioned measure since improving accuracy would mean to improve those shapes and geometries; and providing, to the generative model for rendering one or more subsequent synthetic images for the scene including one or more virtual objects, a final set of lighting parameters in response to the measure of perceptual realism satisfying a threshold corresponding to the criterion.
However, Sunkavalli explicitly teaches updating the lighting parameters based in part on the measure (Sunkavalli, paragraph 116 teaches "In addition to accurately portraying lighting conditions based on multiple light sources, 3D-source-specific-lighting parameters can dynamically capture lighting from different perspectives of a digital image. As the digital image 482 changes in perspective in camera viewpoint, model orientation, or other perspective adjustment...generate new 3D-source-specific-lighting parameters that accurately indicate lighting conditions for the designated position based on such perspective changes"); new lighting parameters to accurately indicate lighting conditions for the designated position shows updating lighting parameters and this would be based on the measure of light effects/shadows from Zhang as well as measure of object shadows here since one of ordinary skill in the art would understand that perspective change would lead to shadow changes which would show that the lighting parameters/source/location/position has also changed; and providing, to the generative model for rendering one or more subsequent synthetic images for the scene including one or more virtual objects, a final set of lighting parameters in response to the measure of perceptual realism satisfying a threshold corresponding to the criterion (paragraph 25 teaches "lighting-estimation-neural network to estimate lighting parameters specific to predicted light sources illuminating a digital image and render a virtual object in the digital image according to such source-specific-lighting parameters...The lighting estimation system can further modify such 3D-source-specific-lighting parameters based on user input, a change in position, or a change in lighting conditions within a digital image.", paragraph 26 teaches "In response to the request to render, the lighting estimation system renders a modified digital image comprising the virtual object at the designated position illuminated according to the 3D-source-specific-lighting parameters." and paragraph 31 teaches "the lighting estimation system can apply a differentiable-projection layer to 3D-source-specific-predicted-lighting parameters to project a predicted environment map corresponding to a digital training image and compare the environment map to a ground-truth-environment map corresponding to the digital training image. By comparing the predicted environment map to the ground-truth-environment map, the lighting estimation system determines an environment-map loss. Through multiple training iterations of the first training stage, the lighting estimation system modifies internal parameters of the source-specific-lighting-estimation-neural network based on such environment-map losses until a point of convergence."); as aforementioned, digital images in Zhang are used as synthetic images for the scene and include virtual object(s), thus modified digital image shows subsequent synthetic image, modifying internal parameters as well as 3D-source-specific-lighting parameters until a point of convergence shows this happening in response to measure of perceptual realism satisfying a threshold which would be corresponding to the criterion (softness value and shadow from there) when view in combination with Zhang because paragraph 143 Sunkavalli mentions "si-RMSE factors out scale differences between a source-specific-lighting-estimation-neural network and Gardner's neural network, on the one hand, and ground-truth-environment maps, on the other hand. The si-RMSE further focuses on cues such as shading and shadows primarily from light-source positions." meaning the losses associated with the ground-truth environment map would also account for shadows (and softness value/criterion causing the shadows from Zhang).
Sunkavalli also teaches generating, by a generative model, a synthetic image … (Sunkavalli, abstract teaches, “using an object relighting neural network to generate digital images”); as aforementioned digital images in Zhang contain synthetic images and since Zhang paragraph 51 mentions “neural network includes …a generative adversarial neural network…some embodiments, a neural network includes a combination of neural networks”, this shows the generation of synthetic images would be by the generative model/generative adversarial neural network; providing, to the generative model for rendering one or more subsequent synthetic images (Sunkavalli, paragraph 64 teaches “shadow stylization neural network generates a soft object shadow for a digital object by modifying a hard object shadow generated for the digital object” and fig. 2B shows digital image 220 provided to Neural network 222[would be generative model when viewed in combination with Zhang] to generate modified digital image); as aforementioned digital images in Zhang contain synthetic images and since Zhang paragraph 51 mentions “neural network includes …a generative adversarial neural network…some embodiments, a neural network includes a combination of neural networks”, this shows the providing of the synthetic/digital images to the generative model/generative adversarial neural network for rendering would occur;
Sunkavalli is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of lighting parameters and calculations of such using comparisons and measurements. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhang's invention with the updating and calculating of light parameter techniques of Sunkavalli to generate more accurate lighting parameters corresponding to a predicted light source (Sunkavalli, paragraph 33). This means higher user satisfaction due to more accurate results.
Regarding claim 2, the combination of Zhang and Sunkavalli teaches wherein the synthetic image is generated using at least one of a differentiable renderer or a generative neural network (Zhang, paragraph 82 teaches "system 106 utilizes training digital images 360 for generating a height prediction neural network 366. In particular, as shown in FIG. 3D, the training digital images 360 include synthetic training images 362" and paragraph 51 teaches "a neural network includes...a generative adversarial neural network"); this means synthetic image is generated by neural network which includes generative adversarial neural network.
Regarding claim 5, the combination of Zhang and Sunkavalli teaches wherein the lighting parameters are determined and updated using at least one of an environmental map, a spherical Gaussian, or a neural radiance model (Sunkavalli, paragraph 61 teaches "c.sub.i represents ground-truth-source-specific-color parameters estimating a color of a light source...in some embodiments, the lighting estimation system 108 projects... into a spherical gaussian representation." and paragraph 57 teaches "the ground-truth-environment map 206 corresponding to the digital training image 204, in certain implementations, the digital imagery system 106 derives or otherwise generates ground-truth-source-specific-lighting parameters"); this shows both environment map and spherical Gaussian used explicitly for lighting parameter determination and updates. The same motivations used in claim 1 apply here in claim 5.
Regarding claim 7, the combination of Zhang and Sunkavalli teaches wherein the lighting effects include at least one of one or more shadows, one or more reflections, one or more refractions, one or more diffractions, one or more material properties, or one or more camera properties (Zhang, paragraph 64 teaches "the height-based shadowing system 106 utilizes a shadow stylization neural network 242 to generate a soft object shadow 244 for the digital object 224 of the digital image 220" and Sunkavalli, paragraph 116 teaches "capture lighting from different perspectives of a digital image. As the digital image 482 changes in perspective in camera viewpoint" and Sunkavalli paragraph 157 teaches " a digital image comprising floor reflections from a light source."); this shows shadows from Zhang and both reflection and camera viewpoint/properties due to perspective in Sunkavalli. The same motivations used in claim 1 apply here in claim 7.
Regarding claim 8, the combination of Zhang and Sunkavalli teaches further comprising:
generating one or more additional synthetic images to provide as input to the machine learning model, (Zhang, fig. 3D shows synthetic training images 362 used as input to the machine learning model/neural network 366); and updating based in part on one or more loss values output by the machine learning model, the updated lighting parameters until the measure is determined to at least satisfy the threshold (Sunkavalli, paragraph 31 teaches "the lighting estimation system can apply a differentiable-projection layer to 3D-source-specific-predicted-lighting parameters to project a predicted environment map corresponding to a digital training image and compare the environment map to a ground-truth-environment map corresponding to the digital training image. By comparing the predicted environment map to the ground-truth-environment map, the lighting estimation system determines an environment-map loss. Through multiple training iterations of the first training stage, the lighting estimation system modifies internal parameters of the source-specific-lighting-estimation-neural network based on such environment-map losses until a point of convergence." and Sunkavalli, paragraph 25 teaches "lighting-estimation-neural network to estimate lighting parameters specific to predicted light sources illuminating a digital image and render a virtual object in the digital image according to such source-specific-lighting parameters...The lighting estimation system can further modify such 3D-source-specific-lighting parameters based on user input, a change in position, or a change in lighting conditions within a digital image."); this shows lighting parameters updated/modified until a point of convergence (measure of light has realism threshold satisfied) based on losses, and the loss when viewed in combination can be loss value (loss function 372) output by machine learning model in fig. 3D of Zhang. The same motivations used in claim 1 apply here in claim 8.
Regarding claim 9, the combination of Zhang and Sunkavalli teaches wherein the machine learning model is trained to perform physics-based rendering, (Zhang, paragraph 104 teaches "the height-based shadowing system 106 utilizes synthetic images composed of three-dimensional models as the training images and further utilizes a physics-based shadow rendering model to generate soft object shadow...the height-based shadowing system 106 utilizes the physics-based rendering model described in PBR Render" and paragraph 244 teaches "height-based shadowing system 106 utilizes the shadow stylization neural network 242 to generate the soft object shadow 244"); since height-based shadowing system utilizes both neural network and PBR (physics-based rendering) to generate shadow, this shows ML/neural network using the PBR;
and further updated using training data for a plurality of lighting effects applied to a plurality of objects in a plurality of environments (Zhang, paragraph 67 teaches "height-based shadowing system 106 generates soft object shadows using data channels that provide guidance on the shapes, geometries, and/or environments of the contents of digital images"); shadows/lighting effects using data channels that provide guidance on environments ([plural] and plurality of environments would have plurality of object) shows the ML would be updated using this training data.
Regarding claim 10, Zhang teaches A processor, comprising: one or more circuits to: (fig. 11 shows a processor 1102 which one of ordinary skill in the art would understand has circuits); generate a synthetic image using a generative model, (Zhang, paragraph 51 teaches "a neural network includes a generative adversarial neural network" and paragraph 82 teaches "as shown in FIG. 3D, the training digital images 360 include synthetic training images 362 and real training images 364. In one or more embodiments, the synthetic training images 362 include computer-generated digital images. For instance, in some cases, the synthetic training images 362 include computer-generated three-dimensional models of digital objects and/or backgrounds"); since synthetic include computer-generated, this shows generating the synthetic images and this shows generative model which is used for generating images;
wherein one or more lighting parameters are applied to the virtual object to produce one or more lighting effects of the virtual object (paragraph 63 teaches "the height-based shadowing system 106 receives the light source information 238 from user input via a computing device, such as a client device 240. In some implementations, however, the height-based shadowing system 106 determines the light source information 238 by analyzing the digital image 220" and paragraph 64 teaches "the height-based shadowing system 106 utilizes a shadow stylization neural network 242 to generate a soft object shadow 244 for the digital object 224 of the digital image 220"); shadow generated is lighting effect for digital/virtual object and this is according to light source information which is considered lighting parameters (thus produced by such) since Zhang, paragraph 63 also teaches "For instance, in some cases, the height-based shadowing system 106 identifies a light source within the digital image 220 and determines a position of the light source by determining a coordinate of the light source" [Note: this definition of lighting parameters is consistent with applicant's disclosure paragraph 30 that says “Any of a number of different lighting parameters can be used or determined as may vary for different implementations or embodiments, including parameters such as intensity, color, location, orientation, direction, source type, light category, irradiance budget, irradiance quality, lighting mode, baked occlusion, lighting mode, decay, distance, strength, and/or contribution, among many other possibilities”]; process the synthetic image using a machine learning model to determine a loss value with respect to the synthetic image (fig. 3D teaches inputting synthetic images 362 into neural network/ML 366 leading to a loss function 372 and paragraph 83 teaches "the height-based shadowing system 106 utilizes the height prediction neural network 366 to analyze a training digital image from the training digital images 360 (e.g., one of the synthetic training images 362 or the real training images 364) and generate a predicted height map 368 based on the analysis. Further, the height-based shadowing system 106 compares the predicted height map 368 to a ground truth 370 via a loss function 372."); one of ordinary skill in the art would understand that the loss function provides a loss value and since this is due to the height map from the synthetic training image being analyzed, the loss value would be with respect to the synthetic image.
However, Zhang fails to explicitly teach update one or more of the lighting parameters (Although, Zhang, paragraph 103 teaches "the height-based shadowing system 106 generates the shadow stylization neural network 428 by utilizing multiple refinement iterations to refine its parameters and improve its accuracy in generating soft object shadow...generate a predicted soft object shadow, compares the predicted soft object shadow to a ground truth via a loss function, and back propagates the determined loss to the shadow stylization neural network 428 to update its parameters."); this implies lighting parameters/light source information is updated alongside the parameters; and generate an updated synthetic image using the generative model based at least in part on the loss value and the one or more updated lighting parameters.
However, Sunkavalli explicitly teaches update one or more of the lighting parameters (Sunkavalli, paragraph 116 teaches "In addition to accurately portraying lighting conditions based on multiple light sources, 3D-source-specific-lighting parameters can dynamically capture lighting from different perspectives of a digital image. As the digital image 482 changes in perspective in camera viewpoint, model orientation, or other perspective adjustment...generate new 3D-source-specific-lighting parameters that accurately indicate lighting conditions for the designated position based on such perspective changes"); new lighting parameters to accurately indicate lighting conditions for the designated position shows updating lighting parameters; and generate an updated synthetic image using the generative model based at least in part on the loss value and the one or more updated lighting parameters (Sunkavalli, paragraph 50 teaches "after generating such lighting parameters, the digital imagery system 106 renders the modified digital image 114 comprising the virtual object 104 at the designated position illuminated according to the 3D-source-specific-lighting parameters 112" and paragraph 63 teaches "In addition to generating ground-truth-source-specific-lighting parameters corresponding to digital training images, in some embodiments, the digital imagery system 106 further adjusts or tunes the light intensities within the digital training images using a rendering-based-optimization process."); since digital images include synthetic images from Zhang (as aforementioned), this update to digital image here shows synthetic image is also updated due to adjusting/based on light intensity [Note: light parameter also includes light intensities from paragraph 30 of applicant's disclosure], it would be done using the generative neural network of Zhang because Zhang, paragraph 82 teaches "system 106 utilizes training digital images 360 for generating a height prediction neural network....as shown in FIG. 3D, the training digital images 360 include synthetic training images 362 and real training images 364"[model would learn from training how to generate those images], and comparing to ground truth here also shows based on loss value which can be of Zhang as well when viewed in combination.
Sunkavalli also teaches generate a synthetic image using a generative model, (Sunkavalli, abstract teaches, “using an object relighting neural network to generate digital images”); as aforementioned digital images in Zhang contain synthetic images and since Zhang paragraph 51 mentions “neural network includes …a generative adversarial neural network…some embodiments, a neural network includes a combination of neural networks”, this shows the generation of synthetic images would be by the generative model/generative adversarial neural network;
Sunkavalli is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of lighting parameters and calculations of such using comparisons and measurements. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhang's invention with the updating and calculating of light parameter techniques of Sunkavalli to generate more accurate lighting parameters corresponding to a predicted light source (Sunkavalli, paragraph 33). This means higher user satisfaction due to more accurate results.
Regarding claim 13, the combination of Zhang and Sunkavalli teaches wherein the one or more circuits are further to: generate additional synthetic images and update the lighting parameters until at least one criterion threshold is reached (Sunkavalli, paragraph 25 teaches "lighting-estimation-neural network to estimate lighting parameters specific to predicted light sources illuminating a digital image and render a virtual object in the digital image according to such source-specific-lighting parameters...The lighting estimation system can further modify such 3D-source-specific-lighting parameters based on user input, a change in position, or a change in lighting conditions within a digital image.", paragraph 26 teaches "In response to the request to render, the lighting estimation system renders a modified digital image comprising the virtual object at the designated position illuminated according to the 3D-source-specific-lighting parameters." and paragraph 31 teaches "the lighting estimation system can apply a differentiable-projection layer to 3D-source-specific-predicted-lighting parameters to project a predicted environment map corresponding to a digital training image and compare the environment map to a ground-truth-environment map corresponding to the digital training image. By comparing the predicted environment map to the ground-truth-environment map, the lighting estimation system determines an environment-map loss. Through multiple training iterations of the first training stage, the lighting estimation system modifies internal parameters of the source-specific-lighting-estimation-neural network based on such environment-map losses until a point of convergence."); as aforementioned, digital images in Zhang are used as synthetic images for the scene and include virtual object(s), thus modified digital image shows additional synthetic image, and this also shows lighting parameters updated/modified until a point of convergence (criteria of realism threshold satisfied). The same motivations used in claim 10 apply here in claim 13.
Regarding claim 15, the combination of Zhang and Sunkavalli teaches wherein the processor is comprised in at least one of:
a system for performing light transport simulation (Sunkavalli, paragraph 125 teaches "the lighting estimation system 108 can generate new 3D-source-specific-lighting parameters and an adjusted digital image in response to a position-adjustment request to render a virtual object at a new designated position. FIG. 5C depicts an example of such an adjusted digital image reflecting new 3D-source-specific-lighting parameters.");
a system for generating or presenting augmented reality (AR) content (Sunkavalli, paragraph 167 teaches "the digital imagery system 1104 facilitates the generation, modification, sharing, accessing, storing, and/or deletion of digital content in augmented-reality-based imagery or visual-effects-based imagery (e.g., two-dimensional-digital image of a scene or three-dimensional-digital model of a scene)."); a system for synthetic data generation (Zhang, paragraph 82 teaches "system 106 utilizes training digital images 360 for generating a height prediction neural network 366. In particular, as shown in FIG. 3D, the training digital images 360 include synthetic training images 362 and real training images 364. In one or more embodiments, the synthetic training images 362 include computer-generated digital images"). The same motivations used in claim 10 apply here in claim 15.
Regarding claim 16, Zhang teaches A system, comprising: one or more processors to (Zhang, fig. 11 shows device/system 1100 with processor 1102 and paragraph 174 teaches "general-purpose computer including computer hardware, such as, for example, one or more processors and system memory"); generating, by a generative model, a sequence of synthetic images (Zhang, paragraph 51 teaches "a neural network includes a generative adversarial neural network"); this shows generative model which is used for generating images; determine a set of lighting parameters for a scene represented in an input image by, in part, generating a sequence of synthetic images including at least one virtual object inserted into the input image (paragraph 63 teaches "the height-based shadowing system 106 receives the light source information 238 from user input via a computing device, such as a client device 240. In some implementations, however, the height-based shadowing system 106 determines the light source information 238 by analyzing the digital image 220", paragraph 64 teaches "the height-based shadowing system 106 utilizes a shadow stylization neural network 242 to generate a soft object shadow 244 for the digital object 224 of the digital image 220" and paragraph 82 teaches "as shown in FIG. 3D, the training digital images 360 include synthetic training images 362 and real training images 364. In one or more embodiments, the synthetic training images 362 include computer-generated digital images. For instance, in some cases, the synthetic training images 362 include computer-generated three-dimensional models of digital objects and/or backgrounds"); shadow generated is lighting effect for digital/virtual object and this is according to light source information being determined which is considered lighting parameters since paragraph 63 also teaches "For instance, in some cases, the height-based shadowing system 106 identifies a light source within the digital image 220 and determines a position of the light source by determining a coordinate of the light source" [Note: this definition of lighting parameters is consistent with applicant's disclosure paragraph 30 that says “Any of a number of different lighting parameters can be used or determined as may vary for different implementations or embodiments, including parameters such as intensity, color, location, orientation, direction, source type, light category, irradiance budget, irradiance quality, lighting mode, baked occlusion, lighting mode, decay, distance, strength, and/or contribution, among many other possibilities”], also, since synthetic include computer-generated, this shows generating the synthetic images (sequence), the digital objects act as virtual object inserted in scene, and as shown in fig. 3D cited, this is an input image; wherein one or more lighting parameters are applied to the virtual object to produce one or more lighting effects of the virtual object, (paragraph 63 teaches "the height-based shadowing system 106 receives the light source information 238 from user input via a computing device, such as a client device 240. In some implementations, however, the height-based shadowing system 106 determines the light source information 238 by analyzing the digital image 220" and paragraph 64 teaches "the height-based shadowing system 106 utilizes a shadow stylization neural network 242 to generate a soft object shadow 244 for the digital object 224 of the digital image 220"); shadow generated is lighting effect for digital/virtual object and this is according to light source information which is considered lighting parameters (thus produced by such) since Zhang, paragraph 63 also teaches "For instance, in some cases, the height-based shadowing system 106 identifies a light source within the digital image 220 and determines a position of the light source by determining a coordinate of the light source" [Note: this definition of lighting parameters is consistent with applicant's disclosure paragraph 30 that says “Any of a number of different lighting parameters can be used or determined as may vary for different implementations or embodiments, including parameters such as intensity, color, location, orientation, direction, source type, light category, irradiance budget, irradiance quality, lighting mode, baked occlusion, lighting mode, decay, distance, strength, and/or contribution, among many other possibilities”];.
However, Zhang fails to explicitly teach generating, by a generative model, a sequence of synthetic images; with lighting effects being updated according to a sequence of updated lighting parameters until at least one synthetic image of the sequence satisfies a criterion, wherein the updated lighting parameters are iteratively updated based in part upon loss values determined by a machine learning model processing the sequence of synthetic images (Although, Zhang, paragraph 87 teaches "height-based shadowing system 106 back propagates the determined loss to the height prediction neural network 366 (as shown by the dashed line 374) to update the parameters of the height prediction neural network 366. In particular, the height-based shadowing system 106 updates the parameters to minimize the error of the height prediction neural network 366 in generating height maps for digital objects and/or backgrounds portrayed in digital images." and Zhang, paragraph 103 teaches "the height-based shadowing system 106 generates the shadow stylization neural network 428 by utilizing multiple refinement iterations to refine its parameters and improve its accuracy in generating soft object shadow...generate a predicted soft object shadow, compares the predicted soft object shadow to a ground truth via a loss function, and back propagates the determined loss to the shadow stylization neural network 428 to update its parameters.").
However, Sunkavalli explicitly teaches with lighting effects being updated according to a sequence of updated lighting parameters until at least one synthetic image of the sequence satisfies a criterion, wherein the updated lighting parameters are iteratively updated based in part upon loss values determined by a machine learning model processing the sequence of synthetic images (Sunkavalli, paragraph 25 teaches "lighting-estimation-neural network to estimate lighting parameters specific to predicted light sources illuminating a digital image and render a virtual object in the digital image according to such source-specific-lighting parameters...The lighting estimation system can further modify such 3D-source-specific-lighting parameters based on user input, a change in position, or a change in lighting conditions within a digital image.", paragraph 26 teaches "In response to the request to render, the lighting estimation system renders a modified digital image comprising the virtual object at the designated position illuminated according to the 3D-source-specific-lighting parameters." and paragraph 31 teaches "the lighting estimation system can apply a differentiable-projection layer to 3D-source-specific-predicted-lighting parameters to project a predicted environment map corresponding to a digital training image and compare the environment map to a ground-truth-environment map corresponding to the digital training image. By comparing the predicted environment map to the ground-truth-environment map, the lighting estimation system determines an environment-map loss. Through multiple training iterations of the first training stage, the lighting estimation system modifies internal parameters of the source-specific-lighting-estimation-neural network based on such environment-map losses until a point of convergence."); as aforementioned, digital images in Zhang are used as synthetic images for the scene and include virtual object(s), thus modified digital image shows subsequent synthetic image (sequence), modifying/updating internal parameters as well as 3D-source-specific-lighting parameters until a point of convergence shows this happening iteratively in response to loss value/comparing to ground truth which would be corresponding to and until satisfying criterion (point of convergence/minimized error), also when viewed in combination this would be done using the generative neural network/ML of Zhang because Zhang, paragraph 82 teaches "system 106 utilizes training digital images 360 for generating a height prediction neural network....as shown in FIG. 3D, the training digital images 360 include synthetic training images 362 and real training images 364"[model would learn from training how to generate those images].
Sunkavalli also teaches generating, by a generative model, a sequence of synthetic images (Sunkavalli, abstract teaches, “using an object relighting neural network to generate digital images”); as aforementioned digital images in Zhang contain synthetic images and since Zhang paragraph 51 mentions “neural network includes …a generative adversarial neural network…some embodiments, a neural network includes a combination of neural networks”, this shows the generation of synthetic images would be by the generative model/generative adversarial neural network.
Sunkavalli is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of lighting parameters and calculations of such using comparisons and measurements. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhang's invention with the updating and calculating of light parameter techniques of Sunkavalli to generate more accurate lighting parameters corresponding to a predicted light source (Sunkavalli, paragraph 33). This means higher user satisfaction due to more accurate results.
Regarding claim 20, the combination of Zhang and Sunkavalli teaches
wherein the system comprises at least one of: a system for performing light transport simulation (Sunkavalli, paragraph 125 teaches "the lighting estimation system 108 can generate new 3D-source-specific-lighting parameters and an adjusted digital image in response to a position-adjustment request to render a virtual object at a new designated position. FIG. 5C depicts an example of such an adjusted digital image reflecting new 3D-source-specific-lighting parameters.");
a system for generating or presenting augmented reality (AR) content (Sunkavalli, paragraph 167 teaches "the digital imagery system 1104 facilitates the generation, modification, sharing, accessing, storing, and/or deletion of digital content in augmented-reality-based imagery or visual-effects-based imagery (e.g., two-dimensional-digital image of a scene or three-dimensional-digital model of a scene)."); a system for synthetic data generation (Zhang, paragraph 82 teaches "system 106 utilizes training digital images 360 for generating a height prediction neural network 366. In particular, as shown in FIG. 3D, the training digital images 360 include synthetic training images 362 and real training images 364. In one or more embodiments, the synthetic training images 362 include computer-generated digital images"). The same motivations used in claim 16 apply here in claim 20.
Claim(s) 3, 12 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Sunkavalli as applied to claims 1, 10 and 16 above, and further in view of Kumari et al. (U.S. Patent Application Publication No. 2024/0185588), hereinafter referenced as Kumari.
Regarding claim 3, the combination of Zhang and Sunkavalli fails to teach wherein the machine learning model is a diffusion model, and wherein the measure is determined based at least in part upon a comparison of the synthetic image to a diffused image generated by the diffusion model receiving the synthetic image as input.
However, Kumari teaches wherein the machine learning model is a diffusion model, and wherein the measure is determined based at least in part upon a comparison of the synthetic image to a diffused image generated by the diffusion model receiving the synthetic image as input (Kumari, paragraph 67 teaches "the system compares synthetic image to image. For example, a training component may measure the differences between the synthetic image and the image and generate a loss term. At operation 425, the system updates diffusion model" and fig. 4 teaches image by diffusion model at step 405, and comparing synthetic image to image at step 420, then updating diffusion model at step 425.); This shows ML being diffusion model and the measure of loss based on comparing a synthetic image to diffused image which would happen after synthetic image is generated/received by the model. Kumari is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of using diffusion model for synthetic image comparisons. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Zhang and Sunkavalli with the comparison using diffusion model techniques of Kumari to minimize the loss term (Kumari, paragraph 67). This means more accurate data and better results.
Regarding claim 12, the combination of Zhang and Sunkavalli fails to teach wherein the one or more circuits are further to:
calculate the loss value in part by comparing a reconstructed image, generated by a diffusion model receiving the synthetic image as input, with the synthetic image.
However, Kumari teaches wherein the one or more circuits are further to:
calculate the loss value in part by comparing a reconstructed image, generated by a diffusion model receiving the synthetic image as input, with the synthetic image (Kumari, paragraph 67 teaches "the system compares synthetic image to image. For example, a training component may measure the differences between the synthetic image and the image and generate a loss term. At operation 425, the system updates diffusion model" and fig. 4 teaches image by diffusion model at step 405, and comparing synthetic image to image at step 420, then updating diffusion model at step 425.); This shows ML being diffusion model and the measure of loss based on comparing a synthetic image (which must be input to be compared) to diffused image (image by diffusion model means generated by diffusion model) which would be reconstructed since diffusion model is updated at step 425 meaning image would be too and would be reconstructed with each iteration. The same motivations used in claim 3 apply here in claim 12.
Regarding claim 18, the combination of Zhang and Sunkavalli fails to teach wherein the loss values are calculated in part by comparing reconstructed images, generated by a diffusion model receiving the synthetic images as input, with the corresponding synthetic images.
However, Kumari teaches wherein the loss values are calculated in part by comparing reconstructed images, generated by a diffusion model receiving the synthetic images as input, with the corresponding synthetic images (Kumari, paragraph 67 teaches "the system compares synthetic image to image. For example, a training component may measure the differences between the synthetic image and the image and generate a loss term. At operation 425, the system updates diffusion model" and fig. 4 teaches image by diffusion model at step 405, and comparing synthetic image to image at step 420, then updating diffusion model at step 425.); This shows ML being diffusion model and the measure of loss based on comparing a synthetic image (which must be input to be compared) to diffused image (image by diffusion model means generated by diffusion model) which would be reconstructed since diffusion model is updated at step 425 meaning image would be too and would be reconstructed with each iteration. The same motivations used in claim 3 apply here in claim 18.
Claim(s) 4 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Sunkavalli as applied to claims 1 and 16 above, and further in view of Li et al. (U.S. Patent Application Publication No. 2022/0084204), hereinafter referenced as Li.
Regarding claim 4, the combination of Zhang and Sunkavalli fails to teach wherein the machine learning model includes a discriminator, and wherein the measure is provided as output of the discriminator based in part on processing the synthetic image.
However, Li teaches wherein the machine learning model includes a discriminator, and wherein the measure is provided as output of the discriminator based in part on processing the synthetic image (Li, paragraph 78 teaches "second discriminator network of GAN's two discriminator networks takes as a first input a synthetic image and as a second input one or more generated labels and/or other outputs associated with synthetic image, and outputs a second score for synthetic image and associated generated labels"); score/measure is shown as output of discriminator here and based on processing the synthetic image. Li is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of output of discriminator providing measure based on synthetic image. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Zhang and Sunkavalli with the output of discrimainator techniques of Li to improve reliability, safety and performance (Li, paragraph 224). This would be due to the hardware structure associated with the specific neural network.
Regarding claim 17, the combination of Zhang and Sunkavalli fails to teach wherein the machine learning model is a discriminator model receiving the sequence of synthetic images as input and inferring a probability of realism to be used to calculate the loss values.
However, Li teaches wherein the machine learning model is a discriminator model receiving the sequence of synthetic images as input and inferring a probability of realism to be used to calculate the loss values (Li, paragraph 78 teaches "Images used in training a GAN can be real images, synthetic images, ... first discriminator network of a GAN's two discriminator networks takes as an input a synthetic image that was generated by generator network of GAN, and outputs a first score for synthetic image. First score represents a probability that synthetic image is a real image. A second discriminator network of GAN's two discriminator networks takes as a first input a synthetic image and as a second input one or more generated labels and/or other outputs associated with synthetic image, and outputs a second score for synthetic image and associated generated labels. Second score represents a probability that synthetic image and associated labels are real" and paragraph 93 teaches "a difference between synthetic version and input image can be determined using a loss function"); GAN's two discriminator network shows machine learning being discriminator model which receive synthetic images (sequence/plural) as input and as stated the score is a probability which would be of realism because it’s for determining if the image is real, and loss calculation mentioned determines difference between synthetic version and input image (would have to use determination of realness) meaning this probability of realism is used to calculate loss value to see the difference. Li is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of output of discriminator providing probability based on synthetic image. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Zhang and Sunkavalli with the output of discrimainator techniques of Li to improve reliability, safety and performance (Li, paragraph 224). This would be due to the hardware structure associated with the specific neural network.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Sunkavalli as applied to claim 5 above, and further in view of Chakraborty et al. (U.S. Patent No. 12,367,636), hereinafter referenced as Chakraborty.
Regarding claim 6, the combination of Zhang and Sunkavalli fails to explicitly teach wherein the neural radiance model is further used to generate the synthetic image.
However, Chakraborty explicitly teaches wherein the neural radiance model is further used to generate the synthetic image (Chakraborty, claim 1 teaches “neural radiance model is used to generate a plurality of synthetic images,”); this shows neural radiance model generating synthetic image. Chakraborty is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of neural radiance model generating synthetic image. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Zhang and Sunkavalli with the neural radiance model of Chakraborty for ensuring this enables the neural radiance model to be used as an efficient data generator (Chakraborty, col.7, lines 64-66). This would be done by maximizing the usage of other models in other areas while using neural radiance for synthetic image generator. Therefore, leading to a more efficient system overall.
Claim(s) 11 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Sunkavalli as applied to claim 10 above, and further in view of Rama (U.S. Patent Application Publication No. 2023/0334290), hereinafter referenced as Rama.
Regarding claim 11, the combination of Zhang and Sunkavalli fails to teach wherein the one or more circuits are further to: calculate the loss value using a discriminator network receiving the synthetic image as input.
However, Rama teaches wherein the one or more circuits are further to: calculate the loss value using a discriminator network receiving the synthetic image as input (Rama, paragraph 77 teaches "method 450 begins by the discriminator network 216 receiving original input data 202 and synthetic data 214" and paragraph 80 teaches "The loss function of the discriminator network 216 is designed to have a large loss if the discriminator network 216 incorrectly predicts the original input data 202 and the synthetic data 214"); this shows discriminator receiving synthetic data/image and loss function/value calculated for such. Rama is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of loss value calculation using a discriminator. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Zhang and Sunkavalli with the discriminator for loss value techniques of Rama to ensure the generated synthetic data is iteratively improved to the point it sufficiently resembles original data and thus is eligible for use in training an additional ML model that, due to privacy or security concerns, cannot or will not use the original data (Rama, paragraph 18). This means further versatility.
Regarding claim 14, the combination of Zhang and Sunkavalli fails to teach wherein the lighting parameters are updated in part by adjusting one or more network parameters of the generative model, wherein the generative model is fine-tuned for the scene.
However, Rama teaches wherein the lighting parameters are updated in part by adjusting one or more network parameters of the generative model, wherein the generative model is fine-tuned for the scene (Rama, paragraph 18 teaches "By iteratively minimizing loss, parameters for the neural network are iteratively improved as well. By applying stochastic gradient descent to competing neural networks, such as the a generator network ML model and a discriminator network ML model that generate synthetic data and predict whether the synthetic data is original or synthetic"); the light parameters updated in Sunkavalli would be due to this iterative minimization of loss and parameters of neural network improved/adjusted (since Sunkavalli, paragraph 25 teaches "lighting-estimation-neural network to estimate lighting parameters" meaning the neural network parameters directly affect light parameters)
and minimizing loss also shows fine-tuning the generative model (here as generator model + discriminator model). The same motivations used in claim 11 apply here in claim 14.
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Sunkavalli as applied to claim 16 above, and further in view of Lee et al. (U.S. Patent Application Publication No. 2021/0133938), hereinafter referenced as Lee.
Regarding claim 19, the combination of Zhang and Sunkavalli fails to teach wherein the set of lighting parameters are provided as input to a generative model to generate the synthetic images, or learned by the generative model.
However, Lee teaches wherein the set of lighting parameters are provided as input to a generative model to generate the synthetic images, or learned by the generative model (Lee, paragraph 24 teaches "generative adversarial network model may be trained using original high illumination intensity images inputted to the discriminator and fake high illumination intensity images generated by the generator."); using high illumination intensity images input to generative model shows light parameters input since applicant's disclosure paragraph 30 describes light parameter can be intensity (a set can include one item) and the fake images generated are the synthetic images. Lee is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of generative model inputting light parameters and outputting synthetic/fake images. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Zhang and Sunkavalli with the light parameter to generative model techniques of Lee to enhance illumination intensity of an image according to embodiments of the present disclosure enable image quality to be enhanced in the most effective manner by designing an efficient image processing neural network model (Lee, paragraph 30). This would mean a more aesthetically pleasing display/image in reduced time.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NAUMAN U AHMAD whose telephone number is (703)756-5306. The examiner can normally be reached Monday - Friday 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571) 272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
/N.U.A./Examiner, Art Unit 2611
Prosecution Insights