Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 3 and 23 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, regards as the invention.
Regarding claims 3 and 23, each of these claims recites “the lighting image” which does not have antecedent basis in the claims; it is unclear to which image this was intended to refer; hence the scope of the claim is indefinite. The Examiner suggests amending this phrase such that it properly references the element to which it was intended to refer. For the purposes of examination, consistent with the specification, this will be interpreted to mean that the foreground image relighting may be based on a panoramic image (see Applicant’s specification, paragraph 97).
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 2, 4, 5, 7-9, 18-22, 24, 25, 27, and 28 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Zeng et al. (DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation), August 1, 2024, referred herein as Zeng.
Regarding claim 1, Zeng teaches a method for image generation, comprising: obtaining a source image and a lighting input, wherein the lighting input indicates a lighting condition for the source image (figure 1 and caption; figure 3 and caption, “We start by generating a provisional image using a pretrained diffusion model under uncontrolled lighting given a text prompt and a content-seed”; abstract, “In the first stage, we leverage a standard pretrained diffusion model to generate a provisional image under uncontrolled lighting.”; section 3, subsection 1, “Provisional…”; a provisional [source] image and a lighting input is obtained, where the lighting input indicates a desired lighting condition for the source image);
generating, using an image generation model, a relighted foreground image and a relighted background image based on the source image and on the lighting input, wherein the relighted foreground image depicts a foreground element with the lighting condition and the relighted background image depicts a background element with the lighting condition (figure 3 and caption, “Next, we pass an appearance-seed, the provisional image, and a set of radiance hints (computed from the target lighting and a coarse estimate of the depth) to DiLightNet that will resynthesize the image such that becomes consistent with the target lighting while retaining the content of the provisional image.”; abstract, “[I]n the second stage, we resynthesize and refine the foreground object in the generated image by passing the target lighting to a refined diffusion model, named DiLightNet, using radiance hints computed on a coarse shape of the foreground object inferred from the provisional image. To retain the texture details, we multiply the radiance hints with a neural encoding of the provisional synthesized image before passing it to DiLightNet. Finally, in the third stage, we resynthesize the background to be consistent with the lighting on the foreground object.”; section 3, subsections 2 and 3, “Synthesis…” and “Background…”; a relit foreground and relit background are generated based on the source image and lighting input, where the relit foreground and background depicts a foreground element and background element, respectively, each with the lighting condition); and
combining the relighted foreground image and the relighted background image to obtain a relighted image, wherein the relighted image depicts the foreground element and the background element with the lighting condition (figure 1; figure 3 and caption; abstract, “Finally, in the third stage, we resynthesize the background to be consistent with the lighting on the foreground object.”; section 5, “We composite the foreground on the background using the previously computed segmentation mask filtered with a 3 × 3 average filter to smooth the mask edges...Diffusion-based Inpainting. For all other lighting conditions, we use a pretrained diffusion-based inpainting model...We input the synthesized foreground image along with the (inverse) segmentation mask, as well as the original text prompt, to complete the foreground image with a consistent background.”; the relit foreground and background are combined to obtain a relighted image depicting the foreground and background element with the lighting condition).
Regarding claim 2, Zeng teaches the method of claim 1, further comprising: extracting a foreground image and a background image from the source image, wherein the relighted foreground image is based on the foreground image and the relighted background image is based on the background image (figure 3 and caption; section 3, subsections 1-3; section 4.1, “To estimate the shape of the foreground object, we first segment the foreground object from the provisional image using an off-the-shelf salient object detection network. Practically, we use U2Net...as it offers a good trade-off between speed and accuracy; we revert to SAM...for the rare cases where U2Net fails to provide a clean foreground segmentation. Next, we apply another off-the-shelf depth estimation network...on the segmented foreground object.”; section 5, second paragraph, “Diffusion…”).
Regarding claim 4, Zeng teaches the method of claim 1, wherein further comprising: generating a lighting image based on the lighting input, wherein the relighted foreground image is based on the lighting image (figure 3, radiance hints or generated foreground, and caption, “Next, we pass an appearance-seed, the provisional image, and a set of radiance hints (computed from the target lighting and a coarse estimate of the depth) to DiLightNet that will resynthesize the image such that becomes consistent with the target lighting while retaining the content of the provisional image.”; abstract, “[I]n the second stage, we resynthesize and refine the foreground object in the generated image by passing the target lighting to a refined diffusion model, named DiLightNet, using radiance hints computed on a coarse shape of the foreground object inferred from the provisional image. To retain the texture details, we multiply the radiance hints with a neural encoding of the provisional synthesized image before passing it to DiLightNet”; section 3, subsection 2).
Regarding claim 5, Zeng teaches the method of claim 1, wherein obtaining the lighting input comprises: identifying a sensory category; selecting a word based on the sensory category; and generating the lighting input to include the selected word (figure 1, “rusty steel toy frog with spatially varying materials with the body diffuse but shinny [sic] eyes” and caption, “Examples of generated images specified via a text-prompt (listed below each example) and with fine-grained lighting control. Each prompt is plausibly visualized under two different user-provided lighting environments”; section 4.3, the paragraph beginning “Empirically…”, the “Lighting Diversity” subsection, and the “Rendering” subsection, lines 1-8; a sensory category, such as the way the object looks or feels, is identified, a words indicating the lighting input are selected, such as “diffuse” or “shiny”).
Regarding claim 7, Zeng teaches the method of claim 1, wherein generating the relighted background image comprises: determining albedo information, depth information, and surface normal information based on the lighting input; and transferring the lighting condition from the lighting image to the background image based on the albedo information, the depth information, and the surface normal information (figure 3 and caption; section 2, “Relighting” subsection, lines 1-4; section 4.1, the paragraph beginning “To estimate…”; section 4.3, “Shape and Material Diversity” subsection, and the “Rendering” subsection).
Regarding claim 8, Zeng teaches the method of claim 7, wherein generating the relighted background image comprises: identifying a plurality of point lights based on the lighting input; and transferring the lighting condition from the plurality of point lights to the background image based on the albedo information, the depth information, and the surface normal information (figure 3 and caption; section 1, the last paragraph; section 4.3, “Lighting Diversity” subsection, parts 1, 2, and 5).
Regarding claim 9, Zeng teaches the method of claim 1, further comprising: creating a dataset including the relighted image; and training an image generation model using the dataset (section 4.3, the first paragraph, the “Shape and Material Diversity” subsection, and the “Training Dataset” subsection).
Regarding claim 18, the limitations of this claim substantially correspond to the limitations of claim 1 (except for the system, memory, and processor, which are inherent to a relighting rendering pipeline computational system such as that disclosed in Zeng); thus they are rejected on similar grounds.
Regarding claims 19 and 20, the limitations of these claims substantially correspond to the limitations of claims 2 and 4, respectively; thus they are rejected on similar grounds as their corresponding claims.
Regarding claim 21, the limitations of this claim substantially correspond to the limitations of claim 1 (except for the medium and processor, which are inherent to a relighting rendering pipeline computational system such as that disclosed in Zeng); thus they are rejected on similar grounds.
Regarding claims 22, 24, 25, 27, and 28, the limitations of these claims substantially correspond to the limitations of claims 2, 4, 5, 7, and 8, respectively; thus they are rejected on similar grounds as their corresponding claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Zeng, in view of LeGendre et al. (U.S. Patent Application Publication No. 2024/0212106), referred herein as LeGendre.
Regarding claim 3, Zeng teaches the method of claim 2, wherein: the foreground image depicts a plurality of versions of the foreground element under different lighting conditions (figure 2 and caption; figure 3, radiance hints; section 4.1, the last paragraph, “[W]e also encode the impact of different frequencies in the target lighting on the appearance of the foreground shape in separate radiance hints. Leveraging the fact that a BRDF acts as a band-pass filter on the incident lighting, we generate 4 radiance hints...We render the radiance hints, inclusive of shadows and indirect lighting, with Blender’s Cycles path tracer”; section 4.3, the “Rendering” subsection, lines 1-8).
Zeng does not explicitly teach that the lighting image [please see the 112(b) rejection above] comprises a panoramic image.
However, in a similar field of endeavor, LeGendre teaches a method for image generation comprising obtaining a source image and lighting input, generating a relit foreground image and background image with the lighting condition, and combining the relit foreground and background to obtain an output relighted image (figs 2 and 7; paragraphs 37 and 38; paragraph 49; paragraph 71), wherein the foreground may be relit based on a lighting image comprising a panoramic image (paragraphs 72 and 114).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the panoramic lighting image of LeGendre with the relighting of Zeng because this helps provide high fidelity features with a wider range of environmental lighting input, such that higher quality relit output images may be generated (see, for example, LeGendre, paragraph 46; paragraphs 114 and 118).
Regarding claim 23, the limitations of this claim substantially correspond to the limitations of claim 3; thus they are rejected on similar grounds.
Claims 6 and 26 are rejected under 35 U.S.C. 103 as being unpatentable over Zeng, in view of Park et al. (U.S. Patent Application Publication No. 2026/0010980), referred herein as Park.
Regarding claim 6, Zeng teaches the method of claim 1, comprising generating the relighted foreground image based on the lighting input (see the rejection of claim 1, above), but does not teach obtaining a noise map and denoising the noise map.
However, in a similar field of endeavor, Park teaches a method for image generation comprising obtaining a source image and lighting input, and generating an image with the lighting condition (figs 2, 3 and 8; paragraph 49; paragraphs 51 and 52; paragraph 63), and further comprising obtaining a noise map and denoising the noise map (fig 3; paragraphs 63 and 64).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the noise map and denoising of Park with the image generation of Zeng because this helps produce higher quality images without sacrificing processing efficiency (paragraphs 45 and 46; paragraph 63)
Regarding claim 26, the limitations of this claim substantially correspond to the limitations of claim 6; thus they are rejected on similar grounds.
Conclusion
The following prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Zirnheld (U.S. Patent No. 10,015,464); Method, apparatus and computer program product for modifying illumination in an image.
Lim (U.S. Patent Application Publication No. 2018/0293734); Visual anomaly detection system.
Truong (U.S. Patent No. 10,614,207); Generating captcha images using variations of the same object.
Sunkavalli (U.S. Patent Application Publication No. 2021/0065440); Dynamically estimating light-source-specific parameters for digital images using a neural network.
Legendre (U.S. Patent Application Publication No. 2022/0027659); Learning illumination from diverse portraits.
Neofytou (U.S. Patent Application Publication No. 2022/0284551); Relighting system for single images.
Lin (U.S. Patent Application Publication No. 2023/0325992); Recommending objects for image composition using geometry-and-lighting aware search and efficient user interface workflows.
Zhang (U.S. Patent Application Publication No. 2024/0005574); Applying object-aware style transfer to digital images.
Olszewski (U.S. Patent No. 12,315,075); Object-centric neural decomposition for image re-rendering.
Zhang (U.S. Patent Application Publication No. 2023/0325996); Generating composite images using user interface features for auto-compositing and composite-aware search.
Chen (U.S. Patent No. 12,579,615); Machine-learning models for generating a refined image.
Couleaud (U.S. Patent Application Publication No. 2025/0078347); Systems and methods for layered image generation.
Pinhasov (U.S. Patent Application Publication No. 2025/0095283); Image background effects.
Jawade (U.S. Patent Application Publication No. 2025/0315922); Systems and methods for image compositing via machine learning.
Jawade (U.S. Patent Application Publication No. 2025/0292368); Systems and methods for image compositing via machine learning.
He (U.S. Patent Application Publication No. 2026/0011074); Systems and methods for diffusion-based facial performance relighting.
Mei (U.S. Patent Application Publication No. 2026/0134527); Systems and methods for conditional video diffusion relighting.
Tewel (U.S. Patent Application Publication No. 2026/0120362); Text-to-image product placement.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID T WELCH whose telephone number is (571)270-5364. The examiner can normally be reached on Monday-Thursday, 8:30-5:30 EST, and alternate Fridays, 9:00-2:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached on 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
DAVID T. WELCH
Primary Examiner
Art Unit 2613
/DAVID T WELCH/Primary Examiner, Art Unit 2613
