DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
An attempt by the Office to electronically retrieve, under the priority document exchange program, the foreign application 23180383.4 to which priority is claimed has FAILED on 11/20/2024. Useful information is provided at the Electronic Priority Document Exchange (PDX) Program Website (https://www.uspto.gov/patents/basics/international-protection/electronic-priority-document-exchange-pdx), including practice tips for priority document exchange (https://www.uspto.gov/patents/basics/international-protection/electronic-priority-document-exchange-pdx#Practice%20tips).
Claim Objections
Claim 9 is objected to because of the following informalities:
In claim 9, line 2, “the one or more values of the derived GAN” should read “the one or more values of the one or more parameters of the derived GAN”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 6, 9, and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Ren et al. (Data-driven Digital Lighting Design for Residential Indoor Spaces, ACM Transactions on Graphics, Vol. 42, No. 3, Article 28. Publication date: March 2023, hereinafter “Ren”) in view of Cherepkov et al. (Navigating the GAN Parameter Space for Semantic Image Editing, arXiv.org, arXiv:2011.13786v3 [cs.LG] 21 Apr 2021, hereinafter “Cherepkov”).
Regarding claim 1, Ren discloses A computer implemented lighting appearance virtualization method, comprising: (page 28:1, col. left, para. 1, “The proposed framework utilizes neural networks to retrieve and learn underlying design guidelines and the principles beneath the existing lighting designs”; page 28:10, col. right, para. 2, “we train and test our framework on a PC with an Intel Xeon E5-2630 v3 CPU and an NVIDIA GeForce RTX 2080Ti GPU”).
receiving a user image of an area (page 28:1, Fig. 1: “Given an interior scene with furniture installed” and “Input 3D room”). Note that: the interior scene of the 3D room can be regarded as a user image. luminaire information of one or more luminaires including a luminaire (page 28:1, Fig. 1: “different types of lights in the
scenes”). Note that: different types of lights can be regarded as luminaire information. and light appearance information; (page 28:9, col. left, para. 5, “We use a network named IntensityNet to generate the coarse image. IntensityNet takes seven illumination per light category images and two illumination images for the environment light and
sunlight as input (Figure 3(e)), where each light illuminates the scene using a unit emitted radiance (i.e., cd/m2), and all illuminations of lights in the same category are summed to be one illumination per category image”). Note that: illumination is one of appearance of light or a lighting fixture.
generating, using a trained generative adversarial network (GAN), a first synthetic image of the area based on the user image and the luminaire information, wherein the first synthetic image shows the luminaire in the area; and (page 28:2. Fig. 3: “(a) Input”, “Scene with lights places”, and “Given (a) an indoor scene with the furniture as input, our pipeline first extracts (b) an image-based scene representation and then predicts (c) the light arrangement in the room. Once all lights have been placed, we use a path tracer to render (d) images with each light on and the others off. Using these illumination images as input, we generate (e) a lighting guidance image and optimize (f) the intensity and color of each light. Finally, we obtain (g) the resultant lighting layout and rendering results with pleasing lighting effects”, and “
PNG
media_image1.png
216
1148
media_image1.png
Greyscale
”; page 28:7, col. right, para. 2, “We propose a conditional generative adversarial network, named DownlightGAN (Figure 6 III), to predict the arrangement of all downlights at once rather than an iterative placement … The network input is the image-based scene representation. The target is a labeled image that contains the square mask of downlights and the labels of pixels INSIDE and OUTSIDE. We use GAN loss and feature matching loss as in Wang et al. [2018a]”; page 28:9, col. right, paras. 1-4, “we use GAN loss to enhance our image quality. Inspired by Bi et al. [2019], we use two discriminators to guide the predicted shading and refined lighting guidance … A latent code is added as input of both IntensityNet and ShadingRefineNet to control the lighting style. With the ability of BicycleGAN loss, the mapping of the latent code and
lighting style is ensured … we utilize CycleGAN [Zhu et al. 2017a], a classic image-to-image translation method for unpaired data”). Note that: (1) the “Rendering results” are a set of rendered images in which one of the images can be regarded as a first synthetic image; (2) a set of GANs and corresponding losses are used in the framework; (3) with the trained GAN model, the latent code space of the GAN model is spanned related to the user images, luminaire information, and light appearance information that are showed in the targeted or training scene images for the training process, and are treated as the classes in the latent code of the GAN model; (4) the trained generator can generate a rendered synthetic image based on input user image and luminaire information with the proper latent code; and (5) the first synthetic image as “Rendering results” includes and shows the luminaire in the scene area.
generating, using a derived GAN, a second synthetic image of the area based on the first synthetic image, wherein the second synthetic image of the area shows the luminaire and a synthetic light appearance associated with the luminaire in the area, wherein the light appearance information is related to (page 28:2. Fig. 3: “(a) Input”, “Scene with lights places”, and “Given (a) an indoor scene with the furniture as input, our pipeline first extracts (b) an image-based scene representation and then predicts (c) the light arrangement in the room. Once all lights have been placed, we use a path tracer to render (d) images with each light on and the others off. Using these illumination images as input, we generate (e) a lighting guidance image and optimize (f) the intensity and color of each light. Finally, we obtain (g) the resultant lighting layout and rendering results with pleasing lighting effects”, and “
PNG
media_image1.png
216
1148
media_image1.png
Greyscale
”). Note that: (1) the GAN-based framework can be retrained or tuned-up with more datasets and different training strategies to obtained a set of trained models with different model parameters; (2) one of the different models from the trained models can be regarded as a derived GAN model; (3) the derived model can take the first synthetic image and the proper latent code as the input to generate a second synthetic image as one of the rendered scene images in another “Rendering results”; and (4) the light appearance information can be regarded as one(s) of the classes, and is related to latent code of the derived GAN model.
However, Ren fails to disclose, but in the same art of computer graphics, Cherepkov discloses
… is related to one or more parameters of the trained GAN, wherein the trained GAN is modified to derive the derived GAN, wherein one or more values of one or more parameters of the derived GAN are different from one or more values of the one or more parameters of the trained GAN, wherein the one or more parameters of the trained GAN correspond to the one or more parameters of the derived GAN, and wherein the synthetic light appearance depends on the one or more values of the one or more parameters of the derived GAN. (Cherepkov, page 2, col. left, para. 4, “We propose to use the interpretable directions in the space of the generator parameters for semantic editing … remarkable visual effects can be achieved by slightly changing the GAN parameters”; page 3, col. left, paras. 4-5, “
PNG
media_image2.png
576
570
media_image2.png
Greyscale
”; page 4, col. left, Fig. 2, “The additive shift
PNG
media_image3.png
24
18
media_image3.png
Greyscale
is added to the convolutional kernel weight in the StyleGAN2 demodulation block”; page 5, Fig. 3: “Visual effects achieved by navigating the Style-GAN2 parameter space. Rows 1-3 correspond to the LSUN-Horse dataset and rows 4-6 correspond to the LSUNChurch dataset”, and “Belly size”, “Moving legs”, “Head size”, “Roof height”, and “Trees”). Note that: (1) the trained GAN model has the corresponding trained generator that generates a synthetic scene image with a latent code z; (2) the generator parameters along a set of learned vectors
PNG
media_image4.png
32
164
media_image4.png
Greyscale
can be changed or modified to correspond to the interpretable visual effects (e.g., “Belly size”, “Moving legs”, “Head size”, “Roof height”, and “Trees”), consistent for all latent codes. Therefore, without changing the latent code, the visual effects can be changed or updated by adding the additive shift
PNG
media_image3.png
24
18
media_image3.png
Greyscale
to the corresponding weights of the trained GAN model; (4) in this way the trained model is modified or derived to formulate the derived GAN model(s); (5) the parameters of the derived GAN model(s) are different from that of the trained GAN; (6) the parameters of the derived GAN model(s) correspond to that of the trained GAN; and (7) the visual effects (e.g., “Belly size”, “Moving legs”, “Head size”, “Roof height”, and “Trees”) can be substituted by the synthetic light appearance in lighting appearance virtualization.
Ren and Cherepkov are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply modifying the parameters of a trained GAN to achieving visual effects, as taught by Cherepkov into Ren. The motivation would have been “We propose to use the interpretable directions in the space of the generator parameters for semantic editing … remarkable visual effects can be achieved by slightly changing the GAN parameters” (Cherepkov, page 2, col. left, para. 4). The suggestion for doing so would allow to derive a set of GANs to change or enhance the light appearance in lighting appearance virtualization. Therefore, it would have been obvious to combine Ren and Cherepkov.
Regarding claim 2, Ren in view of Cherepkov discloses The computer implemented lighting appearance virtualization method of Claim 1, wherein the second synthetic image of the area is generated in response to receiving an approval of the first synthetic image of the area from a user. (Ren, page 28:2. Fig. 3: “
PNG
media_image1.png
216
1148
media_image1.png
Greyscale
”). Note that: (1) the derived model can take the first synthetic image and the proper latent code as the input to generate the second synthetic image as one of the rendered scene images in another “Rendering results”; and (2) it is obvious to one having ordinary skills in the art that the second synthetic image can be performed after a user approves the first synthetic image as input to the trained GAN model.
Regarding claim 3, Ren in view of Cherepkov discloses The computer implemented lighting appearance virtualization method of Claim 1, wherein the one or more values of the one or more parameters of the trained GAN are modified based on the light appearance information to derive the derived GAN from the trained GAN such that the synthetic light appearance corresponds to a desired light appearance indicated by the light appearance information. (Cherepkov, page 3, col. left, paras. 4-5, “
PNG
media_image2.png
576
570
media_image2.png
Greyscale
; page 5, col. left, para. 2,
PNG
media_image5.png
288
554
media_image5.png
Greyscale
). Note that: (1) the methods from sections 3.2-3.4 are used to produce a set of directions corresponding to the light appearance information; (2) a set of derived GAN models from the trained model can be generated by adding the additive shift
PNG
media_image3.png
24
18
media_image3.png
Greyscale
to the corresponding weights of the trained GAN model; and (3) a person can inspect manually the synthetic light appearance corresponding to a desired light appearance.
The motivation to combine Ren and Cherepkov given in claim 1 is incorporated here.
Regarding claim 6, Ren in view of Cherepkov discloses The computer implemented lighting appearance virtualization method of Claim 1, wherein one or more lighting artifacts are suppressed in the second synthetic image. (Ren, page 28:7, col. right, para. 3, “However, even with the generative adversarial network, some artifacts may exist; e.g., some downlights may not be exactly in a straight line. To handle these problematic cases, we align the downlights with heuristic lighting design guidelines … Specifically, we align downlights vertically and horizontally … Note that DownlightGAN predicts all downlights at once, which means there is no need to place downlights again for NextCategoryNet”). Note that: (1) lighting artifacts for downlights can appear in a GAN-generated synthetic image (e.g., the first synthetic image); and (2) with vertical and horizontal alignment of the downlights, DownlightGAN may generate an improved synthetic image (the second synthetic image) in which the downlights artifacts can be suppressed after the alignment.
Regarding claim 9, Ren in view of Cherepkov discloses The computer implemented lighting appearance virtualization method of Claim 1, wherein the one or more values of the derived GAN are set such that the synthetic light appearance shows a brightness level that is higher than the trained GAN is configured to generate in the first synthetic image of the area. (Cherepkov, page 3, col. left, paras. 4-5, “
PNG
media_image2.png
576
570
media_image2.png
Greyscale
; page 5, col. left, para. 2,
PNG
media_image5.png
288
554
media_image5.png
Greyscale
). Note that: (1) the methods from sections 3.2-3.4 are used to produce a set of directions corresponding to the light appearance with a brightness level; (2) a set of derived GAN models from the trained model can be generated by adding the additive shift
PNG
media_image3.png
24
18
media_image3.png
Greyscale
to the corresponding weights of the trained GAN model along the directions corresponding to brightness level; and (3) the generators of the set of derived GAN models can generate a set of synthetic scene images with various brightness levels, and a user can inspect manually the brightness level of the synthetic images and select one image showing a brightness level higher than the trained GAN configured to generate the first synthetic image. The derived models corresponding to the synthetic images with higher brightness are considered to be set.
The motivation to combine Ren and Cherepkov given in claim 1 is incorporated here.
Regarding claim 11, Ren in view of Cherepkov discloses The computer implemented lighting appearance virtualization method of Claim 1, wherein the one or more parameters of the derived GAN include one or more weights of a neural unit of a convolutional layer of the derived GAN. (Cherepkov, page 4, col. right, Figure 2:
PNG
media_image6.png
236
554
media_image6.png
Greyscale
). Note that: the additive shift
PNG
media_image7.png
24
14
media_image7.png
Greyscale
is added to the weight parameters of neuron unit of a Conv layer of the trained GAN to formulate the derived GAN.
The motivation to combine Ren and Cherepkov given in claim 1 is incorporated here.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Ren in view of Cherepkov, and further in view of Edmund (Laser Beam Shaping Overview, archive.org, https://web.archive.org/web/20230309201922/https://www.edmundoptics.com/knowledge-center/application-notes/optics/laser-beam-shaping-overview/, hereinafter “Edmund”) and Unity (Micro Shadows, https://docs.unity3d.com/Packages/com.unity.render-pipelines.high-definition@12.1/manual/Override-Micro-Shadows.html, hereinafter “Unity”).
Regarding claim 4, Ren in view of Cherepkov discloses The computer implemented lighting appearance virtualization method of Claim 3, wherein the light appearance information indicates at least one of a light brightness level, a correlated color temperature, a color, a beam size, a polarization, a beam shape, a micro-shadow level, and an edge sharpness level. (Ren, page 28:2, col. left, para. 5, “Annotations of the lights, such as different light type, intensity, and color and emission surfaces, are all labeled and stored in these scenes”; page 28:9, col. left, para 5, “the intensity and color temperature of each light”). Note that: a light brightness level as intensity, color, and color temperature are described as the light appearance information.
However, Ren in view of Cherepkov fails to disclose, but in the same art of light application, Edmund discloses a beam size … a polarization … a beam shape … an edge sharpness level … (Edmund, page 1, para. 1, “A laser beam shape is typically defined by its irradiance distribution and phase”; page 4, para. 3, “The relationship between divergence and beam size is described in our Gaussian Beam Propagation application note”; page 3, para 1, “Clean cuts with sharp edges can be generated in the DOF because of the uniform beam diameter”; page 4, para. 5, “it is useful for one prism to be oriented at Brewster’s angle, or the angle of incidence where there is no reflection of p-polarized light”). Note that: beam shape, beam size, sharp edges (edge sharpness), p-polarized light (a polarization), are described and can be included in the light appearance information for a laser light.
Ren in view of Cherepkov, and Edmund, are in the same field of endeavor, namely light application. Before the effective filing date of the claimed invention, it would have been obvious to apply the light properties, as taught by Edmund into Ren in view of Cherepkov. The motivation would have been “A laser beam shape is typically defined by its irradiance distribution and phase. The latter is essential in determining the uniformity of a beam profile over its propagation distance” (Edmund, page 1, para. 1). The suggestion for doing so would allow to add the light properties to the light appearance information. Therefore, it would have been obvious to combine Ren, Cherepkov, and Edmund.
However, the combination of Ren, Cherepkov, and Edmund fails to disclose, but Unity in the same art of computer graphics, Unity discloses … a micro-shadow level … (Unity, page 1, para 1, “Micro shadows are shadows that the High Definition Render Pipeline (HDRP) simulates for small details”). Note that: micro-shadow is described and can be included in the light appearance information.
The combination of Ren, Cherepkov, and Edmund, and Unity, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply micro shadows, as taught by Unity into the combination of Ren, Cherepkov, and Edmund. The motivation would have been “Micro shadows are shadows that the High Definition Render Pipeline (HDRP) simulates for small details” (Unity, page 1, para 1). The suggestion for doing so would allow to add the micro-shadow effects to the light appearance information. Therefore, it would have been obvious to combine Ren, Cherepkov, Edmund, and Unity.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Ren in view of Cherepkov, and further in view of Xia et al. (GAN Inversion: A Survey, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 45, NO. 3, MARCH 2023, hereinafter “Xia”).
Regarding claim 10, Ren in view of Cherepkov fails to disclose, but in the same art of computer graphics, Xia discloses determining an input noise vector by performing a GAN inversion based on an input image of the area and the trained GAN, wherein the input image shows the luminaire in the area, wherein the input image of the area is generated from the user image of the area and the luminaire information of the luminaire, and wherein the first synthetic image is generated using the input noise vector as an input of the trained GAN. (Xia, page 3121, col. right, para. 2, “GAN inversion aims to invert a given image back into the latent space of a pretrained GAN model. The image can then be faithfully reconstructed from the inverted code by the generator”; page 3122 / col. left / para. 3 – page 3122 / col. right / para. 1, “
PNG
media_image8.png
66
510
media_image8.png
Greyscale
PNG
media_image9.png
438
518
media_image9.png
Greyscale
”). Note that: (1) the first synthetic image can be regarded as an input image. By solving the GAN inversion using the optimization method above and the trained GAN, the latent code (a Gaussian distributed noise vector) for the input image can be determined; (2) with the noise vector as the input noise vector, the first synthetic image can be reconstructed faithfully using the trained GAN; (3) since the first synthetic image shows the luminaire, the input image identical to the synthetic image also shows the luminaire; and (4) the first synthetic image is generated with the user image of the area and the luminaire information of the luminaire through the trained GAN.
Ren in view of Cherepkov, and Xia, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply GAN inversion, as taught by Xia into Ren in view of Cherepkov. The motivation would have been “GAN inversion aims to invert a given image back into the latent space of a pretrained GAN model. The image can then be faithfully reconstructed from the inverted code by the generator” (Xia, page 3121, col. right, para. 2). The suggestion for doing so would allow to determine a noise vector by performing a GAN inversion based on an input image of the area and the trained GAN. Therefore, it would have been obvious to combine Ren, Cherepkov, and Xia.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Ren in view of Cherepkov, and further in view of Karras et al. (A Style-Based Generator Architecture for Generative Adversarial Networks, arXiv.org, arXiv:1812.04948v3 [cs.NE] 29 Mar 2019, hereinafter “Karras”).
Regarding claim 12, Ren in view of Cherepkov discloses The computer implemented lighting appearance virtualization method of Claim 1, wherein the one or more parameters of the derived GAN
However, Ren in view of Cherepkov fails to disclose, but in the same art of computer graphics, Karras discloses include one or more input parameters provided to one or more adaptive instance normalization (AdaIN) layers of the derived GAN or to one or more affine transformation layers of the derived GAN, wherein the one or more AdaIN layers each have an output that is provided to a respective convolutional layer of the derived GAN. (Karras, page 2, col. left, “
PNG
media_image10.png
700
568
media_image10.png
Greyscale
”). Note that: the derived GAN’s latent space can be mapped into an intermediate latent space with one or more input parameters
PNG
media_image11.png
22
68
media_image11.png
Greyscale
as one or more input parameters provided to one or more adaptive instance normalization (AdaIN) layers of the derived GAN through “A” (a learned affine transform), and one or more AdaIN layers each have an output that is provided to a respective convolutional layer (e.g., Conv 3x3 layers) of the derived GAN.
Allowable Subject Matter
Claims 5, 7-8, and 13-15 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding dependent claim 5, in the context of claim as a whole, the prior art either alone or in combination does not teach or suggest the additional elements of: “wherein the trained GAN is a first trained GAN of the multiple trained GANs that is trained using first training images that include a first light appearance and that exclude a second light appearance and wherein a second trained GAN of the multiple
trained GANs is trained using second training images that include the second light
appearance and that exclude the first light appearance.”
Claim 7 depends from claim 5.
Regarding dependent claim 8, in the context of claim as a whole, the prior art either alone or in combination does not teach or suggest the additional elements of: “the trained GAN is trained such that the synthetic light appearance in the second synthetic image of the area depends on at least one of a type of the luminaire and a location of the luminaire in the area as shown in the second synthetic image of the area.”
Regarding dependent claim 13, in the context of claim as a whole, the prior art either alone or in combination does not teach or suggest the additional elements of: “receiving luminaire information of a second luminaire, wherein the first synthetic image of the area is generated further based on the luminaire information of the second luminaire such that the first synthetic image shows the luminaire and the second luminaire in the area and wherein the second synthetic image of the area shows the luminaire, the second luminaire, the synthetic light appearance associated with the luminaire, and a second synthetic light appearance associated with the second luminaire.”
Regarding dependent claim 14, in the context of claim as a whole, the prior art either alone or in combination does not teach or suggest the additional elements of: “generating a combined synthetic image of the area that includes a portion of the second synthetic image that includes the luminaire and the synthetic light appearance and a portion of the third synthetic image that includes second luminaire and the second synthetic light appearance.”
Regarding dependent claim 15, in the context of claim as a whole, the prior art either alone or in combination does not teach or suggest the additional elements of: “generating a combined synthetic image of the area that includes a portion of the second synthetic image that includes the luminaire and the synthetic light appearance and a portion of the fourth synthetic image that includes second luminaire and the second synthetic light appearance.”
Response to Arguments
Applicant's arguments filed 08/06/2025 have been fully considered but they are not persuasive.
Applicant alleges, “A Notice of Allowance is respectfully requested.” (page 3, line 2). However, Examiner respectfully disagrees about the respective allegations as whole because:
The independent claim 1 is rejected under 35 U.S.C. 103 as being unpatentable over Ren in view of Cherepkov. Please see the corresponding prior art and citations above.
All non-objected dependent claims of claim 1 are rejected under 35 U.S.C. 103, respectively. Please see the corresponding prior art and citations above.
The arguments are not persuasive.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BIAO CHEN whose telephone number is (703)756-1199. The examiner can normally be reached M-F 8am-5pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee M Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Biao Chen/
Patent Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611