DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 20 July 2026 has been entered.
Response to Arguments
Applicant's arguments filed 20 July 2026 have been fully considered but they are not persuasive.
Applicant argues “the lighting described in Lombardi is referring to the lighting during image capture of the subject. By contrast, the claims clarify that the compressed lighting representation corresponds to lighting characteristics for a different environment than the physical environment in which the subject is captured” (Remarks, pg. 8). The Examiner respectfully disagrees. Lombardi discloses a multi-camera setup for capturing many viewpoints and lighting configurations of a subject. This is for the purpose of generating training data to train a machine-learning rendering model (“receive geometry information and a plurality of images … to train a model,” Lombardi, para. 50; “collect a large amount of data of the subject,” Lombardi, para. 60; “the light conditions may change dynamically throughout the capture operations,” Lombardi, para. 62).
Using a broadest reasonable interpretation, the claims merely require capturing a physical scene having second lighting characteristics and rendering an avatar using first lighting characteristics different than the second lighting characteristics. Since the physical scene is captured using many lighting characteristics, this would include capturing first and second lighting characteristics. Even if one assumes, for the sake of argument, that Lombardi can only reproduce lighting characteristics that were physically captured with the subject, Lombardi could render the avatar with the captured first lighting characteristics that are different from the captured second lighting characteristics, which would teach the claim limitations. While the claim requires the second lighting characteristics to be lighting characteristics of a captured physical scene, nothing in the claim precludes the first lighting characteristics from also being lighting characteristics of a captured physical scene.
Even if the claim is interpreted narrowly or amended such that the subject is never captured using the first lighting characteristics, Lombardi still teaches this concept: “enable relighting” (Lombardi, para. 43). The Rafii reference provides a reasonable definition of relighting (paras. 111-112 of Rafii describe how “relighting” allows a new lighting environment other than one in which an object is captured to be used in rendering an object). The Rafii reference is not used in the rejection of claim 1, but the Examiner is showing how one having ordinary skill in the art would interpret the word “relighting” when reading Lombardi.
Applicant argues “Lombardi is light on discussion about how the network could be used to obtain a texture map based on lighting, and is silent regarding any lighting representation being used as an input into the trained model” (Remarks, pg. 8). The Examiner respectfully disagrees. Lombardi discloses “The autoencoder may be a conditional autoencoder” (para. 5) and “condition the network on illumination” (para. 39), which means using illumination as an input to the network to infer an output based on that chosen illumination.
Applicant argues “nothing in Sunkanvalli indicates that those parameters are applied to a network in conjunction with a geometry, to generate a texture of a subject” (Remarks, pg. 8). The Examiner does not rely on Sunkavalli to teach these features. Sunkavalli is merely relied on to teach that, when relighting an object, a compressed representation of lighting corresponding to an environment map of a desired scene can be used. When applied to Lombardi, the modified Lombardi would render obvious using a compressed representation of lighting representing an environment map of a desired scene when conditioning the autoencoder to generate a shaded texture output.
Applicant argues “Sunkanvalli is concerned with lighting at a particular location within a scene, and not selection of a lighting representation for a scene itself” (Remarks, pg. 8). The Examiner respectfully disagrees. Sunkanvalli is directed to an Augmented Reality application where a user can place virtual objects into a scene, and the application renders the virtual objects with the scene lighting. Since a user can physically change scene lighting, such as by physically moving lights, a virtual object would be lit based on lighting characteristics that were not previously captured while training the model. Thus, a virtual object would be relit using new lighting characteristics selected by a user.
Any remaining arguments are considered moot based on the foregoing.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-11, 13-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Lombardi et al. (US 2019/0213772; hereinafter “Lombardi”) in view of Sunkavalli et al. (US 2020/0302684; hereinafter “Sunkavalli”).
Regarding claim 1, Lombardi discloses A non-transitory computer readable medium comprising computer readable instructions executable by one or more processors (“non-transitory-type media,” para. 94) to: obtain a request to represent an avatar of a subject (“render avatar geometry and texture,” para. 65), wherein the request to represent the avatar identifies a requested scene (“provide photorealistic rendering of dynamic and socially interactive scenes,” para. 2); obtain a lighting representation for the requested scene, wherein the lighting representation encodes first lighting characteristics (“prescribe a particular lighting environment when rendering the avatar,” para. 62; “enable relighting,” para. 43); obtain a geometric representation of the subject from sensor data of the subject in a physical environment (“To provide the tracked three-dimensional mesh, the building engine may use images captured from the multi-camera setup,” para. 64) wherein the physical environment comprises second lighting characteristics different than the first lighting characteristics (“Light sources may provide various light conditions during the capture operations,” para. 61; “enable relighting,” para. 43); apply the lighting representation and the geometric representation of the subject to a network to obtain a texture map corresponding to the avatar of the subject (“condition the network on illumination,” para. 39; “use the dynamic light conditions as input in the model, which may allow the systems described herein to prescribe a particular lighting environment when rendering the avatar,” para. 62; “encode … the geometry information,” para. 52; “use the latent vector to infer … an inferred view-dependent texture of the subject,” para. 52;); cause the avatar of the subject to be rendered in a view of the requested scene based on the texture map and the geometric representation of the subject (“render reconstructed images of the subject to provide an avatar, for example, using the inferred geometry and the inferred view-dependent texture,” para. 50).
Lombardi does not disclose using a compressed lighting representation or wherein the compressed lighting representation encodes first lighting characteristics represented in an environment map of the requested scene.
In the same art of graphics rendering, Sunkavalli teaches using a compressed lighting representation, wherein the compressed lighting representation encodes first lighting characteristics represented in an environment map of the requested scene (“When generating location-specific-lighting parameters, the lighting estimation system can generate spherical-harmonic coefficients that indicate lighting conditions,” para. 30; lighting depictions 306a-306e for a full-environment map 304 of a position within a digital scene. The lighting depictions 306a, 306b, 306c, 306d, and 306e respectively correspond to spherical-harmonic coefficients,” para. 61).
Before the effective filing date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Sunkavalli to Lombardi. The motivation would have been “to realistically depict changes in lighting” (Sunkavalli, para. 30).
Regarding claim 2, the combination of Lombardi and Sunkavalli renders obvious wherein the compressed representation comprises a compressed representation of lighting and color in the requested scene (“renders pixels for the virtual object that reflect lighting, shading, or appropriate color hues indicated by the location-specific-lighting parameters,” Sunkavalli, para. 47; see claim 1 for motivation to combine).
Regarding claim 3, the combination of Lombardi and Sunkavalli renders obvious wherein the compressed lighting representation corresponds to at least one selected from a group consisting of spherical harmonic coefficients, spherical gaussians, and spherical wavelets for the requested scene (“When generating location-specific-lighting parameters, the lighting estimation system can generate spherical-harmonic coefficients that indicate lighting conditions,” Sunkavalli, para. 30; see claim 1 for motivation to combine).
Regarding claim 4, the combination of Lombardi and Sunkavalli renders obvious obtain one or more images of the subject (“record the subject from a plurality of cameras,” Lombardi, para. 60); and obtain a latent representation of a geometry of the subject based on the one or more images (“encode … the geometry information,” Lombardi, para. 52).
Regarding claim 6, the combination of Lombardi and Sunkavalli renders obvious wherein the avatar of the subject is further rendered based on a head pose of the subject (“tracked head pose,” Lombardi, para. 45; “a viewer's point of view, relative to the position and orientation of the avatar,” Lombardi, para. 37).
Regarding claim 7, the combination of Lombardi and Sunkavalli renders obvious wherein the avatar is rendered by applying the texture map to the geometric representation of the subject (“use the reconstructed texture maps and the reconstructed three-dimensional mesh of the subject to render a reconstructed image of the subject, thus providing a data-driven avatar of the subject,” Lombardi, para. 43).
Regarding claims 8-11, 13, and 14, they are rejected using the same citations and rationales described in the rejections of claims 1-4, 6, and 7, respectively.
Regarding claims 15-18 and 20, they are rejected using the same citations and rationales described in the rejections of claims 1-4 and 6, respectively.
Claims 5, 12, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Lombardi and Sunkavalli, and further in view of Rafii et al. (US 2018/0114264; hereinafter “Rafii”).
Regarding claim 5, the combination of Lombardi and Sunkavalli does not disclose wherein the requested scene is selected by a user.
In the same art of relighting objects in computer graphics scenes, Rafii teaches wherein the requested scene is selected by a user (“proper global relighting of the object,” para. 49; “multiple potential scenes may be automatically generated by the system, and the user may select one or more of these scenes,” para. 86).
Before the effective filing date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Rafii to the combination of Lombardi and Sunkavalli. The motivation would have been “for a more accurate depiction of these objects in the 3D environments” (Rafii, para. 110).
Regarding claims 12 and 19, they are rejected using the same citations and rationales described in the rejection of claim 5.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ryan McCulley whose telephone number is (571)270-3754. The examiner can normally be reached Monday through Friday, 8:00am - 4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached on (571) 272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN MCCULLEY/Primary Examiner, Art Unit 2611
Prosecution Insights