DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 8-9, 13, 15-16, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nguyen et al. (US 11308669) in view of Fidler et al. (US 20240054720).
Regarding claim 1, Nguyen discloses A computer-implemented method comprising: generating a source light representation of a real-world scene from an image of the real-world scene (Nguyen, “(32) the feature extractor 210 extracts one or more appearance features 214 (“appearance features 214”, hereinafter for the sake of brevity) from the images 204. (36) In some implementations, the appearance features 214 include one or more lighting condition values 234 that indicate physical lighting conditions in a physical environment where the images 204 were generated when the images 204 were generated (e.g., captured or hand-drawn)”);
augmenting the real-world scene in an object representation of the real-world scene to generate an augmented scene (Nguyen, “(26) the CGR environment 106 includes an augmented environment that is a modified version of a physical environment. For example, in some implementations, the controller 102 and/or the electronic device 103 modify (e.g., augment) the physical environment where the electronic device 103 is located in order to generate the CGR environment 106”);
selecting a target lighting representation identifying a target light source (Nguyen, “(54) the CGR lighting conditions 274 include a simulated ambient lighting level that is determined based on the ambient lighting level in the physical environment (e.g., the simulated ambient lighting level is set to within a degree of similarity of the ambient lighting level in the physical environment). (60) an intensity 304 of the simulated light source 300 is selected based on the CGR shadow size value 268”);
processing the augmented scene to generate a plurality of augmented image buffers (Nguyen, “(32) In various implementations, the feature extractor 210 extracts one or more appearance features 214 (“appearance features 214”, hereinafter for the sake of brevity) from the images 204. The appearance features 214 define an appearance of one or more objects represented in the images 204.”);
processing, by a neural deferred rendering model, the plurality of augmented image buffers, the source lighting representation, and the target lighting representation to generate an augmented image having a lighting appearance according to the target light source (Nguyen, fig.2&3D, “(55) the CGR object renderer 250 includes a deferred neural renderer that takes the feature map 212 as an input, and generates the CGR shading effects 254 as outputs. In some implementations, the CGR object renderer 250 includes an object-specific deferred neural renderer that generates the CGR shading effects 254 for CGR objects that are within a degree of similarity to (e.g., similar to) an object that the object-specific deferred neural renderer corresponds to. In some implementations, the CGR object renderer 250 includes a generalized deferred neural renderer that generates the CGR shading effects 254 for various CGR objects (e.g., for any CGR object, for example, for CGR objects that are beyond a degree of similarity to objects represented by the images 204). (69) As represented by block 415, in some implementations, the method 400 includes varying a simulated light source of a CGR environment in which the CGR object is displayed in order to generate the one or more simulated shading effects. as illustrated in FIG. 2, in some implementations, the CGR object renderer 250 varies one or more operating characteristics of the simulated light source in accordance with the CGR lighting conditions 274”. Therefore, the features in the feature map corresponds to the augmented image buffers, lighting condition values in the feature map corresponds to the source lighting representation, and the simulated light source corresponds to the target lighting representation); and
outputting the augmented image (Nguyen, “(64) as shown in FIG. 2, the CGR object renderer 250 displays the CGR object 252 with the CGR shading effects 254 based on the feature map 212. Moreover, as described in relation with FIG. 2, the CGR shading effects 254 are within a degree of similarity to the shading effects 206. (88) In some implementations, the one or more I/O devices 510 include a display for displaying a computer-generated graphical environment (e.g., the CGR environment 106 shown in FIG. 1)”).
On the other hand, Nguyen fails to explicitly disclose but Fidler discloses generating a source light representation of a real-world scene from a panoramic image of the real-world scene (Fidler, “[0039] The first image may be an LDR panorama of an outdoor scene, where the image may depict, at least in part, features associated with a sky. A sky encoder may be used to generate, from the first image, a sky vector 504. The sky vector may represent information such as sky latency, peak direction, and/or peak intensity. These features may then be used to generate an HDR sky dome 506, for example using a sky decoder. The HDR sky dome may be representative of an environmental lighting model for an outdoor scene”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined Fidler and Nguyen, to include all limitations of claim 1. That is, adding the HDR sky dome generating based on the panoramic image of Fidler to the method of Nguyen. The motivation/ suggestion would have been to provide improved realism over a generic lighting volume (Fidler, [0022]).
Regarding claim 15, it recites similar limitations as claim 1, except that it further recites “A system comprising: memory; and a computer processor comprising computer readable program code for performing operations”.
Regarding claim 20, it recites similar limitations as claim 1, except that it further recites “A non-transitory computer readable medium comprising computer readable program code for performing operations”.
Regarding claims 15, 20, Nguyen further discloses “(9) In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs. In some implementations, the one or more programs are stored in the non-transitory memory and are executed by the one or more processors. In some implementations, the one or more programs include instructions for performing or causing performance of any of the methods described herein. In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions that, when executed by one or more processors of a device, cause the device to perform or cause performance of any of the methods described herein. In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and means for performing or causing performance of any of the methods described herein”.
Regarding claim 2, Nguyen in view of Fidler discloses The computer-implemented method of claim 1.
Nguyen further discloses wherein the neural deferred rendering model further uses the real-world scene to generate the augmented image (Nguyen, “(26) the CGR environment 106 includes an augmented environment that is a modified version of a physical environment… the controller 102 and/or the electronic device 103 modify (e.g., augment) the physical environment where the electronic device 103 is located in order to generate the CGR environment 106. (55) the CGR object renderer 250 utilizes deferred neural rendering to generate the CGR shading effects 254. (65) Referring to FIG. 4B, as represented by block 408, in some implementations, displaying the CGR object includes performing deferred neural rendering in order to display the CGR object”).
Regarding claim(s) 16, it is interpreted and rejected for the same reasons set forth in claim(s) 2.
Regarding claim 8, Nguyen in view of Fidler discloses The computer-implemented method of claim 1.
On the other hand, Nguyen fails to explicitly disclose but Fidler discloses wherein generating the augmented scene comprises at least one selected from a group consisting of moving an actor in the augmented scene from the real-world scene, adding the actor to the augmented scene as compared to the real-world scene, and removing the actor from the augmented scene as compared to the real-world scene (Fidler, “[0031] the virtual object 402 may come from a database of objects, such as a set of CAD models that have been generated for use with the system. By way of example and not limitation, the models may include cars, buildings, plants, animals, and/or the like. [0040] In at least one embodiment, a second image is received 508, where the second image may correspond to an outdoor scene. As will be described, the second image may be an image that will be used for rendering and insertion of one or more virtual objects. [0042] virtual objects may be added to scenes, which may be helpful for renderings to illustrate how items may look or to generate training data to train one or more machine learning systems.”). The same motivation of claim 1 applies here.
Regarding claim 9, Nguyen in view of Fidler discloses The computer-implemented method of claim 1.
On the other hand, Nguyen fails to explicitly disclose but Fidler discloses wherein generating the source lighting representation is performed using a lighting estimator estimating a sky dome from the panoramic image, wherein the source lighting representation is the sky dome (Fidler, fig.2A, “[0021] In various embodiments, the input 102 includes at least one LDR panoramic image that is used by an HDR sky model 106. The HDR sky model 106 is used to estimate lighting caused by the extreme intensity (e.g., intensity several orders of magnitude greater than other lighting sources) found in outdoor scenes. The HDR sky model 106 may be used to model the sky dome that can be used alongside a lighting volume. [0025] noted, the HDR sky dome may be used to model the sunlight or other sources of light that may be several orders of magnitude greater than other light sources within a scene… the HDR sky dome 216 may be provided to model or otherwise represent a light source within an outdoor scene”). The same motivation of claim 1 applies here.
Regarding claim 13, Nguyen in view of Fidler discloses The computer-implemented method of claim 1.
Nguyen further discloses wherein processing the augmented scene to generate the plurality of augmented image buffers is performed by a physics-based renderer (Nguyen, fig.5, “(86) the memory 504 or the non-transitory computer readable storage medium of the memory 504 stores the following programs, modules and data structures, or a subset thereof including an optional operating system 506, the data obtainer 202, the feature extractor 210 and the CGR object renderer 250. (87) the feature extractor 210 includes instructions 210a, and heuristics and metadata 210b”).
Claim(s) 3, 4 , 17, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nguyen et al. (US 11308669) in view of Fidler et al. (US 20240054720), and further in view of Gausebeck et al. (US 12513404).
Regarding claim 3, Nguyen in view of Fidler discloses The computer-implemented method of claim 1.
On the other hand, Nguyen in view of Fidler fails to explicitly disclose but Gausebeck discloses obtaining a plurality of real-world images in the real-world scene; and generating the panoramic image from the plurality of real-world images (Gausebeck, “(67) The images captured by the lens assembly 704 may be stitched together to form a 2D panoramic image of the physical environment. A 3D panoramic may be generated by combining the depth data captured by the lidar 708 with the 2D panoramic image generated by stitching together multiple images from the lens assembly 704”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined Gausebeck into the combination of Fidler and Nguyen, to include all limitations of claim 3. That is, adding the generating the 3D panoramic image of Gausebeck to generate the panoramic image of Nguyen and Fidler. The motivation/ suggestion would have been to assist the user in taking multiple images in positions that improve the quality, time efficiency, and effectiveness of image stitching for the desired panorama (Gausebeck, (147)).
Regarding claim 4, Nguyen in view of Fidler and Gausebeck discloses The computer-implemented method of claim 3.
On the other hand, Nguyen in view of Fidler fails to explicitly disclose but Gausebeck discloses for each image pixel of a first plurality of image pixels in a real-world image of the plurality of real-world images: mapping the image pixel to a location in the object representation of the real-world scene, determining a distance to the object representation, and transforming the image pixel to a corresponding panoramic pixel at a corresponding position in the panoramic image (Gausebeck, “(52) the depth data may be associated with coordinates about the environmental capture system 400. Similarly, pixels or parts of images may be associated with the coordinates about the environmental capture system 400 to enable the creation of the 3D visualization (e.g., an image from different directions, a 3D walkthrough, or the like) to be generated using the images and the depth data. Claim 8, generating, by the LiDAR device, the depth information based on the corresponding plurality of reflected laser pulses, wherein the first, second, and third blended images of the first FOV, the second FOV, and the third FOV each comprise a plurality of pixels, wherein each of the plurality of pixels is associated with numerical coordinates that identify a location of the pixel in the environment, wherein each of the corresponding plurality of reflected laser pulses is associated with corresponding numerical coordinates that identify a location of the depth information, and wherein combining the depth information for the environment with the panoramic image of the environment to create the 3D panoramic image of the environment comprises combining the depth information at a location in the environment with the pixel at the location in the environment to create the 3D panoramic image of the environment”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined Gausebeck into the combination of Fidler and Nguyen, to include all limitations of claim 4. That is, adding the generating the 3D panoramic image of Gausebeck to generate the panoramic image of Nguyen and Fidler. The motivation/ suggestion would have been to assist the user in taking multiple images in positions that improve the quality, time efficiency, and effectiveness of image stitching for the desired panorama (Gausebeck, (147)).
Regarding claim(s) 17-18, they are interpreted and rejected for the same reasons set forth in claim(s) 3-4, respectively.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nguyen et al. (US 11308669) in view of Fidler et al. (US 20240054720), and further in view of Gausebeck et al. (US 12513404) and Baldwin (US 9036943).
Regarding claim 7, Nguyen in view of Fidler and Gausebeck discloses The computer-implemented method of claim 4.
On the other hand, Nguyen in view of Fidler and Gausebeck fails to explicitly disclose but Baldwin discloses detecting that a second plurality of pixels of the plurality of real-world images are mapped to a same corresponding position in the panoramic image, generating a combined color value for the corresponding panoramic pixel at the same corresponding position from the second plurality of pixels, and wherein transforming the image pixel for the second plurality of pixels comprises using the combined color value for the corresponding panoramic pixel (Baldwin, “(60) Additionally, in at least some instances, the panoramic constitute images overlap 460. In this instance, in response to detecting at least a portion of one of the plurality of panoramic constituent images (i.e., 430) overlapping an adjacent portion of one of the plurality of panoramic constituent images (i.e., 431) the overlapped areas (i.e., 460) of the images can be stacked, aligned and combined, as described above, such that the pixel values for corresponding locations of the overlapped images can be statistically combined”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined Baldwin into the combination of Fidler and Nguyen, Gausebeck, to include all limitations of claim 7. That is, applying the combining pixels when overlapping of Baldwin to the method of Gausebeck, Nguyen and Fidler. The motivation/ suggestion would have been Images are blended together and seam line adjustment is done to minimize the visibility of seams between images (Baldwin, (59)).
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nguyen et al. (US 11308669) in view of Fidler et al. (US 20240054720), and further in view of Sunkavalli et al. (US 20200151509)
Regarding claim 10, Nguyen in view of Fidler discloses The computer-implemented method of claim 9.
On the other hand, Nguyen in view of Fidler fails to explicitly disclose but Sunkavalli discloses downscaling a plurality of high-dynamic range training panoramic images to generate a plurality of low-dynamic range training panoramic images (Sunkavalli, “[0025] More particularly, the illumination estimation system trains the neural network using a set of LDR panoramic images. For example, the system can use a set of outdoor LDR panoramic images generated from a corresponding set of outdoor HDR panoramic images. [0054] To leverage the known HDR parameters of a synthetic HDR image in training a neural network system, a HDR outdoor panorama scene can be turned into a LDR image of the scene. [0067] Turning now to FIG. 3, FIG. 3 provides a process flow showing an embodiment of method 300 for training a neural network system to estimate HDR lighting parameters from a LDR image [0069] At block 304 the type of training image can be determined. A training LDR image can either be a synthetic LDR image or a real LDR image. A synthetic LDR image is generated from a synthetic HDR image (e.g., generated using a real HDR image of the sky and rendered outdoor panoramic scene).”);
processing, by the lighting estimator, the plurality of low-dynamic range training panoramic images to generate a plurality of predicted sky (Sunkavalli, “[0032] These HDR images can then be cropped and used to train a further neural network system to predict HDR parameters for limited-field-of-view LDR images. [0051] Training engine 206 may be used to train neural network system 214 to estimate HDR illumination of a LDR panorama image. [0063] The output HDR lighting parameters can be used to generate an estimated render image. [0072] At block 310, a training estimated render image can be rendered using the HDR lighting parameters estimated by the neural network system. [0073] The pre-rendered image for environmental light and sun light can be rendered into the training estimated render image”);
generating a loss based on a comparison of the plurality of high-dynamic range training panoramic images to the plurality of predicted sky domes; and updating the lighting estimator according to the loss (Sunkavalli, “[0025] The illumination estimation system uses the set of outdoor HDR panoramic images to identify ground-truth lighting parameters for each of the outdoor LDR panoramic images. [0055] errors in the present neural network system can be determined using a pixel-wise comparison between a ground-truth IBL render image created using the known lighting parameters from a synthetic HDR image and an estimated render image created using the estimated lighting parameters determined by the neural network for an input training LDR image. [0077] At block 316, loss determined at 310 and/or loss determined at steps 312-314 can be used to adjust the neural network. It should be appreciated additional types of loss can also be used to correct for errors in the network.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined Sunkavalli into the combination of Fidler and Nguyen, to include all limitations of claim 10. That is, modifying the estimated render images of the sky of Sunkavalli to sky domes, and then applying the adjusting neural network for estimating lighting parameters of Sunkavalli to the method of Nguyen and Fidler. The motivation/ suggestion would have been to accurately illuminate composite objects or an entire scene for scene reconstruction (Sunkavalli, [0003]).
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nguyen et al. (US 11308669) in view of Fidler et al. (US 20240054720), and further in view of Spivack et al. (US 20190108686)
Regarding claim 14, Nguyen in view of Fidler discloses The computer-implemented method of claim 1.
On the other hand, Nguyen in view of Fidler fails to explicitly disclose but Spivack discloses training a virtual driver of an autonomous system using the augmented image (Spivack, “[0655] Users can see, train and interact with very realistic or visual intelligent avatars in the AR environment. [0665] The tech used for AR Pets can also be used to: [0666] Power NPCs in the AR environment (branded experiences, games, etc.) [0667] Power virtual assistants and avatars in third party VR apps and games [0668] Power third party autonomous devices (robots, cars, etc.)”. Therefore, the avatar in the autonomous cars corresponds to a virtual driver).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined Spivack into the combination of Fidler and Nguyen, to include all limitations of claim 14. That is, applying the train the avatar of autonomous cars of Spivack to the method of Nguyen and Fidler. The motivation/ suggestion would have been perceptibility of the virtual object and perceptibility of the representation of the real environment is configurable or adjustable by the human user (Spivack, abstract).
Allowable Subject Matter
Claims 5-6, 11-12, 19 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 5, it recites, generating a partial panoramic image from the plurality of real-world images; completing, via a panoramic completion network, the partial panoramic image to generate a low dynamic range image; and wherein generating the panoramic image is from the low dynamic range image. None of the prior arts on the record or any of the prior arts searched, alone or in combination, renders obvious the combination of elements recited in the claim(s) as a whole.
Regarding claim 19, it is interpreted and allowed under similar rationale as claim 5.
Regarding claim 11, it recites, determining a location of a peak amplitude in the sky dome; determining an azimuth, an elevation, and an intensity of a sky dome light source from the location of the peak amplitude; modifying at least one of the azimuth, the elevation, and the intensity to select the target light source; and generating the target lighting representation based on the target light source. None of the prior arts on the record or any of the prior arts searched, alone or in combination, renders obvious the combination of elements recited in the claim(s) as a whole.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GRACE Q LI whose telephone number is (571)270-0497. The examiner can normally be reached Monday - Friday, 8:00 am-5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, DEVONA FAULK can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GRACE Q LI/Primary Examiner, Art Unit 2618 1/8/2026