DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. 17686683, filed on 02/07/2024.
Claim Objections
Claim 17 objected to because of the following informalities:
Claim 17, line 11, recites “processing the input data with a first model”, on line 14, recites “processing the input data with a first model”. It is unclear if the second “a first model” is the same as the first “a first model”. For the purpose of examination, line 14, reads as “processing the input data with the first model”.
Claim 17, line 15, recites “generate background values descriptive of undetermined determined depth”. It is unclear of the meaning of “generate background values descriptive of undetermined determined depth”. For the purpose of examination, it is interpreted as “generate background values descriptive of determined depth”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-4 and 8-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xian et al. (US 11748940 B1), hereinafter as Xian, in view of Gausebeck et al. (US 20190026956 A1), hereinafter as Gausebeck.
Regarding claim 1, Xian teaches A computing system (Col. 18, lines 34-36, “The method may begin at step 410, where a computing system may determine a view position, a view direction, and a time with respect to a scene.”), the system comprising: one or more processors (Col. 20, lines 33-36, “computer system 500 includes a processor 502, memory 504, storage 1006, an input/output (I/O) interface 508, a communication interface 510, and a bus 512.”); and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations (Col. 21, lines 11-13, “memory 504 includes main memory for storing instructions for processor 502 to execute or data for processor 502 to operate on.”, Col. 23, lines 8-10, “a computer-readable non-transitory storage medium or media may include one or more semiconductor-based or other integrated circuits”), the operations comprising: obtaining input data, wherein the input data is descriptive of a three-dimensional position and a two-dimensional view direction (Xian teaches a 6D query input data with (x, y, z) as the 3D position and (θ, ϕ) as the 2D view direction, Col. 15, lines 8-11, “the system may feed the query input data including the view positions, the view directions, and the time 222 to the trained neural network 221”, Col. 8, lines 47-49, “the system may use a space-time neural radiance field (NeRF) framework to build a space-time representation (e.g., a 6D representation of (x, y, z, θ, ϕ, t)”);
processing the input data with a machine-learned view synthesis model to generate a view synthesis output (Col. 15, lines 8-12, “the system may feed the query input data including the view positions, the view directions, and the time 222 to the trained neural network 221 to determine the corresponding image color values 223 for the image to be rendered to the user.”), wherein the machine-learned view synthesis model comprises a neural radiance field model …… (Col. 9, lines 64-67, “Under the framework of the neural radiance field (NeRF), the system may represent the radiance as c=(r, g, b) and differential volume density σ at a 3D location x=(x, y, z) of a scene observed from a viewing direction d=(θ, ϕ) as a continuous multi-variate function using a multi-layer perceptron”)
and wherein training comprised adjusting parameters of the neural radiance field model based on evaluating a loss function that evaluates predicted view synthesis outputs based at least in part on one or more expected depth values, …… (Col. 14, lines 55-64, “During the training process, the system may feed the randomly selected sample locations 202 to the neural network 201 (e.g., MLP) which may generate the output data including the RGB color values, the depths, the empty space locations, the static scene locations. Then, the system may use the color value loss function 213 to compare the network output to the ground truth RGB color values 203, use the depth reconstruction loss function 214 to compare the ground truth depth values from the depth maps 304”)
and providing the view synthesis output as an output, wherein the view synthesis output comprises a novel view synthesis that differs from …… (Col. 9, lines 41-44, “Particular embodiments of the system may use a single video to generate a space-time representation that is a globally consistent and dynamic to represent a scene that can later be rendered from a novel viewpoint.”, Col. 1, lines 48-50, “The system may use neural radiance fields (NeRF) to create new views from arbitrary viewpoints and arbitrary times for dynamic scenes.”).
Xian does not explicitly teach ……trained with a plurality of panoramic images and lidar data, wherein the plurality of panoramic images are descriptive of an environment, …… wherein the one or more expected depth values are determined based at least in part on the lidar data; …… the plurality of panoramic images. Gausebeck teaches ……trained with a plurality of panoramic images and lidar data (Gausebeck paragraph [0100] “The one or more panorama models 514 can employ a neural network model that has been trained on panoramic images with 3D ground truth data associated therewith ……. The 2D/3D panoramic capture device can incorporate one or more cameras (or one or more camera lenses) that provide a field-of-view up to a 360°, as well as one or more depth sensors that provide a filed-of-view up to 360°, thereby providing for capture of an entire panoramic image and panoramic depth data associated therewith to be captured simultaneously and merged into a 2D/3D panoramic image……the depth sensors can include but are not limited to LiDAR sensors/device”), wherein the plurality of panoramic images are descriptive of an environment (paragraph [0098] “The term panoramic image or panoramic image data is used herein to refer to a 2D image of an environment that has a relatively wide field-of-view compared to standard 2D images”), …… wherein the one or more expected depth values are determined based at least in part on the lidar data (Gausebeck teaches merging depth data captured by lidar sensor into panoramic image, further teaches using neural network to determine depth value, paragraph [0034] “The method can further comprise, based on the receiving, deriving, by the system, depth data for an entirety of the panoramic image using a neural network model configured to derive depth data from a single 2D image.”); …… the plurality of panoramic images (paragraph [0146] “the received 2D image can comprise pairs of panoramic images with fields-of-view spanning 360° (or up to 360°) that were captured from different vertical positions”).
Xian and Gausebeck are in the same field of endeavor, namely computer graphics, especially in the field of using neural network to derive 3D data. Gausebeck teaches using deep neural network based on panoramic image and lidar depth information to generate accurate depth prediction (paragraph [0123] “System 800 introduces usage of various types of auxiliary input data that can be associated with a 2D image to facilitate improving the accuracy of 3D-from-2D predictions.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Gausebeck with the method of Xian to improve depth prediction.
Regarding claim 3, Xian in view of Gausebeck teach The system of claim 1, and further teach wherein the plurality of panoramic images comprise image data generated by one or more cameras with a fisheye lens (Gausebeck paragraph [0106] “the received pano-image data 502 can include an image with a 180° field-of-view captured as a single image capture using a fisheye lens for example.”), wherein the one or more cameras were calibrated with estimated intrinsic parameters and poses relative to a camera rig (Gausebeck paragraph [0133-0134] “the capture device location data 906 can include position information indicating a relative position of the capture device (e.g., the camera and/or a 3D sensor) to its environment, such as a relative or calibrated position of the capture device to an object in the environment, another camera in the environment, another device in the environment …… The camera/image parameters 908 can include information regarding operating parameters and/or settings of the one or more cameras (or one or more camera lenses) used to capture the 2D image data 102 …… the camera/image parameters 908 can include camera settings and capture context information associated with a 2D image (e.g., as metadata or otherwise associated with the received 2D image), including but not limited to: focal length, aperture, field-of-view, shutter speed, lens distortion, lighting (exposure, gamma, tone mapping, black level)”).
Xian and Gausebeck are in the same field of endeavor, namely computer graphics, especially in the field of using neural network to derive 3D data. Gausebeck teaches using deep neural network based on panoramic image and lidar depth information to generate accurate depth prediction (paragraph [0123] “System 800 introduces usage of various types of auxiliary input data that can be associated with a 2D image to facilitate improving the accuracy of 3D-from-2D predictions.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Gausebeck with the method of Xian to improve depth prediction.
Regarding claim 4, Xian in view of Gausebeck teach The system of claim 1, and further teach wherein the lidar data comprises asynchronously captured lidar data (Gausebeck teaches capturing images at different time, mapping images with depth data captured by lidar sensor, this implies the lidar data captured with different timestamps, paragraph [0141] “Related 2D images can also include frames of video captured by a video camera with a fixed position/orientation yet captured at different points in time as one or more characteristics of the environment change at the different points in time.”), and wherein the plurality of panoramic images comprise exposure variations between captured images (Gausebeck paragraph [0158] “the pre-processing component 926 can correct or modify image defects to account for lens distortion, lighting variations (exposure, gamma, tone mapping, black level), color space (white balance) variations, and/or other image defects.” And paragraph [0112] “the stitching component 508 can fill in any possible small holes in the panorama with neighboring color data, thereby unifying exposure data across the boundaries between the respective 2D images (if necessary)”).
Xian and Gausebeck are in the same field of endeavor, namely computer graphics, especially in the field of using neural network to derive 3D data. Gausebeck teaches using deep neural network based on panoramic image and lidar depth information to generate accurate depth prediction (paragraph [0123] “System 800 introduces usage of various types of auxiliary input data that can be associated with a 2D image to facilitate improving the accuracy of 3D-from-2D predictions.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Gausebeck with the method of Xian to improve depth prediction.
Regarding claim 8, Xian in view of Gausebeck teach The system of claim 1, and further teach wherein the one or more expected depth values are determined based at least in part on three-dimensional point cloud data of the lidar data (Gausebeck paragraph [0112] “the stitching component 508 can further generate panoramic 3D images (e.g., point clouds, depth maps, etc.) based on the projected points relative to the 3D coordinate space. For example, the stitching component 508 can employ the initial depth data to create a sinusoidal depth map or a point cloud comprising 3D points projected onto a common 3D spatial coordinate plane.” And paragraph [0135] “The 3D sensor data 910 can include any type of 3D associated data with 2D images included in the received 2D image data 102 that was captured by a 3D sensor or 3D capture hardware. This can include 3D data or depth data captured using one or more, structured light sensor devices, LiDAR devices”).
Xian and Gausebeck are in the same field of endeavor, namely computer graphics, especially in the field of using neural network to derive 3D data. Gausebeck teaches using deep neural network based on panoramic image and lidar depth information to generate accurate depth prediction (paragraph [0123] “System 800 introduces usage of various types of auxiliary input data that can be associated with a 2D image to facilitate improving the accuracy of 3D-from-2D predictions.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Gausebeck with the method of Xian to improve depth prediction.
Regarding claim 9, Xian in view of Gausebeck teach The system of claim 1, and further teach wherein the neural radiance field model was trained based on a loss function that evaluates the predicted view synthesis output based at least in part on a determined line-of-sight (Xian Figure 2A and Figure 3, ray 306 from a view position 303A as the determined line-of-sight, Col. 14, lines 55-67, “During the training process, the system may feed the randomly selected sample locations 202 to the neural network 201 (e.g., MLP) which may generate the output data including the RGB color values, the depths, the empty space locations, the static scene locations. Then, the system may use the color value loss function 213 to compare the network output to the ground truth RGB color values 203, use the depth reconstruction loss function 214 to compare the ground truth depth values from the depth maps 304, use the empty-space loss function 215 to compare toe the ground truth empty space locations 205, and use the static scene loss function 216 to compare to the ground truth static scene locations 206. All comparison results may be fed back to the neural network 201 to further adjust the network parameters to minimize the corresponding loss metrics.”).
Regarding claim 10, Xian in view of Gausebeck teach The system of claim 9, and further teach wherein the determined line-of-sight is associated with a radiance being concentrated at a single point along a ray (Xian Figure 3, Col. 15, lines 59-62, “The system may determine a corresponding color value by aggregating all color values at the 3D locations along the ray 303A as modulated by the corresponding volume density values.”).
Regarding claim 11, Xian teaches A computer-implemented method (Col. 1, lines 36-41, “Particular embodiments described herein relate to systems and methods of generating 4D representations (consisting of three spatial dimensions and one temporal dimension) of AR/VR scenes based on a series of images included in a casually captured video to represent the scene in both spatial and temporal domains.”), the method comprising: obtaining, by a computing system comprising one or more processors (Col. 20, lines 33-34, “computer system 500 includes a processor 502, memory 504, storage 1006”), input data, wherein the input data is descriptive of a three-dimensional position and a two-dimensional view direction (Xian teaches a 6D query input data with (x, y, z) as the 3D position and (θ, ϕ) as the 2D view direction, Col. 15, lines 8-11, “the system may feed the query input data including the view positions, the view directions, and the time 222 to the trained neural network 221”, Col. 8, lines 47-49, “the system may use a space-time neural radiance field (NeRF) framework to build a space-time representation (e.g., a 6D representation of (x, y, z, θ, ϕ, t)”);
processing, by the computing system, the input data with a neural radiance field model to generate a view synthesis output (Col. 15, lines 8-12, “the system may feed the query input data including the view positions, the view directions, and the time 222 to the trained neural network 221 to determine the corresponding image color values 223 for the image to be rendered to the user.”), wherein the neural radiance field model ……(Col. 9, lines 64-67, “Under the framework of the neural radiance field (NeRF), the system may represent the radiance as c=(r, g, b) and differential volume density σ at a 3D location x=(x, y, z) of a scene observed from a viewing direction d=(θ, ϕ) as a continuous multi-variate function using a multi-layer perceptron”)
and wherein training comprised adjusting parameters of the neural radiance field model based on evaluating a loss function that evaluates predicted view synthesis outputs based at least in part on one or more expected depth values, …… (Col. 14, lines 55-64, “During the training process, the system may feed the randomly selected sample locations 202 to the neural network 201 (e.g., MLP) which may generate the output data including the RGB color values, the depths, the empty space locations, the static scene locations. Then, the system may use the color value loss function 213 to compare the network output to the ground truth RGB color values 203, use the depth reconstruction loss function 214 to compare the ground truth depth values from the depth maps 304”)
and providing, by the computing system, the view synthesis output as an output, wherein the view synthesis output comprises a novel view synthesis that differs from …… (Col. 9, lines 41-44, “Particular embodiments of the system may use a single video to generate a space-time representation that is a globally consistent and dynamic to represent a scene that can later be rendered from a novel viewpoint.”, Col. 1, lines 48-50, “The system may use neural radiance fields (NeRF) to create new views from arbitrary viewpoints and arbitrary times for dynamic scenes.”).
Xian does not explicitly teach ……was trained with a plurality of panoramic images and lidar data, wherein the plurality of panoramic images are descriptive of an environment, …… wherein the one or more expected depth values are determined based at least in part on the lidar data; …… the plurality of panoramic images. Gausebeck teaches ……was trained with a plurality of panoramic images and lidar data (paragraph [0100] “The one or more panorama models 514 can employ a neural network model that has been trained on panoramic images with 3D ground truth data associated therewith ……. The 2D/3D panoramic capture device can incorporate one or more cameras (or one or more camera lenses) that provide a field-of-view up to a 360°, as well as one or more depth sensors that provide a filed-of-view up to 360°, thereby providing for capture of an entire panoramic image and panoramic depth data associated therewith to be captured simultaneously and merged into a 2D/3D panoramic image……the depth sensors can include but are not limited to LiDAR sensors/device”), wherein the plurality of panoramic images are descriptive of an environment (paragraph [0098] “The term panoramic image or panoramic image data is used herein to refer to a 2D image of an environment that has a relatively wide field-of-view compared to standard 2D images”), …… wherein the one or more expected depth values are determined based at least in part on the lidar data (Gausebeck teaches merging depth data captured by lidar sensor into panoramic image, further teaches using neural network to determine depth value, paragraph [0034] “The method can further comprise, based on the receiving, deriving, by the system, depth data for an entirety of the panoramic image using a neural network model configured to derive depth data from a single 2D image.”); …… the plurality of panoramic images (paragraph [0146] “the received 2D image can comprise pairs of panoramic images with fields-of-view spanning 360° (or up to 360°) that were captured from different vertical positions”).
Xian and Gausebeck are in the same field of endeavor, namely computer graphics, especially in the field of using neural network to derive 3D data. Gausebeck teaches using deep neural network based on panoramic image and lidar depth information to generate accurate depth prediction (paragraph [0123] “System 800 introduces usage of various types of auxiliary input data that can be associated with a 2D image to facilitate improving the accuracy of 3D-from-2D predictions.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Gausebeck with the method of Xian to improve depth prediction.
Regarding claim 12, Xian in view of Gausebeck teach The method of claim 11, and further teach wherein the loss function comprises a penalization term for non-zero density values outside of a determined high density area associated with a surface in the environment (Xian teaches the empty space loss function as the loss function with a penalization term, Col. 2, lines 25-31, “the system may constrain the empty space between the camera and the first visible scene surface by penalizing non-zero volume densities along each ray up to the point no closer than a pre-determined threshold margin. Empty-space loss combined with the depth reconstruction loss may provide geometric constraints for the representation up to and around visible scene surfaces at each frame.”).
Regarding claim 13, Xian in view of Gausebeck teach The method of claim 11, and further teach wherein the neural radiance field model was trained based on training data comprising a plurality of training positions, a plurality of training view directions, a plurality of training images, and the lidar data (Xian teaches a training sample pool with all locations and viewing directions, further teaches training using the estimated depth map data, Gausebeck teaches generating the depth data from lidar sensor, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to substitute Xian’s estimated depth map data with the lidar data of Gausebeck. Xian, Col. 1, lines 53-57, “the system may use training samples of that are randomly selected from a training sample pool which includes training samples locations in the 3D space over time generated based on RGB images and estimated depth map data.”, Col. 2, lines 41-45, “To generate the training samples, the system may take the union of all sampling locations along all rays of all frames to form a sample pool (including a large number of sampling locations in the 3D space).”).
Xian and Gausebeck are in the same field of endeavor, namely computer graphics, especially in the field of using neural network to derive 3D data. Gausebeck teaches using deep neural network based on panoramic image and lidar depth information to generate accurate depth prediction (paragraph [0123] “System 800 introduces usage of various types of auxiliary input data that can be associated with a 2D image to facilitate improving the accuracy of 3D-from-2D predictions.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Gausebeck with the method of Xian to improve depth prediction.
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xian et al. (US 11748940 B1), hereinafter as Xian, in view of Gausebeck et al. (US 20190026956 A1), hereinafter as Gausebeck, further in view of NPL Martin-Brualla et al. (“NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections”), hereinafter as Martin-Brualla.
Regarding claim 2, Xian in view of Gausebeck teach The system of claim 1, but are not relied on for the below claim language wherein the machine-learned view synthesis model comprises scene-level neural network parameters and per-image exposure parameters, wherein the scene-level neural network parameters and the per-image exposure parameters were jointly trained. Martin-Brualla teaches wherein the machine-learned view synthesis model comprises scene-level neural network parameters (Page 2, Left Column, Second paragraph, “Second, we model the scene as the union of shared and image-dependent elements, thereby enabling the unsupervised decomposition of scene content into “static” and “transient” components.”) and per-image exposure parameters (Page 2, Left Column, Second paragraph, “First, we model per-image appearance variations such as exposure, lighting, weather, and post-processing in a learned low-dimensional latent space. Following the frame work of Generative Latent Optimization [3], we optimize an appearance embedding for each input image, thereby granting NeRF-W the flexibility to explain away photometric and environmental variations between images by learning a shared appearance representation across the entire photo collection.”), wherein the scene-level neural network parameters and the per-image exposure parameters were jointly trained (Page 4, Left Column, Fourth paragraph, “To adapt NeRF to variable lighting and photometric post processing, we adopt the approach of Generative Latent Optimization (GLO) [3] in which each image Ii is assigned a corresponding real-valued appearance embedding vector (a) i of length n(a). We replace the image-independent radiance c(t) in Equation (1) with an image-dependent radiance ci(t), which also introduces a dependency on image index i to the approximated pixel color ˆ Ci:
PNG
media_image1.png
113
464
media_image1.png
Greyscale
The
PNG
media_image2.png
48
116
media_image2.png
Greyscale
embeddings are optimized alongside θ.”).
Xian, Gausebeck and Martin-Brualla are in the same field of endeavor, namely computer graphics, especially in the field of using neural network to derive 3D data. Martin-Brualla teaches a NeRF based neural network based on unstructured images to improve rendering result (Page 2, Left Column, Third paragraph, “We find that NeRF-W significantly improves quality over NeRF in the presence of appearance variation and transient occluders while achieving similar quality in controlled settings.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Martin-Brualla with the method of Xian in view of Gausebeck to improve rendering result.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xian et al. (US 11748940 B1), hereinafter as Xian, in view of Gausebeck et al. (US 20190026956 A1), hereinafter as Gausebeck, further in view of Rong et al. (US 20210383616 A1), hereinafter as Rong.
Regarding claim 7, Xian in view of Gausebeck teach The system of claim 1, but are not relied on for the below claim language wherein the loss function comprises a lidar loss that evaluates a difference between opacity values of the predicted view synthesis output and one or more points of the lidar data. Rong teaches wherein the loss function comprises a lidar loss that evaluates a difference between opacity values of the predicted view synthesis output and one or more points of the lidar data (Rong teaches a lidar loss function in training a neural network, paragraph [0176] “The system may then evaluate a loss function by comparing the predicted image data and predicted LiDAR data against the original image data 802 and the original LiDAR data 804. The loss function may include a silhouette loss 814, a LiDAR loss 812, and a regularization loss……The LiDAR loss 812 may compare the predicted LiDAR data and the original LiDAR data 804”).
Xian, Gausebeck and Rong are in the same field of endeavor, namely computer graphics, especially in the field of using neural network to derive 3D data. Rong teaches using lidar loss function to train neural network to improve accuracy. Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Rong with the method of Xian in view of Gausebeck to improve accuracy.
Claim(s) 5-6 and 14-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xian et al. (US 11748940 B1), hereinafter as Xian, in view of Gausebeck et al. (US 20190026956 A1), hereinafter as Gausebeck, and further in view of Hao et al. (US 20220180602 A1), hereinafter as Hao.
Regarding claim 5, Xian in view of Gausebeck teach The system of claim 1, wherein the machine-learned view synthesis model was trained based on: and further teach ……evaluating a second loss function that evaluates the predicted view synthesis output generated with the machine-learned view synthesis model and at least one of one or more of the plurality of ……; and adjusting one or more parameters of the view synthesis model based at least in part on the loss function (Xian Col. 59, lines 59-68, “Then, the system may use the color value loss function 213 to compare the network output to the ground truth RGB color values 203, use the depth reconstruction loss function 214 to compare the ground truth depth values from the depth maps 304, use the empty-space loss function 215 to compare the the ground truth empty space locations 205, and use the static scene loss function 216 to compare to the ground truth static scene locations 206. All comparison results may be fed back to the neural network 201 to further adjust the network parameters to minimize the corresponding loss metrics.”).
Xian in view of Gausebeck fail to teach processing the plurality of panoramic images with a pre-trained semantic segmentation model to generate a plurality of augmented images …… augmented images …… Hao teaches processing the plurality of panoramic images with a pre-trained semantic segmentation model to generate a plurality of augmented images …… augmented images …… (Hao teaches a semantic segmentation model based on input image, and further teaches using semantic data to generate augmented images. Gausebeck teaches a neural network based on input of panoramic images, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Hao with Gausebeck. Hao paragraph [0065] “a user can use an image generation application or system, such as GauGAN from NVIDIA Corporation. In at least one embodiment, an application such as GauGAN can create photorealistic images from semantic data, such as semantic segmentation masks or maps, used to depict a layout of a scene. In at least one embodiment, a latent space can be provided as input to a segmentation mask generator, which may include one or more neural networks trained to accept features or a feature space as input and generate one or more corresponding segmentation masks.”, paragraph [0071] “labeled images can be provided that can be used to generate training data. In at least one embodiment, segmentation masks can be generated from these images to generate pairs of segmentation masks and images as ground truth data, which can be used to train a network to generate images from segmentation masks.”).
Xian, Gausebeck and Hao are in the same field of endeavor, namely computer graphics, especially in the field of using neural network to derive 3D data. Hao teaches using semantic segmentation to achieve realistic results (paragraph [0048] “semantic features of blocks in FIG. 1A can be used to identify semantic segmentation regions corresponding to trees 102, bushes 104, mountains, dirt paths, grass, sky, and so on. In at least one embodiment, being able to determine these regions of similar semantic features, or semantic segmentation regions, enables realistic (or at least more realistic or stylized) objects or elements to be generated or rendered in those approximate locations.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Hao with the method of Xian and Gausebeck to achieve realistic results.
Regarding claim 6, Xian in view of Gausebeck and Hao teach The system of claim 5, and further teach wherein the second loss function comprises a photometric-based loss that evaluates a difference between the predicted view synthesis output and one or more of the plurality of augmented images (Xian teaches a photometric loss function between the generated images and the ground truth image, Hao teaches using semantic segmentation model to generate augmented ground truth image, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Hao with Xian, Xian Col. 2, lines 5-10, “The training process may use multiple loss functions and constraints. The first loss function may be for multiple posed images capturing a scene from different viewpoints. This loss function may minimize the photometric loss (color values) of the ground truth images (input video frames) and the generated images.”).
Regarding claim 14, Xian in view of Gausebeck teach The method of claim 13, wherein the neural radiance field model was trained by: and further teach obtaining, by the computing system, the training data (Xian, Col. 1, lines 53-57, “the system may use training samples of that are randomly selected from a training sample pool which includes training samples locations in the 3D space over time generated based on RGB images and estimated depth map data.”); …… processing, by the computing system, a training position and a training view direction with the neural radiance field model to generate the predicted view synthesis output (Xian Col. 15, lines 8-12, “the system may feed the query input data including the view positions, the view directions, and the time 222 to the trained neural network 221 to determine the corresponding image color values 223 for the image to be rendered to the user.”); evaluating, by the computing system, a loss function that evaluates a difference between the predicted view synthesis output and at least one of …… (Xian Col 59, lines 59-68, “Then, the system may use the color value loss function 213 to compare the network output to the ground truth RGB color values 203, use the depth reconstruction loss function 214 to compare the ground truth depth values from the depth maps 304, use the empty-space loss function 215 to compare the ground truth empty space locations 205, and use the static scene loss function 216 to compare to the ground truth static scene locations 206.”); and adjusting, by the computing system, one or more parameters of the neural radiance field model based at least in part on the loss function (Xian Col. 59, lines 68, “All comparison results may be fed back to the neural network 201 to further adjust the network parameters to minimize the corresponding loss metrics.”).
Xian in view of Gausebeck fail to teach …… processing, by the computing system, the plurality of training images with a pre-trained semantic segmentation model to generate a plurality of augmented images; …… one or more of the plurality of augmented images or one or more points of the lidar data …… Hao teaches …… processing, by the computing system, the plurality of training images with a pre-trained semantic segmentation model to generate a plurality of augmented images; …… one or more of the plurality of augmented images or one or more points of the lidar data …… (Hao teaches a semantic segmentation model based on input image, and further teaches using semantic data to generate augmented images. Gausebeck teaches a neural network based on input of panoramic images, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Hao with Xian and Gausebeck. Hao paragraph [0065] “a user can use an image generation application or system, such as GauGAN from NVIDIA Corporation. In at least one embodiment, an application such as GauGAN can create photorealistic images from semantic data, such as semantic segmentation masks or maps, used to depict a layout of a scene. In at least one embodiment, a latent space can be provided as input to a segmentation mask generator, which may include one or more neural networks trained to accept features or a feature space as input and generate one or more corresponding segmentation masks.”, paragraph [0071] “labeled images can be provided that can be used to generate training data. In at least one embodiment, segmentation masks can be generated from these images to generate pairs of segmentation masks and images as ground truth data, which can be used to train a network to generate images from segmentation masks.”).
Xian, Gausebeck and Hao are in the same field of endeavor, namely computer graphics, especially in the field of using neural network to derive 3D data. Hao teaches using semantic segmentation to achieve realistic results (paragraph [0048] “semantic features of blocks in FIG. 1A can be used to identify semantic segmentation regions corresponding to trees 102, bushes 104, mountains, dirt paths, grass, sky, and so on. In at least one embodiment, being able to determine these regions of similar semantic features, or semantic segmentation regions, enables realistic (or at least more realistic or stylized) objects or elements to be generated or rendered in those approximate locations.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Hao with the method of Xian and Gausebeck to achieve realistic results.
Regarding claim 15, Xian in view of Gausebeck and Hao teach The method of claim 14, and further teach wherein the pre-trained semantic segmentation model was trained to remove occlusions from images (Hao paragraph [0062] “some amount of pre-processing may be performed to remove from consideration unnecessary blocks, such as blocks that are underground or hidden behind or beneath other blocks. In at least one embodiment, processing requirements can be reduced by only considering blocks that contribute to a current view, not blocks that correspond to hidden or unexplored areas that will not be visible in a view or scene.”).
Xian, Gausebeck and Hao are in the same field of endeavor, namely computer graphics, especially in the field of using neural network to derive 3D data. Hao teaches using semantic segmentation to achieve realistic results (paragraph [0048] “semantic features of blocks in FIG. 1A can be used to identify semantic segmentation regions corresponding to trees 102, bushes 104, mountains, dirt paths, grass, sky, and so on. In at least one embodiment, being able to determine these regions of similar semantic features, or semantic segmentation regions, enables realistic (or at least more realistic or stylized) objects or elements to be generated or rendered in those approximate locations.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Hao with the method of Xian and Gausebeck to achieve realistic results.
Regarding claim 16, Xian in view of Gausebeck and Hao teach The method of claim 14, and further teach wherein the predicted view synthesis output comprises one or more predicted color values and one or more predicted opacity values (Xian teaches the NeRF with color value and volume density as the opacity value. Col. 9, lines 64-67, “Under the framework of the neural radiance field (NeRF), the system may represent the radiance as c=(r, g, b) and differential volume density σ at a 3D location x=(x, y, z) of a scene observed from a viewing direction d=(θ, ϕ) as a continuous multi-variate function using a multi-layer perceptron”).
Claim(s) 17 and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xian et al. (US 11748940 B1), hereinafter as Xian, in view of Gausebeck et al. (US 20190026956 A1), hereinafter as Gausebeck, and further in view of Yoon et al. (US 11546568 B1), hereinafter as Yoon.
Regarding claim 17, Xian teaches One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations (Col. 21, lines 11-13, “memory 504 includes main memory for storing instructions for processor 502 to execute or data for processor 502 to operate on.”, Col. 23, lines 8-10, “a computer-readable non-transitory storage medium or media may include one or more semiconductor-based or other integrated circuits (ICs)”), the operations comprising: obtaining input data, wherein the input data is descriptive of a three-dimensional position and a two-dimensional view direction (Xian teaches a 6D query input data with (x, y, z) as the 3D position and (θ, ϕ) as the 2D view direction, Col. 15, lines 8-11, “the system may feed the query input data including the view positions, the view directions, and the time 222 to the trained neural network 221”, Col. 8, lines 47-49, “the system may use a space-time neural radiance field (NeRF) framework to build a space-time representation (e.g., a 6D representation of (x, y, z, θ, ϕ, t)”);
processing the input data with a machine-learned view synthesis model to generate a view synthesis output (Col. 15, lines 8-12, “the system may feed the query input data including the view positions, the view directions, and the time 222 to the trained neural network 221 to determine the corresponding image color values 223 for the image to be rendered to the user.”), wherein the machine-learned view synthesis model comprises a neural radiance field model …… (Col. 9, lines 64-67, “Under the framework of the neural radiance field (NeRF), the system may represent the radiance as c=(r, g, b) and differential volume density σ at a 3D location x=(x, y, z) of a scene observed from a viewing direction d=(θ, ϕ) as a continuous multi-variate function using a multi-layer perceptron”), wherein processing the input data with a machine-learned view synthesis model to generate a view synthesis output comprises (Col. 15, lines 8-12, “the system may feed the query input data including the view positions, the view directions, and the time 222 to the trained neural network 221 to determine the corresponding image color values 223 for the image to be rendered to the user.”):
…… and providing the view synthesis output as an output, wherein the view synthesis output comprises a novel view synthesis that differs from …… (Col. 9, lines 41-44, “Particular embodiments of the system may use a single video to generate a space-time representation that is a globally consistent and dynamic to represent a scene that can later be rendered from a novel viewpoint.”, Col. 1, lines 48-50, “The system may use neural radiance fields (NeRF) to create new views from arbitrary viewpoints and arbitrary times for dynamic scenes.”).
Xian does not explicitly teach …… trained with a plurality of images and lidar data, wherein the plurality of images are descriptive of an environment, …… processing the input data with a first model of the machine-learned view synthesis model to generate foreground values descriptive of determined depth values within the environment; processing the input data with a first model of the machine-learned view synthesis model to generate background values descriptive of undetermined determined depth values within the environment; and generating the view synthesis output based on the foreground values and the background values; …… the plurality of images. Gausebeck teaches ……trained with a plurality of images and lidar data (paragraph [0100] “The one or more panorama models 514 can employ a neural network model that has been trained on panoramic images with 3D ground truth data associated therewith ……. The 2D/3D panoramic capture device can incorporate one or more cameras (or one or more camera lenses) that provide a field-of-view up to a 360°, as well as one or more depth sensors that provide a filed-of-view up to 360°, thereby providing for capture of an entire panoramic image and panoramic depth data associated therewith to be captured simultaneously and merged into a 2D/3D panoramic image……the depth sensors can include but are not limited to LiDAR sensors/device”), wherein the plurality of images are descriptive of an environment (paragraph [0098] “The term panoramic image or panoramic image data is used herein to refer to a 2D image of an environment that has a relatively wide field-of-view compared to standard 2D images”) …… the plurality of images (paragraph [0146] “the received 2D image can comprise pairs of panoramic images with fields-of-view spanning 360° (or up to 360°) that were captured from different vertical positions”).
Xian and Gausebeck are in the same field of endeavor, namely computer graphics, especially in the field of using neural network to derive 3D data. Gausebeck teaches using deep neural network based on panoramic image and lidar depth information to generate accurate depth prediction (paragraph [0123] “System 800 introduces usage of various types of auxiliary input data that can be associated with a 2D image to facilitate improving the accuracy of 3D-from-2D predictions.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Gausebeck with the method of Xian to improve depth prediction.
Xian in view of Gausebeck fail to teach processing the input data with a first model of the machine-learned view synthesis model to generate foreground values descriptive of determined depth values within the environment; processing the input data with a first model of the machine-learned view synthesis model to generate background values descriptive of undetermined determined depth values within the environment; and generating the view synthesis output based on the foreground values and the background values; Yoon teaches processing the input data with a first model of the machine-learned view synthesis model to generate foreground values descriptive of determined depth values within the environment (Yoon Col. 7, lines 17-20, “a set of single view and multi-view depth maps can be generated 604, where single view depth can provide accurate depth estimates of foreground objects”); processing the input data with a first model of the machine-learned view synthesis model to generate background values descriptive of undetermined determined depth values within the environment (Yoon Col. 7, lines 17-22, “From this collection, a set of single view and multi-view depth maps can be generated 604, ……multi-view depth can provide accurate depth estimates for background objects or regions.”); and generating the view synthesis output based on the foreground values and the background values (Yoon Col. 7, lines 46-50, “a blender network can be designed to combine foreground, background, and missing regions. Any of a number of synthesis algorithms can be used to generate a final output image based on output from blender network 506”);
Xian, Gausebeck and Yoon are in the same field of endeavor, namely computer graphics, especially in the field of using neural network to derive 3D data. Yoon teaches using neural network to combine foreground depth information and background depth information to achieve more accurate, efficient and timely results. Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Yoon with the method of Xian and Gausebeck to achieve more accurate, efficient and timely results.
Regarding claim 19, Xian in view of Gausebeck and Yoon teach The one or more non-transitory computer-readable media of claim 17, and further teach wherein the foreground values comprise foreground color values and foreground opacity values for foreground features (Xian teaches the NeRF with color value and volume density as the opacity value, this includes the foreground feature of the NeRF, Col. 9, lines 64-67, “Under the framework of the neural radiance field (NeRF), the system may represent the radiance as c=(r, g, b) and differential volume density σ at a 3D location x=(x, y, z) of a scene observed from a viewing direction d=(θ, ϕ) as a continuous multi-variate function using a multi-layer perceptron”).
Regarding claim 20, Xian in view of Gausebeck and Yoon teach The one or more non-transitory computer-readable media of claim 17, and further teach wherein the background values comprise background color values and background opacity values for background features (Xian teaches the NeRF with color value and volume density as the opacity value, this includes the background feature of the NeRF, Col. 9, lines 64-67, “Under the framework of the neural radiance field (NeRF), the system may represent the radiance as c=(r, g, b) and differential volume density σ at a 3D location x=(x, y, z) of a scene observed from a viewing direction d=(θ, ϕ) as a continuous multi-variate function using a multi-layer perceptron”).
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xian et al. (US 11748940 B1), hereinafter as Xian, in view of Gausebeck et al. (US 20190026956 A1), hereinafter as Gausebeck, and further in view of Yoon et al. (US 11546568 B1), hereinafter as Yoon, and NPL Hao Zekun et al. (“GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds”), hereinafter as HaoZekun.
Regarding claim 18, Xian in view of Gausebeck and Yoon The one or more non-transitory computer-readable media of claim 17, but fail to teach wherein training the machine-learned view synthesis model comprises leveraging the lidar data to perform supervised learning of predicted densities on rays pointing at a sky. HaoZekun teaches wherein training the machine-learned view synthesis model comprises leveraging the lidar data to perform supervised learning of predicted densities on rays pointing at a sky (HaoZekun teaches using a neural sky dome in a unsupervised neural radiance field for video games, Xian in view of Gausebeck and Yoon teaches supervised learning of NeRF with lidar data, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of sky dome with the method of Xian, Gausebeck and Yoon. HaoZekun Page 5, Left Column, Third paragraph to Right column, first paragraph, “we describe how a scene represented by the above-mentioned neural radiance fields and sky dome can be converted to 2D feature maps via volumetric rendering. Under a perspective camera model, each pixel in the image corresponds to a camera ray, originating from the center of projection o and advances in direction v. The ray travels through the radiance field while accumulating features and transmittance …… As the radiance field is bounded by a finite number of voxels, the ray will eventually exit the voxels and hit the sky dome. We thus consider the sky dome as the last data point on the ray, which is totally opaque.”, Xian Col. 15, lines 24-28, “In particular embodiments, the system may use hierarchical volume sampling as in the NeRF framework and simultaneously train both the coarse and fine networks. The system may apply all losses to supervise the predictions from both networks.”).
Xian, Gausebeck, Yoon and HaoZekun are in the same field of endeavor, namely computer graphics, especially in the field of using neural network to derive 3D data. HaoZekun teaches using neural sky dome with neural radiance field to achieve realistic results (Page 5, Left Column, Second paragraph, “The sky is an indispensable part of photorealistic landscape scenes. However, as it is physically located much farther away from the other objects, it is inefficient to represent it with a layer of voxels.”, Page 7, Left Column, Third paragraph, “We can observe that our outputs are more realistic and view consistent when compared to baselines.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of HaoZekun with the method of Xian, Gausebeck and Yoon to achieve realistic results.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Garbin et al. (US 20220301257 A1) teaches a machine learning model to generate novel images based on training data.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIAOMING WEI whose telephone number is (571)272-3831. The examiner can normally be reached M-F 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XIAOMING WEI/Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights