DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Preliminary Amendment
The preliminary amendment submitted on 03/06/2025 is acknowledged.
Information Disclosure Statement
The information disclosure statement (IDS) submitted is considered by the examiner.
Claim Objections
Claims 1, 10 and 13 are objected to because of the following informalities: it is suggested to delete the reference numbers associated with the claim limitations to make the claim’s legibility clear. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-7, 13-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. (NPL, “Scalable Neural Indoor Scene Rendering”, 07/2022) in view of Laina et al. (NPL, “Deeper Depth Prediction with Fully Convolutional Residual Networks”, 2016).
(1) regarding claim 1:
As shown in fig. 1, Wu disclosed in an encoder, a method to generate a scalable 3D scene representation (fig. 1, col. 1, lines 1-2, note that scalable neural scene reconstruction and rendering method to support distributed training and interactive rendering of large indoor scenes), the method comprising:
accessing a first set of images in a first format (102) for a scene (page 4, as shown in fig. 2, note that the surface MLP that encodes density and view independent color, which is later stored in an octree);
generating a first 3D scene representation (107) for the scene based on the first set of images (page 4, page 3, 3. Scalable Neural Scene Representation, para. [0002], note that tile MLPs are locally low capacity. This allows faster optimization than one large scene network, but still provides the ability to represent a complex scene collectively);
accessing a second set of images in a second format (104) for the scene (page 4, as shown in fig. 2, note that the reflection MLP that encodes view-dependent effects like highlights using virtual points ‘underneath’ the surface at the ray distance of the reflected light);
generating a second 3D scene representation (112) for the scene based on the second set of images, wherein the second 3D representation is better than the first 3D scene representation according to one or more quality criteria (page 4, page 3, 3. Scalable Neural Scene Representation, para. [0002], note that tiles can be trained in parallel using a new background sampling method. Further, along a specific ray, we can coordinate 3D point samplings across MLPs for efficiency);
using a set of original viewing positions and a set of novel viewing positions, generating output image residuals (122) based on the first 3D scene representation and the second 3D scene representation (page 4, 3.2 Representation and Rendering, para. [0001], note that define a 3D point as input to our neural fields as x = {𝑥,𝑦,𝑧}. For view-independent reflection, we define a surface MLP 𝑓𝑠 𝜃 : R3 → R4 with parameters𝜃 that outputs diffuse color c𝑠 and volume density);
transmitting the first 3D scene representation (107) for the scene as a base layer (page 9, 6. Parameters and Training Routines, para. [0001], note that for the surface MLP, we use 8 fully connected layers of size 128. As in Midenhall et al. [2020], we use 10 frequency bands for the positional encoding. For the reflection MLP, we use 8 fully connected layers of size 64, which can be stored in the shared memory of kernel blocks on modern GPUs for efficient rendering).
Wu disclosed most of the subject matter as described as above except for specifically teaching generating output image residuals (122); training a residual neural field network (125) using the output image residuals to generate predicted residual images approximating the output image residuals; and transmitting information about the trained residual neural field network as an enhancement layer.
However, Laina disclosed generating output image residuals (122) (abs. note that a fully convolutional architecture, encompassing residual learning, to model the ambiguous mapping between monocular images and depth maps); training a residual neural field network (125) using the output image residuals to generate predicted residual images approximating the output image residuals (page 7, para. [0002], note that one can clearly see the improvement in quality from AlexNet to ResNet, however the fully connected variant of ResNet, despite its increased accuracy, is still limited to coarse predictions. The proposed fully convolutional model greatly improves edge quality and structure definition in the predicted depth maps.); and transmitting information about the trained residual neural field network as an enhancement layer (page 8, 5 Conclusion, para. [0001], note that up-projection layers that allow for training much deeper configurations, while greatly reducing the number of parameters to be learned and the number of training samples required).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach generating output image residuals; training a residual neural field network using the output image residuals to generate predicted residual images approximating the output image residuals; and transmitting information about the trained residual neural field network as an enhancement layer. The suggestion/motivation for doing so would have been in order to obtain more efficient scheme for up-convolutions and combine it with the concept of residual learning to create up-projection blocks for the effective upsampling of feature maps (page 2, para. [0001]). Therefore, it would have been obvious to combine Wu with Laina to obtain the invention as specified in claim 1.
(2) regarding claim 2:
Wu further disclosed the method of claim 1, further comprising reformatting outputs of the first 3D scene representation or the second 3D scene representations before generating the image residuals (page 3, 3 Scalable Neural Scene Representation, para. [0001], note that to achieve equivalent quality, and require more time in volume rendering to integrate points over larger distances. Our goal is to reduce network size without sacrificing quality such that neural rendering can scale to large scenes both during optimization and for interactive rendering).
(3) regarding claim 3:
Wu further disclosed the method of claim 2, wherein reformatting comprises image upscaling, image downscaling, frame dropping, frame interpolation, or dynamic range/colour gamut extension (fig. 2, note that tiles are allocated based on a global mesh proxy. Each tile has two MLPs: 1) The surface MLP that encodes density and view independent color, which is later stored in an octree for fast rendering. 2) The reflection MLP that encodes view-dependent effects like highlights using virtual points ‘underneath’ the surface at the ray distance of the reflected light. Color outputs from both paths are combined in the final rendering).
(4) regarding claim 4:
Wu further disclosed the method of claim 1, wherein the one or more quality criteria include PSNR scalability, dynamic range scalability, color gamut scalability, spatial resolution scalability, and temporal frame-rate scalability (see table 2, page 10, 7.2 Rendering Results and comparisons, the method produces photo-realistic renderings with higher-quality view-dependent effects like high lights and reflections. Table 2 holds quantitative measures, showing that our method produces competitive PSNR and SSIM scores).
(5) regarding claim 5:
Wu further disclosed the method of claim 1, wherein the first set of images is identical to the second set of images (fig. 12, page 11, Fig. 12. Comparisons with NeRF [Mildenhall et al. 2020], PlenOctrees and NSVF. We fine tune both methods using just images facing the coffee table).
(6) regarding claim 6:
Wu further disclosed the method of claim 1, wherein the first set of images differs from the second set of images in terms of dynamic range or bit-depth, color gamut, spatial resolution, or frame rate (fig. 12, page 11, Fig. 12. Comparisons with NeRF, PlenOctrees and NSVF. We fine tune both methods using just images facing the coffee table, note that although the image is the same they are rendered differently using different resolution).
(7) regarding claim 7:
Wu further disclosed the method of claim 1, wherein a 3D scene representation may be one of multiview plus depth (MVD) representation, a multi-plane imaging (MPI) representation, or a neural radiance field (NeRF) neural network representation (fig. 12, page 11, Fig. 12. Comparisons with NeRF, PlenOctrees and NSVF. We fine tune both methods using just images facing the coffee table; our result has both high quality and interactive rendering).
(8) regarding claim 13:
Wu disclosed most of the subject matter as described as above except for specifically teaching wherein information about the trained residual neural field network comprises one or more of: a quality parameter specifying the one or more quality criteria, camera viewport information, first model parameters for the first 3D representation model, the number of hidden layers in the residual neural field, the input position encoding method, the activation function, parameters related to residual rescaling descriptors of input coordinate parameters, and descriptors of output parameters residual rescaling parameters.
However, Laina disclosed wherein information about the trained residual neural field network comprises one or more of: a quality parameter specifying the one or more quality criteria, camera viewport information, first model parameters for the first 3D representation model, the number of hidden layers in the residual neural field, the input position encoding method, the activation function, parameters related to residual rescaling descriptors of input coordinate parameters, and descriptors of output parameters residual rescaling parameters (page 7, para. [0002], note that one can clearly see the improvement in quality from AlexNet to ResNet, however the fully connected variant of ResNet, despite its increased accuracy, is still limited to coarse predictions. The proposed fully convolutional model greatly improves edge quality and structure definition in the predicted depth maps).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein information about the trained residual neural field network comprises one or more of: a quality parameter specifying the one or more quality criteria, camera viewport information, first model parameters for the first 3D representation model, the number of hidden layers in the residual neural field, the input position encoding method, the activation function, parameters related to residual rescaling descriptors of input coordinate parameters, and descriptors of output parameters residual rescaling parameters. The suggestion/motivation for doing so would have been in order to obtain more efficient scheme for up-convolutions and combine it with the concept of residual learning to create up-projection blocks for the effective upsampling of feature maps (page 2, para. [0001]). Therefore, it would have been obvious to combine Wu with Laina to obtain the invention as specified in claim 13.
(9) regarding claim 14:
Wu disclosed most of the subject matter as described as above except for specifically teaching wherein the information is transmitted as part of supplemental enhancement information messaging.
However, Laina disclosed wherein the information is transmitted as part of supplemental enhancement information messaging (page 6, para. [0004], note that the fully convolutional variant using simple up convolutions (ResNet-UpConv) improves accuracy, and at last, the proposed architecture (ResNet-UpProj), enhanced with the up-projection blocks, gives by far the best results).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the information is transmitted as part of supplemental enhancement information messaging. The suggestion/motivation for doing so would have been in order to obtain more efficient scheme for up-convolutions and combine it with the concept of residual learning to create up-projection blocks for the effective upsampling of feature maps (page 2, para. [0001]). Therefore, it would have been obvious to combine Wu with Laina to obtain the invention as specified in claim 14.
(10) regarding claim 15:
Wu disclosed most of the subject matter as described as above except for specifically teaching wherein training the residual neural field network (125) using the output image residuals is in a first spatial resolution, and further comprising: training the residual neural field network with input the output image residuals at a second spatial resolution lower than the first spatial resolution, and with output the predicted residual images at the first spatial resolution.
However, Laina disclosed wherein training the residual neural field network (125) using the output image residuals is in a first spatial resolution, and further comprising: training the residual neural field network with input the output image residuals at a second spatial resolution lower than the first spatial resolution, and with output the predicted residual images at the first spatial resolution (page 3, 3.1 CNN Architecture, para. [0002], note that the receptive field at the last convolutional layer of AlexNet is 151 × 151 pixels, allowing only very low resolution input images when true global information (e.g. monocular cues) should be captured by the network without fully-connected layers. A larger receptive field of 276×276 is achieved by VGG-16 but still sets a limit to the input resolution).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein training the residual neural field network (125) using the output image residuals is in a first spatial resolution, and further comprising: training the residual neural field network with input the output image residuals at a second spatial resolution lower than the first spatial resolution, and with output the predicted residual images at the first spatial resolution. The suggestion/motivation for doing so would have been in order to obtain more efficient scheme for up-convolutions and combine it with the concept of residual learning to create up-projection blocks for the effective upsampling of feature maps (page 2, para. [0001]). Therefore, it would have been obvious to combine Wu with Laina to obtain the invention as specified in claim 15.
(11) regarding claim 16:
Wu further disclosed a non-transitory computer-readable storage medium having stored thereon computer-executable instructions for executing with one or more processors a method in accordance with claim 1 (page 2, para. [0005], note that tMLPs en able different network capacity, storage, and rendering schemes for view-independent and view-dependent components).
(12) regrading claim 17:
Wu further disclosed an apparatus comprising a processor and configured to perform the methods recited in claim 1 (para. 2, para. [0001], note that reconstructing real-world 3D scenes and rendering them to photo realistically at interactive rates has long been desired in visual computing).
Claim(s) 10-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dore et al. (US Publication Number 2020/0228777 A1) in view of Wu et al. (NPL, “Scalable Neural Indoor Scene Rendering”, 07/2022).
(1) regarding claim 10:
As shown in fig. 5 and 6, Dore disclosed a decoder, a method to generate an output 3D scene (para. [0043], note that a decoder receiving a data stream comprising, in a first element of syntax, an image as the example images of FIGS. 5 and 6 decodes the image using a method correlated to the method used for the encoding of the image), the method comprising:
receiving a base layer bitstream comprising a first 3D scene representation for a scene (para. [0082], note that a bitstream comprising data representative of the volumetric scene is sent to a destination. As an example, the bitstream is stored in a local or remote memory);
receiving an enhancement layer bitstream comprising information to reconstruct a trained residual neural field network (para. [0083], note that the bitstream is obtained from a source. Exemplarily, the bitstream is read from a local memory, e.g. a video memory (124), a RAM (124), a ROM (123), a flash memory (123) or a hard disk (123). In a variant, the bitstream is received from a storage interface (125));
given a viewer position: generating a first 3D output of a scene based on the first 3D scene representation (para. [0035], note that this first element of syntax comprises a color image encoded according to a projection mapping of points of the 3D scene to the color image from a determined point of view); and
generating image residuals using the viewer position and the trained residual neural field network (para. [0034], note that residual points of the scene are points that are not visible from the point of view determined to be the fixed point of view in case of a 3DoF rendering. Pixels corresponding to the residual points are left in the MVD frames. In the second embodiment, the method further encodes the color and the depth of points of the scene that are not already encoded in the first color image. Residual points of the scene are points that are not visible from the point of view determined to be the fixed point of view in case of a 3DoF rendering. Residual points may be encoded in one or more color image and depth image).
Dore disclosed most of the subject matter as described above except for specifically teaching combining the first 3D output of the scene and the image residuals to generate an enhanced 3D output of the scene.
However, Wu disclosed combining the first 3D output of the scene and the image residuals to generate an enhanced 3D output of the scene (page 8, 4.5 Ray Sampling, Octrees and Voxel Pruning, para. [0006], note that sample 3Dpoints for surface and reflection MLPs respectively during training. Combining these 3D points with background sampling points lets each tMLPs be trained in parallel).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to combining the first 3D output of the scene and the image residuals to generate an enhanced 3D output of the scene. The suggestion/motivation for doing so would have been in order to produce realistic interactive rendering results for free-viewpoint navigation of indoor scenes (page 2, 1 Introduction, para. [0007]). Therefore, it would have been obvious to combine Dore with Wu to obtain the invention as specified in claim 10.
(2) regarding claim 11:
Dore further disclosed the method of claim 10, further comprising reformatting the first 3D output of the scene or the image residuals before combining them (para. [0043], note that the decoder may transmit the decoded image (or sequence of images) to a 3DoF renderer or to a module for reformatting for example. A 3DoF renderer would project the image on a surface corresponding to the projection mapping used at the encoding).
(3) regarding claim 12:
Dore further disclosed the method of claim 11, wherein reformatting comprises image upscaling, image downscaling, frame dropping or frame interpolation (para. [0033], note that FIG. 15 shows a multi-view plus depth (MVD) frame comprising two views. Each view of a MVD frame comprises a color image (also called texture) and a depth image. In the example of FIG. 15, the MVD frame comprises two views, the first view comprising a color image 150t and a depth image 150d and the second view comprising a color image 151t and a depth image 151d).
Allowable Subject Matter
Claims 8-9 are objected to as being dependent upon a rejected base claim but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: the prior arts of record do not teach “wherein the first 3D scene representation comprises a first NeRF model and the second 3D scene representation comprises a second NeRF model, wherein the second NeRF model renders better quality images than the first NeRF model, and generating the output image residuals comprises:
PNG
media_image1.png
222
574
media_image1.png
Greyscale
” as recited in claim 8. Claim 9 depends on claim 8.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
He et al. (NPL, “Deep Residual Learning for Image Recognition”, 2015) disclosed provide com prehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.
Kim et al. (US Publication Number 2021/0211724 A1) disclosed a system comprises an encoder configured to compress attribute information and/or spatial information for volumetric visual content and/or a decoder configured to decompress compressed attribute and/or spatial information for the volumetric visual content.
Any inquiry concerning this communication or earlier communication from the examiner should be directed to Hilina K Demeter whose telephone number is (571) 270-1676.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Y. Poon could be reached at (571) 270- 0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about PAIR system, see http://pari-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HILINA K DEMETER/Primary Examiner, Art Unit 2617
