DETAILED ACTION
Notice of Pre-AIA or AIA Status
Claims 1-20 are pending in this application. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Munkberg et al. (US PGPub US20200126192A1, hereby referred to as “Munkberg”).
Consider Claim 1.
Munkberg teaches:
1. A method comprising: (Munkberg: abstract, A neural network-based rendering technique increases temporal stability and image fidelity of low sample count path tracing by optimizing a distribution of samples for rendering each image in a sequence. A sample predictor neural network learns spatio-temporal sampling strategies such as placing more samples in dis-occluded regions and tracking specular highlights. Temporal feedback enables a denoiser neural network to boost the effective input sample count and increases temporal stability. The initial uniform sampling step typically present in adaptive sampling algorithms is not needed. The sample predictor and denoiser operate at interactive rates to achieve significantly improved image quality and temporal stability compared with conventional adaptive sampling techniques. [0004]-[0005], [0025]-[0036], Figures 1A-1B; [0034] FIG. 1B illustrates a flowchart of a method 130 for temporally stable data restoration, in accordance with an embodiment. Although method 130 is described in the context of a processing unit, the method 130 may also be performed by a program, custom circuitry, or by a combination of custom circuitry and a program. For example, the method 130 may be executed by a GPU (graphics processing unit), CPU (central processing unit), or any processor capable of implementing the warped external recurrent neural network 100. Furthermore, persons of ordinary skill in the art will understand that any system that performs method 130 is within the scope and spirit of embodiments of the present invention.)
1. receiving an input digital video having a plurality of frames; (Munkberg: [0025] The encoder/decoder neural network model 110 receives input data at time t and warped external state from the previous iteration i.e., at time t−1. The input data includes artifacts that are removed during the reconstruction process to produce output data that approximates the input data without the artifacts. In the context of the following description, output data that approximates the input data without the artifacts has fewer artifacts compared with the input data. The warped external state from time t−1 includes warped reconstructed data from time t−1. [0035] At step 135, a sequence of input data including artifacts is received by the warped external recurrent neural network 100. The sequence includes a first input data frame and a second input data frame. In the context of the following description, an input data frame may include image data or other types of data. In an embodiment, the input data comprises rendered image frames.)
1. processing, by one or more machine-learning models using a processing device, the input digital video having frames in a first resolution into an output digital video having the frames in a second resolution using a flow guided feature propagation module, the processing including: (Munkberg: [0025] The encoder/decoder neural network model 110 receives input data at time t and warped external state from the previous iteration i.e., at time t−1. The input data includes artifacts that are removed during the reconstruction process to produce output data that approximates the input data without the artifacts. In the context of the following description, output data that approximates the input data without the artifacts has fewer artifacts compared with the input data. The warped external state from time t−1 includes warped reconstructed data from time t−1. [0035] At step 135, a sequence of input data including artifacts is received by the warped external recurrent neural network 100. The sequence includes a first input data frame and a second input data frame. In the context of the following description, an input data frame may include image data or other types of data. In an embodiment, the input data comprises rendered image frames. [0050] In an embodiment, each stage of the decoder portion of the encoder/decoder neural network model 110 uses a nearest upsampling layer followed by two convolutional layers. In an embodiment, a pair of convolutional layers of the decoder portion are N×N, each proceeded by a 2×2 nearest upsampling layer, where N=160, 128, 96, 64, and 32 in a feed-forward sequence with the output of each upsampling layer input to each convolutional layer pair. In an embodiment, the output of each encoder stage is propagated to the corresponding decoder stage via residual skip connections and accumulated with the output of an upsampling layer. The residual skip connections improve reconstruction quality and may enable faster training of deep convolutional networks by improving the back propagation of gradients. [0051] In an embodiment, the encoder/decoder neural network model 110 utilizes specialized tensor cores within a GPU, such as the tensor cores within the PPU 300 shown in FIG. 3. To take advantage of the tensor cores, 16-bit tensors and weights are used. In an embodiment, the slowest layers of an encoder/decoder neural network model 110 are the outermost, high resolution layers. The cost of a 3×3 convolution layer at 1080 pixel resolution varies with input and output channel counts. In particular, little variation occurs as output channel count changes from 8 to 64, but a significant slowdown occurs at 40 input channels.)
1. predicting optical flow maps from the plurality of frames of the input digital video; (Munkberg: [0039] In an embodiment, the parameters (e.g., weights and biases) of the encoder/decoder neural network model 110, determined during training, are immutable at inference time and cannot adapt to particular data regions within each frame. In another embodiment, the parameters of the encoder/decoder neural network model 110 determined during training may be modified or calculated during inferencing. Kernel-predicting networks (KPNs), enable a neural network model to generate adaptive spatially-varying kernels by training the neural network model coupled to a filtering module, such as the combiner function 120. As shown in FIG. 1A, a kernel-predicting autoencoder may be constructed by feeding the spatially-varying kernels output by the encoder/decoder neural network model 110 to the combiner function 120. In contrast with conventional KPNs, image sequences generated by the warped external recurrent neural network 100 are temporally stable and the spatially-varying kernels output by the encoder/decoder neural network model 110 are temporally stable. [0040] In an embodiment, the warped external state and the processed second input data frame are processed by the encoder/decoder neural network model 110 to generate spatially-varying filter kernels. When the sequence of input data is an image sequence, the combiner function 120 applies a first filter kernel to pixels of the reconstructed first data frame (from time t−1) and applies a second filter kernel to pixels of the processed second input data frame (from time t). In an embodiment, the second filter kernel is a hierarchy of filter kernels. The filtered pixels are combined (i.e., summed) to produce external state including pixels of the reconstructed second data frame.)
1. learning temporal-aware features based on the optical flow maps and pixels of the plurality of frames; (Munkberg: [0055] FIG. 2C illustrates a diagram of layers of an external neural network layer 202 without hidden state recursion, in accordance with an embodiment. Instead of receiving hidden state generated by the first convolutional layer during processing of a previous frame, the first convolutional layer receives the warped external state generated by the encoder/decoder neural network model 110 during processing of a previous frame. The encoder/decoder neural network model 110 functions as a single recurrent layer. [0056] The warped external state generated by the encoder/decoder neural network model 110, combiner function 120, and temporal warp function 115 acts as the hidden state for the encoder/decoder neural network model 110 and is used as input along with the next data frame. The external recursion configuration has four key advantages. First, the encoder/decoder neural network model 110 may be trained with data frame sequences, allowing the encoder/decoder neural network model 110 to output high quality as well as temporally coherent results. Second, the external state is reduced to just a few megabytes per data frame. Third, the additional cost to process the external state is virtually zero on modern GPUs. Lastly, at inference time the encoder/decoder neural network model 110 acts like a non-recurrent neural network, allowing use of simpler layer models since no internal state is stored or updated. [0057] However, using external recurrence alone is insufficient to guarantee temporally coherent results. The max pooling layers within the encoder/decoder neural network model 110 effectively force learning of translational invariance of data features within the frames. While useful for classification, translational invariance reduces a neural network's ability to precisely identify correspondences between features from current and prior data frames, degrading temporal reuse along edges (where temporal reuse is most important).)
1.and warping the temporal-aware features guided by the optical flow maps; (Munkberg: [0055] FIG. 2C illustrates a diagram of layers of an external neural network layer 202 without hidden state recursion, in accordance with an embodiment. Instead of receiving hidden state generated by the first convolutional layer during processing of a previous frame, the first convolutional layer receives the warped external state generated by the encoder/decoder neural network model 110 during processing of a previous frame. The encoder/decoder neural network model 110 functions as a single recurrent layer. [0056] The warped external state generated by the encoder/decoder neural network model 110, combiner function 120, and temporal warp function 115 acts as the hidden state for the encoder/decoder neural network model 110 and is used as input along with the next data frame. The external recursion configuration has four key advantages. First, the encoder/decoder neural network model 110 may be trained with data frame sequences, allowing the encoder/decoder neural network model 110 to output high quality as well as temporally coherent results. Second, the external state is reduced to just a few megabytes per data frame. Third, the additional cost to process the external state is virtually zero on modern GPUs. Lastly, at inference time the encoder/decoder neural network model 110 acts like a non-recurrent neural network, allowing use of simpler layer models since no internal state is stored or updated. [0057] However, using external recurrence alone is insufficient to guarantee temporally coherent results. The max pooling layers within the encoder/decoder neural network model 110 effectively force learning of translational invariance of data features within the frames. While useful for classification, translational invariance reduces a neural network's ability to precisely identify correspondences between features from current and prior data frames, degrading temporal reuse along edges (where temporal reuse is most important).)
1.and outputting the output digital video having the frames in the second resolution.
Consider Claim 9.
Munkberg teaches:
9. A computing device comprising: a processing device; and a computer-readable storage medium storing instruction that, responsive to execution by the processing device, causes the processing device to perform operations including (Munkberg: abstract, A neural network-based rendering technique increases temporal stability and image fidelity of low sample count path tracing by optimizing a distribution of samples for rendering each image in a sequence. A sample predictor neural network learns spatio-temporal sampling strategies such as placing more samples in dis-occluded regions and tracking specular highlights. Temporal feedback enables a denoiser neural network to boost the effective input sample count and increases temporal stability. The initial uniform sampling step typically present in adaptive sampling algorithms is not needed. The sample predictor and denoiser operate at interactive rates to achieve significantly improved image quality and temporal stability compared with conventional adaptive sampling techniques. [0004]-[0005], [0025]-[0036], Figures 1A-1B; [0034] FIG. 1B illustrates a flowchart of a method 130 for temporally stable data restoration, in accordance with an embodiment. Although method 130 is described in the context of a processing unit, the method 130 may also be performed by a program, custom circuitry, or by a combination of custom circuitry and a program. For example, the method 130 may be executed by a GPU (graphics processing unit), CPU (central processing unit), or any processor capable of implementing the warped external recurrent neural network 100. Furthermore, persons of ordinary skill in the art will understand that any system that performs method 130 is within the scope and spirit of embodiments of the present invention.)
9. processing, by one or more machine-learning models, an input digital video having frames in a first resolution into an output digital video having the frames in a second resolution, (Munkberg: [0025] The encoder/decoder neural network model 110 receives input data at time t and warped external state from the previous iteration i.e., at time t−1. The input data includes artifacts that are removed during the reconstruction process to produce output data that approximates the input data without the artifacts. In the context of the following description, output data that approximates the input data without the artifacts has fewer artifacts compared with the input data. The warped external state from time t−1 includes warped reconstructed data from time t−1. [0035] At step 135, a sequence of input data including artifacts is received by the warped external recurrent neural network 100. The sequence includes a first input data frame and a second input data frame. In the context of the following description, an input data frame may include image data or other types of data. In an embodiment, the input data comprises rendered image frames.)
9. one or more machine-learning models including: one or more anti-aliasing blocks in an encoder of the one or more machine-learning models to generate low frequency features by removing high-frequency content from the frames of the input digital video; (Munkberg: [0032]-[0033] In an embodiment, a sequence of one sample-per-pixel temporally-unstable images is converted into a temporally-stable image sequence equivalent in quality to 16 (or some other value greater than 1) samples-per-pixel images. When performing antialiasing, the output image sequence does not suffer from the over-blurring, ghosting, flickering artifacts of current solutions and the warped external recurrent neural network 100 produce a temporally-stable image sequence in real time. Reconstruction performance of the warped external recurrent neural network 100 is dependent on image size and is independent of scene complexity. [0072]-[0074])
9. and one or more high-frequency shuttles configured to shuttle high frequency features from layers of the encoder to corresponding layers of a decoder of the one or more machine-learning models. (Munkberg: [0060] A smaller pixel footprint is used in kernel B because image features stored in hidden state are less likely to contain aliasing artifacts and the smaller kernel enables faster performance. In an embodiment, one spatially-varying kernel-predicted weight is removed from each from the (5×5) pixel kernel Ap and the (3×3) pixel kernel B to ensure that the encoder/decoder neural network model 110 output has 32 channels. [0063] Specifically, the outputs of the last three hierarchical levels of the encoder/decoder neural network model 110 are each fed through a small network to predict filter weights of a multi-scale kernel. In an embodiment, the small network is implemented using a fully-connected network (e.g., 1×1 convolutions) or a few layers with rectified linear units (ReLUs) between each layer to map the outputs to filter weights. In an embodiment, the filter weights are a set of unique 5×5 weights (e.g., filter kernel) for each pixel in a scaled image frame. In an embodiment, the scaled image frames are the outputs of the last three hierarchical levels of the encoder/decoder neural network model 110 input image. The last three hierarchical levels, working back from the output, correspond to a full resolution noisy image, a 2× downscaled version of the full resolution noisy image, and a 4× downscaled version of the full resolution noisy image, respectively. The unique filter kernels are applied at each respective pixel in the scaled image frames to produce filtered current frames. The filtered current frames are combined by the combiner function 120 to produce a multi-scale filtered current frame. In an embodiment, the filtered current frames are combined by upscaling the filtered current frames to full resolution and summing the resulting three full resolution multi-scale filtered current frames. In an embodiment, the upscaled filtered current frames Fi are combined using a weighted sum operation sum(Fi*wi), where the weight wi for each pixel is determined during training instead of simply computing sum(Fi) for the upscaled filtered current frames.)
Consider Claim 15.
Munkberg teaches:
15. One or more computer-readable storage media storing instructions that, responsive to execution by a processing device, causes the processing device to perform operations comprising: (Munkberg: abstract, A neural network-based rendering technique increases temporal stability and image fidelity of low sample count path tracing by optimizing a distribution of samples for rendering each image in a sequence. A sample predictor neural network learns spatio-temporal sampling strategies such as placing more samples in dis-occluded regions and tracking specular highlights. Temporal feedback enables a denoiser neural network to boost the effective input sample count and increases temporal stability. The initial uniform sampling step typically present in adaptive sampling algorithms is not needed. The sample predictor and denoiser operate at interactive rates to achieve significantly improved image quality and temporal stability compared with conventional adaptive sampling techniques. [0004]-[0005], [0025]-[0036], Figures 1A-1B; [0034] FIG. 1B illustrates a flowchart of a method 130 for temporally stable data restoration, in accordance with an embodiment. Although method 130 is described in the context of a processing unit, the method 130 may also be performed by a program, custom circuitry, or by a combination of custom circuitry and a program. For example, the method 130 may be executed by a GPU (graphics processing unit), CPU (central processing unit), or any processor capable of implementing the warped external recurrent neural network 100. Furthermore, persons of ordinary skill in the art will understand that any system that performs method 130 is within the scope and spirit of embodiments of the present invention.)
15. processing, by one or more machine-learning models, an input digital video having frames in a first resolution into an output digital video having the frames in a second resolution, (Munkberg: [0025] The encoder/decoder neural network model 110 receives input data at time t and warped external state from the previous iteration i.e., at time t−1. The input data includes artifacts that are removed during the reconstruction process to produce output data that approximates the input data without the artifacts. In the context of the following description, output data that approximates the input data without the artifacts has fewer artifacts compared with the input data. The warped external state from time t−1 includes warped reconstructed data from time t−1. [0035] At step 135, a sequence of input data including artifacts is received by the warped external recurrent neural network 100. The sequence includes a first input data frame and a second input data frame. In the context of the following description, an input data frame may include image data or other types of data. In an embodiment, the input data comprises rendered image frames. [0051] In an embodiment, the encoder/decoder neural network model 110 utilizes specialized tensor cores within a GPU, such as the tensor cores within the PPU 300 shown in FIG. 3. To take advantage of the tensor cores, 16-bit tensors and weights are used. In an embodiment, the slowest layers of an encoder/decoder neural network model 110 are the outermost, high resolution layers. The cost of a 3×3 convolution layer at 1080 pixel resolution varies with input and output channel counts. In particular, little variation occurs as output channel count changes from 8 to 64, but a significant slowdown occurs at 40 input channels.)
15. the processing employing one or more anti-aliasing blocks configured to perform operations including: (Munkberg: [0032]-[0033] In an embodiment, a sequence of one sample-per-pixel temporally-unstable images is converted into a temporally-stable image sequence equivalent in quality to 16 (or some other value greater than 1) samples-per-pixel images. When performing antialiasing, the output image sequence does not suffer from the over-blurring, ghosting, flickering artifacts of current solutions and the warped external recurrent neural network 100 produce a temporally-stable image sequence in real time. Reconstruction performance of the warped external recurrent neural network 100 is dependent on image size and is independent of scene complexity.)
15. downsampling the frames using a convolutional layer of the one or more machine-learning models; (Munkberg: [0060] A smaller pixel footprint is used in kernel B because image features stored in hidden state are less likely to contain aliasing artifacts and the smaller kernel enables faster performance. In an embodiment, one spatially-varying kernel-predicted weight is removed from each from the (5×5) pixel kernel Ap and the (3×3) pixel kernel B to ensure that the encoder/decoder neural network model 110 output has 32 channels.)
15. filtering the downsampled frames using a low-pass filter; (Munkberg: [0063] Specifically, the outputs of the last three hierarchical levels of the encoder/decoder neural network model 110 are each fed through a small network to predict filter weights of a multi-scale kernel. In an embodiment, the small network is implemented using a fully-connected network (e.g., 1×1 convolutions) or a few layers with rectified linear units (ReLUs) between each layer to map the outputs to filter weights. In an embodiment, the filter weights are a set of unique 5×5 weights (e.g., filter kernel) for each pixel in a scaled image frame. In an embodiment, the scaled image frames are the outputs of the last three hierarchical levels of the encoder/decoder neural network model 110 input image. The last three hierarchical levels, working back from the output, correspond to a full resolution noisy image, a 2× downscaled version of the full resolution noisy image, and a 4× downscaled version of the full resolution noisy image, respectively. The unique filter kernels are applied at each respective pixel in the scaled image frames to produce filtered current frames. The filtered current frames are combined by the combiner function 120 to produce a multi-scale filtered current frame. In an embodiment, the filtered current frames are combined by upscaling the filtered current frames to full resolution and summing the resulting three full resolution multi-scale filtered current frames. In an embodiment, the upscaled filtered current frames Fi are combined using a weighted sum operation sum(Fi*wi), where the weight wi for each pixel is determined during training instead of simply computing sum(Fi) for the upscaled filtered current frames.)
15. and subsampling the filtered downsampled frames; (Munkberg: [0064] Additionally, a predicted temporal 5×5 pixel kernel, is applied to the warped external state including the denoised previous frame to produce filtered warped external state that is then combined with the multi-scale filtered current frame to produce a reconstructed image frame that approximates the image frame without artifacts. [0065] FIG. 2D illustrates a block diagram of a temporal adaptive sampling and denoising system 200, in accordance with an embodiment. The warped external recurrent neural network 200 includes a sample map estimator neural network model 210, a renderer 205, a denoiser neural network model and combiner 220, and a temporal warp function 215. Although the temporal adaptive sampling and denoising system 200 is described in the context of processing units, one or more of the sample map estimator neural network model 210, denoiser neural network model and combiner 220, and the temporal warp function 215 may be performed by a program, custom circuitry, or by a combination of custom circuitry and a program.)
15. and outputting the output digital video having the frames in the second resolution. (Munkberg: [0050] . In an embodiment, the convolutional layers of the encoder portion are N×N, each followed by a 2×2 max pooling layer, where N=32, 64, 96, 128, 160, and 160 in a feed-forward sequence with the output of each max pooling layer input to each convolutional layer. In an embodiment, strided convolutions are used instead of pooling. In an embodiment, each stage of the decoder portion of the encoder/decoder neural network model 110 uses a nearest upsampling layer followed by two convolutional layers. In an embodiment, a pair of convolutional layers of the decoder portion are N×N, each proceeded by a 2×2 nearest upsampling layer, where N=160, 128, 96, 64, and 32 in a feed-forward sequence with the output of each upsampling layer input to each convolutional layer pair. In an embodiment, the output of each encoder stage is propagated to the corresponding decoder stage via residual skip connections and accumulated with the output of an upsampling layer. The residual skip connections improve reconstruction quality and may enable faster training of deep convolutional networks by improving the back propagation of gradients. [0067] The denoiser neural network model and combiner 220 is applied to adaptively sampled rendered images that may include artifacts. The denoiser neural network model and combiner 220 receives rendered image frames (e.g., adaptive samples) output by the renderer 205 or receives rendered image frames from other sources to produce reconstructed images. The rendered image frame (time=t) and the warped external state for the reconstructed image frame from the previous timestep (time=t−1) are processed by the denoiser neural network model and combiner 220 to produce external state including the reconstructed image frame (time=t). In an embodiment, the external state comprises features of the rendered image frame that are extracted by the denoiser neural network model and combiner 220. The reconstructed image frame has a higher effective sample count compared with the rendered image frame, meaning the reconstructed image frame appears to have fewer artifacts compared with the rendered image frame. The external state is transmitted by the recursive feedback loop and warped by the temporal warp function 215 according to per-pixel difference data (e.g., motion vectors) to produce warped external state including a processed warped reconstructed image frame (time=t−1). The warping aligns the external state to the next image frame in the sequence to be rendered.)
Consider Claim 2.
Munkberg teaches:
2. The method as described in claim 1, wherein the optical flow maps are bi-directional optical flow maps predicted using an optical flow estimator of the one or more machine-learning models. (Munkberg: [0037] At step 145, the external state is warped by the temporal warp function 115, using difference data corresponding to changes between the first input data frame and the second input data frame (e.g., optical flow, motion vectors, or the like), to produce warped external state. Warping the external state anchors individual characteristics or features to regions within the data frames. The warped external state enables improved tracking over time by integrating information associated with changing features over multiple frames in a sequence, producing more temporally stable and higher quality reconstructed data. At step 150, the second input data frame is processed, based on the warped external state, using the layers of the neural network model to produce a reconstructed second data frame that approximates the second input data frame without artifacts. [0081] In an embodiment, the sample map is produced using direct prediction, by combining all output features into a single component gray-scale image using a final convolutional layer. The sample map may be normalized to reach a desired average sample count:
PNG
media_image1.png
59
444
media_image1.png
Greyscale
where M is the number of pixels in the image, n is the average number of samples per pixel, and s(p) is the unnormalized output of the network. In an embodiment, the normalization calculation is similar to a softmax operation. For training the temporal adaptive sampling and denoising system 200, implementing normalization by applying a softmax operation over the unrolled image tensor may improve stability of the gradient computation for backpropagation as opposed to chaining the individual operations.)
Consider Claim 3.
Munkberg teaches:
3. The method as described in claim 1, wherein the learning is performed using a recurrent neural network (RNN) of the one or more machine-learning models. (Munkberg: [0023] A neural network structure, namely a warped external recurrent neural network, may be used for reconstructing data. The warped external recurrent neural network can be applied to reconstruct image data and non-image data, such as audio data, data acquired by depth sensors (e.g., lidar, radar, and the like), data acquired by temperature sensors, density data (e.g., medical imaging and geological), and the like. The warped external recurrent neural network is not recurrent at each layer and has a feed-forward flow—only warping external state output by the final layer. In contrast, in a conventional recurrent neural network, hidden state generated at each layer is provided as a feedback input to the generating layer. The warped external recurrent neural network is trained end-to-end to minimize the errors, between pairs of aliased and antialiased images. During supervised training, the warped external recurrent neural network learns to identify aliased image features and to adaptively remove (i.e., filter out) the undesirable artifacts (e.g., aliased image features) and/or modify areas with missing and incorrect information. After being trained, the warped external recurrent neural network may be deployed to reconstruct data. [0024] FIG. 1A illustrates a block diagram of a warped external recurrent neural network 100, in accordance with an embodiment. The warped external recurrent neural network includes an encoder/decoder neural network model 110, a temporal warp function 115, and a combiner function 120. Although the warped external recurrent neural network 100 is described in the context of processing units, one or more of the encoder/decoder neural network model 110, the temporal warp function 115, and the combiner function 120 may be performed by a program, custom circuitry, or by a combination of custom circuitry and a program. For example, the encoder/decoder neural network model 110 may be implemented by a GPU (graphics processing unit), CPU (central processing unit), or any processor capable of implementing layers of a neural network. Furthermore, persons of ordinary skill in the art will understand that any system that performs the operations of the warped external recurrent neural network 100 is within the scope and spirit of embodiments of the present invention.)
Consider Claim 4.
Munkberg teaches:
4. The method as described in claim 1, wherein the warping is performed using a backward warping layer of the one-or-more machine-learning models that is guided by the optical flow maps. (Munkberg: [0028] The temporal warp function 115 also receives per-datum differences (difference data) corresponding to the input data at time t and the external state from time t−1. Note, that when the input data is image data, the per-datum differences may be motion flow or motion vectors. The temporal warp function 115 warps the external state based on the per-datum differences to produce the warped external state for time t−1. The warping aligns the external state from time t−1 to the input data at time t. In the context of the following description, the external state includes hidden state from only the last layer of the encoder/decoder neural network model 110 and reconstructed data that approximates the input data without artifacts. The hidden state generated by the last layer may implicitly include information from previous input data frames that is incorporated into the hidden state for each timestep. In an embodiment, the external state comprises features of the input data that are extracted by the encoder/decoder neural network model 110. In an embodiment, the encoder/decoder neural network model 110 receives the warped external state including reconstructed data from time t−1 and the combiner function 120 receives the reconstructed data from time t−1. [0036] At step 140, the first input data frame is processed using layers of a neural network model to produce external state including a reconstructed first data frame that approximates the first input data frame without artifacts. In an embodiment, the first input data frame is processed by layers of the encoder/decoder neural network model 110. Hidden state generated by a particular one of the layers during processing of the first input data frame is not provided as an input to the particular one of the layers to process the second input data frame. In other words, each of the layers does not incorporate a feedback connection to use hidden state generated for a previous frame to generate outputs and/or hidden state for the current frame. [0037] At step 145, the external state is warped by the temporal warp function 115, using difference data corresponding to changes between the first input data frame and the second input data frame (e.g., optical flow, motion vectors, or the like), to produce warped external state. Warping the external state anchors individual characteristics or features to regions within the data frames. The warped external state enables improved tracking over time by integrating information associated with changing features over multiple frames in a sequence, producing more temporally stable and higher quality reconstructed data. At step 150, the second input data frame is processed, based on the warped external state, using the layers of the neural network model to produce a reconstructed second data frame that approximates the second input data frame without artifacts.)
Consider Claim 5.
Munkberg teaches:
5. The method as described in claim 1, wherein the processing augments the frames of the input digital video with the warped temporal-aware features aligned by optical flow. (Munkberg: [0028] The temporal warp function 115 also receives per-datum differences (difference data) corresponding to the input data at time t and the external state from time t−1. Note, that when the input data is image data, the per-datum differences may be motion flow or motion vectors. The temporal warp function 115 warps the external state based on the per-datum differences to produce the warped external state for time t−1. The warping aligns the external state from time t−1 to the input data at time t. In the context of the following description, the external state includes hidden state from only the last layer of the encoder/decoder neural network model 110 and reconstructed data that approximates the input data without artifacts. The hidden state generated by the last layer may implicitly include information from previous input data frames that is incorporated into the hidden state for each timestep. In an embodiment, the external state comprises features of the input data that are extracted by the encoder/decoder neural network model 110. In an embodiment, the encoder/decoder neural network model 110 receives the warped external state including reconstructed data from time t−1 and the combiner function 120 receives the reconstructed data from time t−1. [0036] At step 140, the first input data frame is processed using layers of a neural network model to produce external state including a reconstructed first data frame that approximates the first input data frame without artifacts. In an embodiment, the first input data frame is processed by layers of the encoder/decoder neural network model 110. Hidden state generated by a particular one of the layers during processing of the first input data frame is not provided as an input to the particular one of the layers to process the second input data frame. In other words, each of the layers does not incorporate a feedback connection to use hidden state generated for a previous frame to generate outputs and/or hidden state for the current frame. [0037] At step 145, the external state is warped by the temporal warp function 115, using difference data corresponding to changes between the first input data frame and the second input data frame (e.g., optical flow, motion vectors, or the like), to produce warped external state. Warping the external state anchors individual characteristics or features to regions within the data frames. The warped external state enables improved tracking over time by integrating information associated with changing features over multiple frames in a sequence, producing more temporally stable and higher quality reconstructed data. At step 150, the second input data frame is processed, based on the warped external state, using the layers of the neural network model to produce a reconstructed second data frame that approximates the second input data frame without artifacts. [0058] Therefore, motion data, such as per-pixel motion vectors generated by modern real-time applications or other sources, such as through optical flow analysis, are used to warp the external state. In an embodiment, the external state is warped with a reprojection filter θ to align the external state at t−1 to the current data frame at t. In an embodiment, the reprojection filter is bi-linear, bi-cubic, or any other type of filter. When the warped external recurrent neural network 100 is used to perform antialiasing, the warping aligns pixels from prior image frames to the same surfaces in the current image frame, removing range limitations imposed by a receptive field of the encoder/decoder neural network model 110 and enabling small kernels to find correspondences with prior image frames. At the beginning of each training iteration, difference data for the input data frame are used by the temporal warp function 115 to fetch pixels from the previous reconstructed data frame, using bilinear filtering to smooth the fetched results. Relying on the difference data enables efficient searching to identify correspondences between features. The warped (i.e., reprojected) external state coincides with the final antialiased image. However, the warping is not restricted to antialiasing and applies to any hidden state containing higher level representations of images or other data.)
Consider Claim 6.
Munkberg teaches:
6. The method as described in claim 5, wherein the processing further comprises generating the output digital video having frames in the second resolution using the frames of the input digital video augmented with the temporal-aware features aligned by the optical flow. (Munkberg: [0028] The temporal warp function 115 also receives per-datum differences (difference data) corresponding to the input data at time t and the external state from time t−1. Note, that when the input data is image data, the per-datum differences may be motion flow or motion vectors. The temporal warp function 115 warps the external state based on the per-datum differences to produce the warped external state for time t−1. The warping aligns the external state from time t−1 to the input data at time t. In the context of the following description, the external state includes hidden state from only the last layer of the encoder/decoder neural network model 110 and reconstructed data that approximates the input data without artifacts. The hidden state generated by the last layer may implicitly include information from previous input data frames that is incorporated into the hidden state for each timestep. In an embodiment, the external state comprises features of the input data that are extracted by the encoder/decoder neural network model 110. In an embodiment, the encoder/decoder neural network model 110 receives the warped external state including reconstructed data from time t−1 and the combiner function 120 receives the reconstructed data from time t−1. [0036] At step 140, the first input data frame is processed using layers of a neural network model to produce external state including a reconstructed first data frame that approximates the first input data frame without artifacts. In an embodiment, the first input data frame is processed by layers of the encoder/decoder neural network model 110. Hidden state generated by a particular one of the layers during processing of the first input data frame is not provided as an input to the particular one of the layers to process the second input data frame. In other words, each of the layers does not incorporate a feedback connection to use hidden state generated for a previous frame to generate outputs and/or hidden state for the current frame. [0037] At step 145, the external state is warped by the temporal warp function 115, using difference data corresponding to changes between the first input data frame and the second input data frame (e.g., optical flow, motion vectors, or the like), to produce warped external state. Warping the external state anchors individual characteristics or features to regions within the data frames. The warped external state enables improved tracking over time by integrating information associated with changing features over multiple frames in a sequence, producing more temporally stable and higher quality reconstructed data. At step 150, the second input data frame is processed, based on the warped external state, using the layers of the neural network model to produce a reconstructed second data frame that approximates the second input data frame without artifacts. [0058] Therefore, motion data, such as per-pixel motion vectors generated by modern real-time applications or other sources, such as through optical flow analysis, are used to warp the external state. In an embodiment, the external state is warped with a reprojection filter θ to align the external state at t−1 to the current data frame at t. In an embodiment, the reprojection filter is bi-linear, bi-cubic, or any other type of filter. When the warped external recurrent neural network 100 is used to perform antialiasing, the warping aligns pixels from prior image frames to the same surfaces in the current image frame, removing range limitations imposed by a receptive field of the encoder/decoder neural network model 110 and enabling small kernels to find correspondences with prior image frames. At the beginning of each training iteration, difference data for the input data frame are used by the temporal warp function 115 to fetch pixels from the previous reconstructed data frame, using bilinear filtering to smooth the fetched results. Relying on the difference data enables efficient searching to identify correspondences between features. The warped (i.e., reprojected) external state coincides with the final antialiased image. However, the warping is not restricted to antialiasing and applies to any hidden state containing higher level representations of images or other data.)
Consider Claims 7 and 12.
Munkberg teaches:
7. The method as described in claim 6, wherein the generating the output digital video is performed using a generative adversarial network (GAN) of the one or more machine-learning models. / 12. The computing device as described in claim 9, wherein the one or more machine-learning models include a generative adversarial network (GAN). (Munkberg: [0050] FIG. 2A illustrates a block diagram of the encoder/decoder neural network model 110 from FIG. 1A, in accordance with an embodiment. In an embodiment, each stage of the encoder portion of the encoder/decoder neural network model 110 uses one convolutional layer and a pooling layer. In an embodiment, 3×3 convolutions are used in the encoder/decoder neural network model 110. In an embodiment, the convolutional layers of the encoder portion are N×N, each followed by a 2×2 max pooling layer, where N=32, 64, 96, 128, 160, and 160 in a feed-forward sequence with the output of each max pooling layer input to each convolutional layer. In an embodiment, strided convolutions are used instead of pooling. In an embodiment, each stage of the decoder portion of the encoder/decoder neural network model 110 uses a nearest upsampling layer followed by two convolutional layers. In an embodiment, a pair of convolutional layers of the decoder portion are N×N, each proceeded by a 2×2 nearest upsampling layer, where N=160, 128, 96, 64, and 32 in a feed-forward sequence with the output of each upsampling layer input to each convolutional layer pair. In an embodiment, the output of each encoder stage is propagated to the corresponding decoder stage via residual skip connections and accumulated with the output of an upsampling layer. The residual skip connections improve reconstruction quality and may enable faster training of deep convolutional networks by improving the back propagation of gradients. [0160] During training, data flows through the DNN in a forward propagation phase until a prediction is produced that indicates a label corresponding to the input. If the neural network does not correctly label the input, then errors between the correct label and the predicted label are analyzed, and the weights are adjusted for each feature during a backward propagation phase until the DNN correctly labels the input and other inputs in a training dataset. Training complex neural networks requires massive amounts of parallel computing performance, including floating-point multiplications and additions that are supported by the PPU 300. Inferencing is less compute-intensive than training, being a latency-sensitive process where a trained neural network is applied to new inputs it has not seen before to classify images, translate speech, and generally infer new information.)
Consider Claims 8 and 13.
Munkberg teaches:
8. The method as described in claim 7, wherein the generative adversarial network (GAN) is jointly trained with the flow guided feature propagation module. / 13. The computing device as described in claim 12, wherein the generative adversarial network (GAN) is jointly trained with a flow guided feature propagation module. (Munkberg: [0050] FIG. 2A illustrates a block diagram of the encoder/decoder neural network model 110 from FIG. 1A, in accordance with an embodiment. In an embodiment, each stage of the encoder portion of the encoder/decoder neural network model 110 uses one convolutional layer and a pooling layer. In an embodiment, 3×3 convolutions are used in the encoder/decoder neural network model 110. In an embodiment, the convolutional layers of the encoder portion are N×N, each followed by a 2×2 max pooling layer, where N=32, 64, 96, 128, 160, and 160 in a feed-forward sequence with the output of each max pooling layer input to each convolutional layer. In an embodiment, strided convolutions are used instead of pooling. In an embodiment, each stage of the decoder portion of the encoder/decoder neural network model 110 uses a nearest upsampling layer followed by two convolutional layers. In an embodiment, a pair of convolutional layers of the decoder portion are N×N, each proceeded by a 2×2 nearest upsampling layer, where N=160, 128, 96, 64, and 32 in a feed-forward sequence with the output of each upsampling layer input to each convolutional layer pair. In an embodiment, the output of each encoder stage is propagated to the corresponding decoder stage via residual skip connections and accumulated with the output of an upsampling layer. The residual skip connections improve reconstruction quality and may enable faster training of deep convolutional networks by improving the back propagation of gradients. [0160] During training, data flows through the DNN in a forward propagation phase until a prediction is produced that indicates a label corresponding to the input. If the neural network does not correctly label the input, then errors between the correct label and the predicted label are analyzed, and the weights are adjusted for each feature during a backward propagation phase until the DNN correctly labels the input and other inputs in a training dataset. Training complex neural networks requires massive amounts of parallel computing performance, including floating-point multiplications and additions that are supported by the PPU 300. Inferencing is less compute-intensive than training, being a latency-sensitive process where a trained neural network is applied to new inputs it has not seen before to classify images, translate speech, and generally infer new information.)
Consider Claim 10-11.
Munkberg teaches:
10. The computing device as described in claim 9, wherein the anti-aliasing blocks are included along with respective convolutional layers of an encoder of the one or more machine-learning models. / 11. The computing device as described in claim 9, wherein the anti-aliasing blocks are configured to removes changes in pixel intensity over a threshold amount from the frames of the input digital video. (Munkberg: [0060] A smaller pixel footprint is used in kernel B because image features stored in hidden state are less likely to contain aliasing artifacts and the smaller kernel enables faster performance. In an embodiment, one spatially-varying kernel-predicted weight is removed from each from the (5×5) pixel kernel Ap and the (3×3) pixel kernel B to ensure that the encoder/decoder neural network model 110 output has 32 channels. [0063] Specifically, the outputs of the last three hierarchical levels of the encoder/decoder neural network model 110 are each fed through a small network to predict filter weights of a multi-scale kernel. In an embodiment, the small network is implemented using a fully-connected network (e.g., 1×1 convolutions) or a few layers with rectified linear units (ReLUs) between each layer to map the outputs to filter weights. In an embodiment, the filter weights are a set of unique 5×5 weights (e.g., filter kernel) for each pixel in a scaled image frame. In an embodiment, the scaled image frames are the outputs of the last three hierarchical levels of the encoder/decoder neural network model 110 input image. The last three hierarchical levels, working back from the output, correspond to a full resolution noisy image, a 2× downscaled version of the full resolution noisy image, and a 4× downscaled version of the full resolution noisy image, respectively. The unique filter kernels are applied at each respective pixel in the scaled image frames to produce filtered current frames. The filtered current frames are combined by the combiner function 120 to produce a multi-scale filtered current frame. In an embodiment, the filtered current frames are combined by upscaling the filtered current frames to full resolution and summing the resulting three full resolution multi-scale filtered current frames. In an embodiment, the upscaled filtered current frames Fi are combined using a weighted sum operation sum(Fi*wi), where the weight wi for each pixel is determined during training instead of simply computing sum(Fi) for the upscaled filtered current frames. [0091] FIG. 2H illustrates a flowchart of a method 250 for training the temporal adaptive sampling and denoising system 200 of FIG. 2D, in accordance with an embodiment. Although method 250 is described in the context of a processing unit, the method 250 may also be performed by a program, custom circuitry, or by a combination of custom circuitry and a program. [0092] The method 250 includes steps 235, 237, and 239 from the method 230. At step 235, guide data corresponding to a sequence of target image frames in a training dataset is received by the temporal adaptive sampling and denoising system 200. At step 255, the renderer 205 renders the image according to the sample map. At step 260, the denoiser neural network model and combiner 220 processes the rendered image to produce external state including a reconstructed image frame. [0093] At step 265, the parameter adjustment unit 245 determines if the training is complete. A loss function may be computed by the parameter adjustment unit 245 to measure distances (i.e., differences or gradients) between the target output data and the reconstructed data. The temporal adaptive sampling and denoising system 200 is deemed to be sufficiently trained when the reconstructed data generated for the input data from the training dataset match the target output data or a threshold accuracy is achieved for the training dataset.)
Consider Claim 16.
Munkberg teaches:
16. The one or more computer-readable storage media as described in claim 15, wherein the one or more anti-aliasing blocks are included along with respective convolutional layers of an encoder of the one or more machine-learning models. (Munkberg: [0028] The temporal warp function 115 also receives per-datum differences (difference data) corresponding to the input data at time t and the external state from time t−1. Note, that when the input data is image data, the per-datum differences may be motion flow or motion vectors. The temporal warp function 115 warps the external state based on the per-datum differences to produce the warped external state for time t−1. The warping aligns the external state from time t−1 to the input data at time t. In the context of the following description, the external state includes hidden state from only the last layer of the encoder/decoder neural network model 110 and reconstructed data that approximates the input data without artifacts. The hidden state generated by the last layer may implicitly include information from previous input data frames that is incorporated into the hidden state for each timestep. In an embodiment, the external state comprises features of the input data that are extracted by the encoder/decoder neural network model 110. In an embodiment, the encoder/decoder neural network model 110 receives the warped external state including reconstructed data from time t−1 and the combiner function 120 receives the reconstructed data from time t−1. [0036] At step 140, the first input data frame is processed using layers of a neural network model to produce external state including a reconstructed first data frame that approximates the first input data frame without artifacts. In an embodiment, the first input data frame is processed by layers of the encoder/decoder neural network model 110. Hidden state generated by a particular one of the layers during processing of the first input data frame is not provided as an input to the particular one of the layers to process the second input data frame. In other words, each of the layers does not incorporate a feedback connection to use hidden state generated for a previous frame to generate outputs and/or hidden state for the current frame. [0037] At step 145, the external state is warped by the temporal warp function 115, using difference data corresponding to changes between the first input data frame and the second input data frame (e.g., optical flow, motion vectors, or the like), to produce warped external state. Warping the external state anchors individual characteristics or features to regions within the data frames. The warped external state enables improved tracking over time by integrating information associated with changing features over multiple frames in a sequence, producing more temporally stable and higher quality reconstructed data. At step 150, the second input data frame is processed, based on the warped external state, using the layers of the neural network model to produce a reconstructed second data frame that approximates the second input data frame without artifacts. [0058] Therefore, motion data, such as per-pixel motion vectors generated by modern real-time applications or other sources, such as through optical flow analysis, are used to warp the external state. In an embodiment, the external state is warped with a reprojection filter θ to align the external state at t−1 to the current data frame at t. In an embodiment, the reprojection filter is bi-linear, bi-cubic, or any other type of filter. When the warped external recurrent neural network 100 is used to perform antialiasing, the warping aligns pixels from prior image frames to the same surfaces in the current image frame, removing range limitations imposed by a receptive field of the encoder/decoder neural network model 110 and enabling small kernels to find correspondences with prior image frames. At the beginning of each training iteration, difference data for the input data frame are used by the temporal warp function 115 to fetch pixels from the previous reconstructed data frame, using bilinear filtering to smooth the fetched results. Relying on the difference data enables efficient searching to identify correspondences between features. The warped (i.e., reprojected) external state coincides with the final antialiased image. However, the warping is not restricted to antialiasing and applies to any hidden state containing higher level representations of images or other data.)
Consider Claim 17-18.
Munkberg teaches:
17. The one or more computer-readable storage media as described in claim 15, wherein the filtering the downsampled frames using the low-pass filter removes high-frequency content from the frames of the input digital video. / 18. The one or more computer-readable storage media as described in claim 15, wherein the filtering the downsampled frames using the low-pass filter removes changes in pixel intensity over a threshold amount from the frames of the input digital video. (Munkberg: [0032]-[0033], [0072] The sample map estimator neural network model 210 processes the input data for each image in a sequence based on the warped external state that is also used by the denoiser neural network model and combiner 220. The input data comprises per-pixel guide such as depth, surface normal vector, and motion vectors. Generally, guide data are data defined per-pixel to help the reconstruction process. However, in contrast with conventional adaptive sampling techniques, the guide data does not include sampled color data, such as uniformly sampled one sample-per-pixel “seed” color data. Instead of receiving a low resolution version of the image, the sample map estimator neural network model 210 receives the warped reconstructed image included in the warped external state. Therefore, an initial rendering pass is not required to generate low resolution the color data. In an embodiment, the guide data does not include three-dimensional primitives. [0073] As previously described, the input to the denoiser neural network model and combiner 220 is the adaptively sampled noisy image (e.g., adaptive samples), rendered as dictated by the sample map. The denoiser neural network model and combiner 220 also receives the same inputs as provided to the sample map estimator neural network model 210, namely the input data and the warped external state. The sample map estimator neural network model 210 and the denoiser neural network model and combiner 220 are jointly trained to learn the parameters for generating the sample maps and the external state, respectively. [0074] FIG. 2E illustrates high resolution (e.g., 1024 spp) reference image, the reference image at 4 spp, and a sample map and a reconstructed image generated by the temporal adaptive sampling and denoising system 200, in accordance with an embodiment. The reference image is sampled at 4 spp to produce the 4 sample-per-pixel image. Within the sample map, the darker portions indicate pixels to be sampled at a low rate and the brighter portions indicate pixels to be sampled at higher rates. Note that the brighter portions correspond to edges and areas of more detail or greater color variation in the reference image and the darker portions correspond to areas in the reference image of constant or similar color.)
Consider Claim 14 and 19.
Munkberg teaches:
14. The computing device as described in claim 13, wherein the flow guided feature propagation module is configured to: predict optical flow maps from the frames of the input digital video; learn temporal-aware features based on the optical flow maps and pixels of the frames; and warp the temporal-aware features guided by the optical flow maps. / 19. The one or more computer-readable storage media as described in claim 15, wherein the processing includes using a a flow guided feature propagation module, the processing including: predicting optical flow maps from the frames of the input digital video; learning temporal-aware features based on the optical flow maps and pixels of the frames; and warping the temporal-aware features guided by the optical flow maps. (Munkberg: [0037] At step 145, the external state is warped by the temporal warp function 115, using difference data corresponding to changes between the first input data frame and the second input data frame (e.g., optical flow, motion vectors, or the like), to produce warped external state. Warping the external state anchors individual characteristics or features to regions within the data frames. The warped external state enables improved tracking over time by integrating information associated with changing features over multiple frames in a sequence, producing more temporally stable and higher quality reconstructed data. At step 150, the second input data frame is processed, based on the warped external state, using the layers of the neural network model to produce a reconstructed second data frame that approximates the second input data frame without artifacts. [0081] In an embodiment, the sample map is produced using direct prediction, by combining all output features into a single component gray-scale image using a final convolutional layer. The sample map may be normalized to reach a desired average sample count:
PNG
media_image1.png
59
444
media_image1.png
Greyscale
where M is the number of pixels in the image, n is the average number of samples per pixel, and s(p) is the unnormalized output of the network. In an embodiment, the normalization calculation is similar to a softmax operation. For training the temporal adaptive sampling and denoising system 200, implementing normalization by applying a softmax operation over the unrolled image tensor may improve stability of the gradient computation for backpropagation as opposed to chaining the individual operations. [0058] Therefore, motion data, such as per-pixel motion vectors generated by modern real-time applications or other sources, such as through optical flow analysis, are used to warp the external state. In an embodiment, the external state is warped with a reprojection filter θ to align the external state at t−1 to the current data frame at t. In an embodiment, the reprojection filter is bi-linear, bi-cubic, or any other type of filter. When the warped external recurrent neural network 100 is used to perform antialiasing, the warping aligns pixels from prior image frames to the same surfaces in the current image frame, removing range limitations imposed by a receptive field of the encoder/decoder neural network model 110 and enabling small kernels to find correspondences with prior image frames. At the beginning of each training iteration, difference data for the input data frame are used by the temporal warp function 115 to fetch pixels from the previous reconstructed data frame, using bilinear filtering to smooth the fetched results. Relying on the difference data enables efficient searching to identify correspondences between features. The warped (i.e., reprojected) external state coincides with the final antialiased image. However, the warping is not restricted to antialiasing and applies to any hidden state containing higher level representations of images or other data.)
Consider Claim 20.
Munkberg teaches:
20. The one or more computer-readable storage media as described in claim 15, wherein the one or more machine-learning models include: one or more anti-aliasing blocks in an encoder of the one or more machine-learning models to generate low frequency features by removing high-frequency content from the frames of the input digital video; and one or more high-frequency shuttles configured to shuttle high frequency features from layers of the encoder to corresponding layers of a decoder of the one or more machine-learning models. (Munkberg: [0060] A smaller pixel footprint is used in kernel B because image features stored in hidden state are less likely to contain aliasing artifacts and the smaller kernel enables faster performance. In an embodiment, one spatially-varying kernel-predicted weight is removed from each from the (5×5) pixel kernel Ap and the (3×3) pixel kernel B to ensure that the encoder/decoder neural network model 110 output has 32 channels. [0063] Specifically, the outputs of the last three hierarchical levels of the encoder/decoder neural network model 110 are each fed through a small network to predict filter weights of a multi-scale kernel. In an embodiment, the small network is implemented using a fully-connected network (e.g., 1×1 convolutions) or a few layers with rectified linear units (ReLUs) between each layer to map the outputs to filter weights. In an embodiment, the filter weights are a set of unique 5×5 weights (e.g., filter kernel) for each pixel in a scaled image frame. In an embodiment, the scaled image frames are the outputs of the last three hierarchical levels of the encoder/decoder neural network model 110 input image. The last three hierarchical levels, working back from the output, correspond to a full resolution noisy image, a 2× downscaled version of the full resolution noisy image, and a 4× downscaled version of the full resolution noisy image, respectively. The unique filter kernels are applied at each respective pixel in the scaled image frames to produce filtered current frames. The filtered current frames are combined by the combiner function 120 to produce a multi-scale filtered current frame. In an embodiment, the filtered current frames are combined by upscaling the filtered current frames to full resolution and summing the resulting three full resolution multi-scale filtered current frames. In an embodiment, the upscaled filtered current frames Fi are combined using a weighted sum operation sum(Fi*wi), where the weight wi for each pixel is determined during training instead of simply computing sum(Fi) for the upscaled filtered current frames. [0091] FIG. 2H illustrates a flowchart of a method 250 for training the temporal adaptive sampling and denoising system 200 of FIG. 2D, in accordance with an embodiment. Although method 250 is described in the context of a processing unit, the method 250 may also be performed by a program, custom circuitry, or by a combination of custom circuitry and a program. [0092] The method 250 includes steps 235, 237, and 239 from the method 230. At step 235, guide data corresponding to a sequence of target image frames in a training dataset is received by the temporal adaptive sampling and denoising system 200. At step 255, the renderer 205 renders the image according to the sample map. At step 260, the denoiser neural network model and combiner 220 processes the rendered image to produce external state including a reconstructed image frame. [0093] At step 265, the parameter adjustment unit 245 determines if the training is complete. A loss function may be computed by the parameter adjustment unit 245 to measure distances (i.e., differences or gradients) between the target output data and the reconstructed data. The temporal adaptive sampling and denoising system 200 is deemed to be sufficiently trained when the reconstructed data generated for the input data from the training dataset match the target output data or a threshold accuracy is achieved for the training dataset.)
Conclusion
The prior art made of record in form PTO-892 and not relied upon is considered pertinent to applicant's disclosure.
PNG
media_image2.png
178
895
media_image2.png
Greyscale
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAHMINA ANSARI whose telephone number is 571-270-3379. The examiner can normally be reached on IFP Flex - Monday through Friday 9 to 5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’NEAL MISTRY can be reached on 313-446-4912. The fax phone numbers for the organization where this application or proceeding is assigned are 571-273-8300 for regular communications and 571-273-8300 for After Final communications. TC 2600’s customer service number is 571-272-2600.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to the receptionist whose telephone number is 571-272-2600.
2674
/Tahmina Ansari/
July 23, 2026
/TAHMINA N ANSARI/Primary Examiner, Art Unit 2674
Prosecution Insights