DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5-6, 8-13 and 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Croxford et al. (US 20250111602 A1), hereinafter as Croxford, in view of NPL Hollander et al. (“Adaptive Supersampling for Deferred Anti-Aliasing”), hereinafter as Hollander.
Regarding claim 1, Croxford teaches A processor-implemented method (Abstract, “Example methods, apparatuses, and/or articles of manufacture are disclosed that may be implemented, in whole or in part, using one or more computing devices to enhance a rendered image.”), the method comprising: determining first color values of a portion of pixels of plural pixels, the portion of pixels being in a tile frame corresponding to a partial region of an input frame, by performing shading using a shader module on the portion of pixels included (Croxford teaches tile-based graphics processing pipeline, further teaches using shader core to generate shaded pixels in a tile, Figure 4, paragraph [0042] “Once a rendering pipeline has completed processing of a tile of render output, the rendered data for the tile is exported to memory 5 (e.g., to a frame buffer in memory 5) for storage. A next tile may then be processed by a rendering pipeline, and so on, until sufficient tiles have been processed to generate an entire render output (e.g., frame (image) to be displayed).”, paragraph [0044] “VRS parameters from block 402 may specify multiple different shading rates for a single tile to, for example, address different levels of complexity and detail within the single tile. …… shaded pixel values 406 may comprise, for example, multi-color channel signal intensity values, image depth parameters or surface normal parameters, or a combination thereof, associated with pixel locations in the current image frame.” And paragraph [0052] “a portion of variable rate shading parameters 402 to be applied at block 408 may be obtained from metadata stored in a shader core or iterator (e.g., a shader core or iterator in GPU 3 to render pixel values at block 404)”);
determining second color values of other pixels, the other pixels being pixels of the plural pixels not included in the portion of pixels of the tile frame, by performing neural network-based super-sampling processing on the other pixels (Croxford teaches the predicted pixel values for the upscaled portion as the second color value for the other pixel, implicitly teaches the predicted pixel of the upscaled portion not included in the first color portion of pixel, paragraph [0045] “ML super sample block 408 may generate upscaled content 410 using one or more trained neural networks for at least portions of an image frame containing shaded pixel values 406 ……Neural networks at block 408 may generate predicted pixel values for an upscaled portion at an output tensor.”);
and determining a rendered tile frame comprising the first color values of the portion of pixels and the second color values of the other pixels (Croxford teaches combining different super sampling model to determine the upscaled content in Figure 4, paragraph [0050] “an image frame containing shaded pixel values 406 may be partitioned such that a first portion of shaded pixel values in the image frame is upscaled according to block 408 (e.g., processed by a trained ML model) and a second portion is upscaled according to a simplified (e.g., less computationally intensive) process at block 412. …… Block 412 may apply a simplified upscaling process using, for example, nearest-neighbour interpolation (e.g., pixel replication), nearest neighbour-copying, bilinear interpolation and/or adaptive schemes, just to provide a few example upscaling techniques that may entail reduced computation requirements.”).
Croxford does not explicitly teach wherein the determining of the first color values of the portion of pixels comprises determining edge color values of pixels in an edge region by performing shading using the shader module on the pixels in the edge region of the tile frame. Hollander teaches wherein the determining of the first color values of the portion of pixels comprises determining edge color values of pixels in an edge region by performing shading using the shader module on the pixels in the edge region of the tile frame (Hollander teaches generating shading for each pixel in buffer S, further teaches deciding additional samples based on edge intensity threshold for each pixel in the buffer S, Page 4, 6-7 Paragraph, “we shade one sample for each pixel and store the result in a separate buffer S, which has, therefore, a resolution of w×h. Finally, we will derive the number of additional samples k that should be involved in shading a given pixel via two metrics. The first metric analyzes the contents of the G-buffer within a sampling window, and the second one looks at the actual shading values in S.”, Page 5-6, Last paragraph, “The second metric is based on the shading texture S. For each pixel the range r of luminance values is computed by looking at the four direct neighboring pixels, resulting in r = lmax −lmin. The final sample count is
PNG
media_image1.png
104
662
media_image1.png
Greyscale
“).
Croxford and Hollander are in the same field of endeavor, namely computer graphics, especially in the field of tile based adaptive super sampling. Hollander teaches an adaptive super-sampling method by generating an initial shading buffer with edge and non-edge portion of pixels to improve performance (Page 1, last paragraph, “Our approach concentrates the workload where needed and allows faster shading in various super sampling scenarios, especially when the shading cost per pixel is high.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Hollander with the method of Croxford to improve performance.
Regarding claim 2, Croxford in view of Hollander teach The method of claim 1, and further teach wherein the determining of the first color values further comprises determining non-edge color values of a non-edge portion of pixels in a non-edge region by performing shading using the shader module on the non-edge portion of pixels, the non-edge portion of pixels being included in the non-edge region, and the non-edge region being a region other than the edge region of the tile frame (Hollander teaches generating shading for each pixel in buffer S, further teaches deciding additional samples based on edge intensity threshold, Hollander’s equation of
PNG
media_image1.png
104
662
media_image1.png
Greyscale
teaches non-edge region of the buffer S, if the intensity difference is smaller than an edge threshold, this defines non-edge region, no additional sampling is needed).
Croxford and Hollander are in the same field of endeavor, namely computer graphics, especially in the field of tile based adaptive super sampling. Hollander teaches an adaptive super-sampling method by generating an initial shading buffer with edge and non-edge portion of pixels to improve performance (Page 1, last paragraph, “Our approach concentrates the workload where needed and allows faster shading in various super sampling scenarios, especially when the shading cost per pixel is high.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Hollander with the method of Croxford to improve performance.
Regarding claim 3, Croxford in view of Hollander teach The method of claim 2, and further teach wherein the determining of the second color values comprises performing the super-sampling processing based on the edge color values and the non-edge color values (Croxford teaches the machine learning super sample process in Figure 4, based on the shaded pixel values, including edge value and non-edge value, paragraph [0045] “ML super sample block 408 may generate upscaled content 410 using one or more trained neural networks for at least portions of an image frame containing shaded pixel values 406.”).
Regarding claim 5, Croxford in view of Hollander teach The method of claim 1, and further teach wherein the determining of the edge color values comprises determining all-edge color values of all pixels in the edge region by performing the shading using the shader module on the all pixels included in the edge region (Hollander teaches generating shading for each pixel in buffer S, further teaches deciding additional samples based on edge intensity threshold for each pixel in the buffer S, Page 4, 6-7 Paragraph, “we shade one sample for each pixel and store the result in a separate buffer S, which has, therefore, a resolution of w×h. Finally, we will derive the number of additional samples k that should be involved in shading a given pixel via two metrics. The first metric analyzes the contents of the G-buffer within a sampling window, and the second one looks at the actual shading values in S.”, Page 5-6, Last paragraph, “The second metric is based on the shading texture S. For each pixel the range r of luminance values is computed by looking at the four direct neighboring pixels, resulting in r = lmax −lmin. The final sample count is
PNG
media_image1.png
104
662
media_image1.png
Greyscale
“).
Croxford and Hollander are in the same field of endeavor, namely computer graphics, especially in the field of tile based adaptive super sampling. Hollander teaches an adaptive super-sampling method by generating an initial shading buffer with edge and non-edge portion of pixels to improve performance (Page 1, last paragraph, “Our approach concentrates the workload where needed and allows faster shading in various super sampling scenarios, especially when the shading cost per pixel is high.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Hollander with the method of Croxford to improve performance.
Regarding claim 6, Croxford in view of Hollander teach The method of claim 2, and further teach wherein a number of the non-edge portion of pixels to be shaded using the shader module is determined based on one of an image upscaling ratio, a frame rate, or an estimated amount of resource consumption (Croxford teaches a shading rate as the image upscaling ratio, multiple buffers of shaded pixel values based on the shading rate are generated, paragraph [0049] “block 408 may maintain multiple buffers to provide shaded pixel values to input tensors of an associated multiple trained neural networks to provide pixel values in a spatially upscaled image frame. Individual ones of the multiple buffers may then be selected to receive different portions of shaded pixel values of an image frame rendered at block 404 based, at least in part, on shading rates applied in rendering the respective ones of the one or more different portions……Shaded pixel values 406 in a portion of an image frame rendered by block 404 at a particular shading rate may then be loaded to an individual buffer/queue covering and/or associated with the particular shading rate to be then processed by an associated trained neural network.”).
Regarding claim 8, Croxford in view of Hollander teach The method of claim 1, and further teach wherein the determining of the first color values is performed by the shader module included in a graphics processing unit (GPU) (Croxford teaches using shader core to generate shaded pixels in a tile, the shader core is included in GPU, Figure 4, paragraph [0044] “shaded pixel values 406 may comprise, for example, multi-color channel signal intensity values, image depth parameters or surface normal parameters, or a combination thereof, associated with pixel locations in the current image frame.” And paragraph [0052] “a portion of variable rate shading parameters 402 to be applied at block 408 may be obtained from metadata stored in a shader core or iterator (e.g., a shader core or iterator in GPU 3 to render pixel values at block 404)”), and wherein the determining of the second color values is performed by a super-sampler module configured to perform the super-sampling processing (Croxford Figure 5, paragraph [0055-0057] “process 550 may, at least in part, be by executed by and/or implemented on GPU 3 (FIG. 1)…… If multiple neural networks are available to upscale different VRS regions in block 554 (e.g., as determined at diamond 556), any one of multiple neural networks (e.g., neural networks NN1, NN2, NN3 or NN4) may be selected to upscale a VRS region.”).
Regarding claim 9, Croxford in view of Hollander teach The method of claim 8, and further teach wherein the super-sampler module is included in the GPU (Croxford teaches multiple neural network super-sampler module in Figure 5, and further suggests the process can be implemented in GPU, paragraph [0055] “process 550 may, at least in part, be by executed by and/or implemented on GPU 3 (FIG. 1)).
Regarding claim 10, Croxford in view of Hollander teach A non-transitory computer-readable storage medium storing instructions that, when executed by a processor (Croxford paragraph [0083-0085] “In FIG. 7, computing device 802 may provide one or more sources of executable computer instructions in the form physical states and/or signals (e.g., stored in memory states)…… Memory 822 may comprise any non-transitory storage mechanism……Memory 822 may be utilized to store a program of executable computer instructions. For example, processor 820 may fetch executable instructions from memory and proceed to execute the fetched instructions.”), cause the processor to perform the method of claim 1 (Please refer the rejection rational for claim 1).
Regarding claim 11, Croxford teaches An electronic apparatus (Figure 7, paragraph [0080] “Example devices in FIG. 7 may comprise features, for example, of a client computing device and/or a server computing device”), comprising:
a control processor; and a graphics processing unit (GPU) configured to perform image rendering based on control of the control processor, wherein the GPU comprises: processors configured to execute instructions (paragraph [0080] “A “processor” and/or “processing circuit” for example, is understood to connote a specific structure such as a central processing unit (CPU), digital signal processor (DSP), graphics processing unit (GPU) and/or neural network processing unit (NPU), or a combination thereof, of a computing device which may include a control unit and an execution unit.”);
and a memory storing the instructions, wherein execution of the instructions configures the processors to (paragraph [0085] “Memory 822 may be utilized to store a program of executable computer instructions. For example, processor 820 may fetch executable instructions from memory and proceed to execute the fetched instructions.”):
determine first color values of a first portion of pixels, the first portion of pixels being pixels of plural pixels in a tile frame corresponding to a partial region of an input frame, by performing shading on the first portion of pixels (Croxford teaches tile-based graphics processing pipeline, further teaches using shader core to generate shaded pixels in a tile, Figure 4, paragraph [0042] “Once a rendering pipeline has completed processing of a tile of render output, the rendered data for the tile is exported to memory 5 (e.g., to a frame buffer in memory 5) for storage. A next tile may then be processed by a rendering pipeline, and so on, until sufficient tiles have been processed to generate an entire render output (e.g., frame (image) to be displayed).”, paragraph [0044] “VRS parameters from block 402 may specify multiple different shading rates for a single tile to, for example, address different levels of complexity and detail within the single tile. …… shaded pixel values 406 may comprise, for example, multi-color channel signal intensity values, image depth parameters or surface normal parameters, or a combination thereof, associated with pixel locations in the current image frame.” And paragraph [0052] “a portion of variable rate shading parameters 402 to be applied at block 408 may be obtained from metadata stored in a shader core or iterator (e.g., a shader core or iterator in GPU 3 to render pixel values at block 404)”);
and determine second color values of second pixels, the second portion of pixels being pixels of the plural pixels not included in the first portion, by performing neural network-based super-sampling processing on the second portion of pixels (Croxford teaches the predicted pixel values for the upscaled portion as the second color value for the other pixel, implicitly teaches the predicted pixel of the upscaled portion not included in the first color portion of pixel, paragraph [0045] “ML super sample block 408 may generate upscaled content 410 using one or more trained neural networks for at least portions of an image frame containing shaded pixel values 406 ……Neural networks at block 408 may generate predicted pixel values for an upscaled portion at an output tensor.”);
and a graphics memory configured to store a rendered tile frame comprising the first color values of the first portion of pixels and the second color values of the second portion of pixels (Croxford teaches combining different super sampling model to determine the upscaled content in Figure 4, and further teaches outputting a frame buffer to display unit, paragraph [0050] “an image frame containing shaded pixel values 406 may be partitioned such that a first portion of shaded pixel values in the image frame is upscaled according to block 408 (e.g., processed by a trained ML model) and a second portion is upscaled according to a simplified (e.g., less computationally intensive) process at block 412. …… Block 412 may apply a simplified upscaling process using, for example, nearest-neighbour interpolation (e.g., pixel replication), nearest neighbour-copying, bilinear interpolation and/or adaptive schemes, just to provide a few example upscaling techniques that may entail reduced computation requirements.”, and paragraph [0030] “A completed render output (e.g. frame) may be written in a frame buffer in memory 5 , from where it may be provided for display on a display device, such as a screen or printer, for example.”,),
Croxford does not explicitly teach and wherein the determining of the first color values comprises determining edge color values of pixels in an edge region by performing shading on the pixels in the edge region of the tile frame. Hollander teaches and wherein the determining of the first color values comprises determining edge color values of pixels in an edge region by performing shading on the pixels in the edge region of the tile frame (Hollander teaches generating shading for each pixel in buffer S, further teaches deciding additional samples based on edge intensity threshold for each pixel in the buffer S, Page 4, 6-7 Paragraph, “we shade one sample for each pixel and store the result in a separate buffer S, which has, therefore, a resolution of w×h. Finally, we will derive the number of additional samples k that should be involved in shading a given pixel via two metrics. The first metric analyzes the contents of the G-buffer within a sampling window, and the second one looks at the actual shading values in S.”, Page 5-6, Last paragraph, “The second metric is based on the shading texture S. For each pixel the range r of luminance values is computed by looking at the four direct neighboring pixels, resulting in r = lmax −lmin. The final sample count is
PNG
media_image1.png
104
662
media_image1.png
Greyscale
“).
Croxford and Hollander are in the same field of endeavor, namely computer graphics, especially in the field of tile based adaptive super sampling. Hollander teaches an adaptive super-sampling method by generating an initial shading buffer with edge and non-edge portion of pixels to improve performance (Page 1, last paragraph, “Our approach concentrates the workload where needed and allows faster shading in various super sampling scenarios, especially when the shading cost per pixel is high.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Hollander with the method of Croxford to improve performance.
Regarding claim 12, Croxford in view of Hollander teach The electronic apparatus of claim 11, and further teach wherein the determining of the first color values further comprises determining non-edge color values of a third portion of pixels in a non-edge region by performing shading on the third portion of pixels, the non-edge region being a region other than the edge region of the tile frame (Hollander teaches generating shading for each pixel in buffer S, further teaches deciding additional samples based on edge intensity threshold, Hollander’s equation of
PNG
media_image1.png
104
662
media_image1.png
Greyscale
teaches non-edge region of the buffer S, if the intensity difference is smaller than an edge threshold, this defines non-edge region, no additional sampling is needed).
Croxford and Hollander are in the same field of endeavor, namely computer graphics, especially in the field of tile based adaptive super sampling. Hollander teaches an adaptive super-sampling method by generating an initial shading buffer with edge and non-edge portion of pixels to improve performance (Page 1, last paragraph, “Our approach concentrates the workload where needed and allows faster shading in various super sampling scenarios, especially when the shading cost per pixel is high.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Hollander with the method of Croxford to improve performance.
Regarding claim 13, claim 13 has similar limitations as claim 3, therefore it is rejected under the same rationale as claim 3.
Regarding claim 15, Croxford in view of Hollander teach The electronic apparatus of claim 11, and further teach wherein the determining of the second color value comprises determining all-edge color values of all pixels in the edge region by performing the shading on the all pixels included in the edge region (Hollander teaches generating shading for each pixel in buffer S, further teaches deciding if additional samples are needed based on edge intensity threshold for each pixel in the buffer S, Croxford teaches deciding second color value based on first color values, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the buffer S of Hollander with the second color value calculation of Croxford, Hollander Page 4, 6-7 Paragraph, “we shade one sample for each pixel and store the result in a separate buffer S, which has, therefore, a resolution of w×h. Finally, we will derive the number of additional samples k that should be involved in shading a given pixel via two metrics. The first metric analyzes the contents of the G-buffer within a sampling window, and the second one looks at the actual shading values in S.”, Page 5-6, Last paragraph, “The second metric is based on the shading texture S. For each pixel the range r of luminance values is computed by looking at the four direct neighboring pixels, resulting in r = lmax −lmin. The final sample count is
PNG
media_image1.png
104
662
media_image1.png
Greyscale
“).
Croxford and Hollander are in the same field of endeavor, namely computer graphics, especially in the field of tile based adaptive super sampling. Hollander teaches an adaptive super-sampling method by generating an initial shading buffer with edge and non-edge portion of pixels to improve performance (Page 1, last paragraph, “Our approach concentrates the workload where needed and allows faster shading in various super sampling scenarios, especially when the shading cost per pixel is high.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Hollander with the method of Croxford to improve performance.
Regarding claim 16, Croxford in view of Hollander teach The electronic apparatus of claim 11, and further teach further comprising: a system configured to receive the tile frame rendered by the electronic apparatus from the graphics memory (Croxford paragraph [0029-0030] “ FIG. 1 shows a computer graphics processing system, comprising a host processor (CPU) 1 , a graphics processing unit (GPU) 3 , and a memory 5 for storing parameters required by and/or generated by the host processor 1 and graphics processor 3 …… A completed render output (e.g. frame) may be written in a frame buffer in memory 5 , from where it may be provided for display on a display device, such as a screen or printer, for example.”), and store an output frame obtained by combining rendered tile frames (Croxford paragraph [0059] “pixel values of a tile making up an image frame in frame buffer 552 may be placed in a buffer/queue for an input tensor of an associated neural network (from among networks NN 1 , NN 2 , NN 3 and NN 4 ) based on a highest shading rate applied in rendering the tile. With pixel values of tiles loaded to buffers/queues of input tensors of associated neural networks, neural networks NN 1 , NN 2 , NN 3 and NN 4 may execute concurrently to process/upscale pixel values from tiles in respective buffers/queues. Upscaled pixel values at output tensors of neural networks NN 1 , NN 2 , NN 3 and NN 4 may be assembled (e.g., according to stored metadata) at block 562 to provide an upscaled image frame 564”).
Claim(s) 4 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Croxford et al. (US 20250111602 A1), hereinafter as Croxford, in view of NPL Hollander et al. (“Adaptive Supersampling for Deferred Anti-Aliasing”), hereinafter as Hollander, further in view of NPL Anonymous, ("Tiling based Super-resolution Convolutional Neural Network for Image Processing"), hereinafter as Anonymous.
Regarding claim 4, Croxford in view of Hollander teach The method of claim 1, but fail to teach wherein the determining of the second color values comprises performing the super-sampling processing without using a third color value of a pixel included in another tile frame positioned around the tile frame. Anonymous teaches wherein the determining of the second color values comprises performing the super-sampling processing without using a third color value of a pixel included in another tile frame positioned around the tile frame (Page 3, Second and Third Paragraphs, “by tiling the input image into 256 x 256-sized tiles, the maximum memory activation of our network may be limited to that of the tile rather than the full image. To deploy a super-resolution convolutional neural network on to a memory constrained device, the convolutional neural network may need to operate on smaller sub-tiles of the original full-resolution image.”)
Croxford, Hollander and Anonymous are in the same field of endeavor, namely computer graphics, especially in the field of tile based adaptive super sampling. Anonymous teaches implementing the super resolution neural network based on a single tile to improve performance (Page 3, Last paragraph, “To reduce the maximum memory activation of the super-resolution convolutional neural network, we may
split input images into smaller tiles. The reduced memory activation may enable the super resolution convolutional neural network to solely leverage SRAM rather than spill into DRAM. Accessing 1 byte in DRAM may consume about 80pJ, while accessing the same byte in SRAM may only consume about 10pJ. Hence, preventing DRAM spills may drastically reduce the power consumption of image enhancement models such as the super-resolution convolutional neural network.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Anonymous with the method of Croxford and Hollander to improve performance.
Regarding claim 14, claim 14 has similar limitations as claim 4, therefore it is rejected under the same rationale as claim 4.
Claim(s) 7 and 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Croxford et al. (US 20250111602 A1), hereinafter as Croxford, in view of NPL Hollander et al. (“Adaptive Supersampling for Deferred Anti-Aliasing”), hereinafter as Hollander, further in view of Pohl et al. (US 20210390664 A1), hereinafter as Pohl.
Regarding claim 7, Croxford in view of Hollander teach The method of claim 1, and further teach further comprising, after the second color values are determined, …… the rendered tile frame including the first color values and the second color values …... (Croxford paragraph [0030] “A completed render output (e.g. frame) may be written in a frame buffer in memory 5 , from where it may be provided for display on a display device, such as a screen or printer, for example.”, paragraph [0059] “pixel values of a tile making up an image frame in frame buffer 552 may be placed in a buffer/queue for an input tensor of an associated neural network (from among networks NN 1 , NN 2 , NN 3 and NN 4 ) based on a highest shading rate applied in rendering the tile. With pixel values of tiles loaded to buffers/queues of input tensors of associated neural networks, neural networks NN 1 , NN 2 , NN 3 and NN 4 may execute concurrently to process/upscale pixel values from tiles in respective buffers/queues. Upscaled pixel values at output tensors of neural networks NN 1 , NN 2 , NN 3 and NN 4 may be assembled (e.g., according to stored metadata) at block 562 to provide an upscaled image frame 564”).
Croxford in view of Hollander fail to teach transmitting the rendered tile frame …… to a system memory. Pohl teaches transmitting the rendered tile frame …… to a system memory (Pohl paragraph [0099] “The copy engines 304 can be used to move data out of, into, and between the memory devices 326A-326D and memory that is external to the graphics processor 320 (e.g., system memory).”).
Croxford, Hollander and Pohl are in the same field of endeavor, namely computer graphics, especially in the field of tile based adaptive super sampling. Pohl teaches using a GPU processing cluster for neural network based adaptive super sampling to improve performance (Pohl paragraph [0235] “implementations of the disclosure improve rendered image quality at a lower cost in terms of hardware requirements and power consumption of the processor than previous solutions.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Pohl with the method of Croxford and Hollander to improve performance.
Regarding claim 17, Croxford teaches An electronic apparatus (Figure 7, paragraph [0080] “Example devices in FIG. 7 may comprise features, for example, of a client computing device and/or a server computing device”),comprising: a graphics processing unit (GPU) comprising: processors configured to execute instructions; and a memory storing the instructions, wherein execution of the instructions configures the processors to (Figure 3, paragraph [0033] “FIG. 3 shows graphics processing pipeline of an embodiment in more detail. As shown in FIG. 3, driver 4 sends commands and parameters for a draw call to tiling unit 500 in the form of Command Stream Frontend (CSF) instructions. Command Stream Frontend (CSF) 520 of graphics processor 3 receives these instructions, and sends appropriate commands over a Job Control Network (JCN) of the graphics processor 3”):
determine first color values of a first portion of pixels by performing shading on the first portion of pixels, the first portion of pixels being pixels of plural pixels included in a tile frame corresponding to a partial region of an input frame (Croxford teaches tile-based graphics processing pipeline, further teaches using shader core to generate shaded pixels in a tile, Figure 4, paragraph [0042] “Once a rendering pipeline has completed processing of a tile of render output, the rendered data for the tile is exported to memory 5 (e.g., to a frame buffer in memory 5) for storage. A next tile may then be processed by a rendering pipeline, and so on, until sufficient tiles have been processed to generate an entire render output (e.g., frame (image) to be displayed).”, paragraph [0044] “VRS parameters from block 402 may specify multiple different shading rates for a single tile to, for example, address different levels of complexity and detail within the single tile. …… shaded pixel values 406 may comprise, for example, multi-color channel signal intensity values, image depth parameters or surface normal parameters, or a combination thereof, associated with pixel locations in the current image frame.” And paragraph [0052] “a portion of variable rate shading parameters 402 to be applied at block 408 may be obtained from metadata stored in a shader core or iterator (e.g., a shader core or iterator in GPU 3 to render pixel values at block 404)”);
and determine second color values of a second portion of pixels, the second portion of pixels being pixels of the plural pixels not included in the first portion, by performing neural network-based super-sampling processing on the second portion of pixels (Croxford teaches the predicted pixel values for the upscaled portion as the second color value for a second portion of pixels, implicitly teaches the predicted pixel of the upscaled portion not included in the first color portion of pixels, paragraph [0045] “ML super sample block 408 may generate upscaled content 410 using one or more trained neural networks for at least portions of an image frame containing shaded pixel values 406 ……Neural networks at block 408 may generate predicted pixel values for an upscaled portion at an output tensor.”);
Croxford fails to teach a first graphics memory configured to store the determined color values of the portion of pixels; a second graphics memory configured to store the determined color values of the second portion of pixels;
and a system memory connected to the first graphics memory and the second graphics memory, wherein the determining of the first color values comprises determining edge color values of pixels in an edge region by performing shading on the pixels in the edge region of the tile frame.
Hollander teaches wherein the determining of the first color values comprises determining edge color values of pixels in an edge region by performing shading on the pixels in the edge region of the tile frame (Hollander teaches generating shading for each pixel in buffer S, further teaches deciding additional samples based on edge intensity threshold for each pixel in the buffer S, Page 4, 6-7 Paragraph, “we shade one sample for each pixel and store the result in a separate buffer S, which has, therefore, a resolution of w×h. Finally, we will derive the number of additional samples k that should be involved in shading a given pixel via two metrics. The first metric analyzes the contents of the G-buffer within a sampling window, and the second one looks at the actual shading values in S.”, Page 5-6, Last paragraph, “The second metric is based on the shading texture S. For each pixel the range r of luminance values is computed by looking at the four direct neighboring pixels, resulting in r = lmax −lmin. The final sample count is
PNG
media_image1.png
104
662
media_image1.png
Greyscale
“).
Croxford and Hollander are in the same field of endeavor, namely computer graphics, especially in the field of tile based adaptive super sampling. Hollander teaches an adaptive super-sampling method by generating an initial shading buffer with edge and non-edge portion of pixels to improve performance (Page 1, last paragraph, “Our approach concentrates the workload where needed and allows faster shading in various super sampling scenarios, especially when the shading cost per pixel is high.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Hollander with the method of Croxford to improve performance.
Croxford in view of Hollander fail to teach a first graphics memory configured to store the determined color values of the portion of pixels; a second graphics memory configured to store the determined color values of the second portion of pixels; and a system memory connected to the first graphics memory and the second graphics memory. Pohl teaches a first graphics memory configured to store the determined color values of the portion of pixels; a second graphics memory configured to store the determined color values of the second portion of pixels; and a system memory connected to the first graphics memory and the second graphics memory (Pohl teaches a graphic processing engine cluster in Figure 3B, paragraph [0097-0099] “FIG. 3B illustrates a graphics processor 320 having a tiled architecture …… The graphics processor 300 may be configured with a non-uniform memory access (NUMA) system in which memory devices 326A-326D are coupled with associated graphics engine tiles 310A-310D …… The copy engines 304 can be used to move data out of, into, and between the memory devices 326A-326D and memory that is external to the graphics processor 320 (e.g., system memory).”).
Croxford, Hollander and Pohl are in the same field of endeavor, namely computer graphics, especially in the field of tile based adaptive super sampling. Pohl teaches using a GPU processing cluster for neural network based adaptive super sampling to improve performance (Pohl paragraph [0235] “implementations of the disclosure improve rendered image quality at a lower cost in terms of hardware requirements and power consumption of the processor than previous solutions.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Pohl with the method of Croxford and Hollander to improve performance.
Regarding claim 18, Croxford in view of Hollander and Pohl teach The electronic apparatus of claim 17, and further teach wherein the determining of the first color values further comprises determining non-edge color values of a third portion of pixels in a non-edge region by performing shading on the third portion of pixels of pixels, the non-edge region being a region other than the edge region of the tile frame (Hollander teaches generating shading for each pixel in buffer S, further teaches deciding additional samples based on edge intensity threshold, Hollander’s equation of
PNG
media_image1.png
104
662
media_image1.png
Greyscale
teaches non-edge region of the buffer S, if the intensity difference is smaller than an edge threshold, this defines non-edge region, no additional sampling is needed).
Croxford, Hollander and Pohl are in the same field of endeavor, namely computer graphics, especially in the field of tile based adaptive super sampling. Hollander teaches an adaptive super-sampling method by generating an initial shading buffer with edge and non-edge portion of pixels to improve performance (Page 1, last paragraph, “Our approach concentrates the workload where needed and allows faster shading in various super sampling scenarios, especially when the shading cost per pixel is high.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Hollander with the method of Croxford and Pohl to improve performance.
Regarding claim 19, claim 19 has similar limitations as claim 3, therefore it is rejected under the same rationale as claim 3.
Regarding claim 20, Croxford in view of Hollander and Pohl teach The electronic apparatus of claim 17, and further teach wherein the system memory is configured to store a rendered tile frame comprising the first color values and the second color values (Croxford teaches a frame buffer with combined upscaling results, further teaches outputting frame buffer to display device, Pohl teaches a copy engine to move data from graphics memory to system memory in paragraph [0099], Croxford paragraph [0030] “A completed render output (e.g. frame) may be written in a frame buffer in memory 5 , from where it may be provided for display on a display device, such as a screen or printer, for example.”, paragraph [0059] “pixel values of a tile making up an image frame in frame buffer 552 may be placed in a buffer/queue for an input tensor of an associated neural network (from among networks NN 1 , NN 2 , NN 3 and NN 4 ) based on a highest shading rate applied in rendering the tile. With pixel values of tiles loaded to buffers/queues of input tensors of associated neural networks, neural networks NN 1 , NN 2 , NN 3 and NN 4 may execute concurrently to process/upscale pixel values from tiles in respective buffers/queues. Upscaled pixel values at output tensors of neural networks NN 1 , NN 2 , NN 3 and NN 4 may be assembled (e.g., according to stored metadata) at block 562 to provide an upscaled image frame 564”).
Croxford, Hollander and Pohl are in the same field of endeavor, namely computer graphics, especially in the field of tile based adaptive super sampling. Pohl teaches using a GPU processing cluster for neural network based adaptive super sampling to improve performance (Pohl paragraph [0235] “implementations of the disclosure improve rendered image quality at a lower cost in terms of hardware requirements and power consumption of the processor than previous solutions.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Pohl with the method of Croxford and Hollander to improve performance.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Janis et al. (US 20230206394 A1) teaches a deep learning-based super sampling process.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIAOMING WEI whose telephone number is (571)272-3831. The examiner can normally be reached M-F 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XIAOMING WEI/Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611