DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 7-8, 12, 14-18 are rejected under 35 U.S.C. 103 as being unpatentable over Kaskela (US 20230196662) in view of Thomas (US 20230148225).
Regarding claim 1:
Kaskela discloses: A supersampling method (¶ [0004] “FIG. 1 illustrates an example temporal upsampling pipeline”; ¶ [0045] “In at least one embodiment, an upscaling system 100 such as is illustrated in FIG. 1, can be used to increase a resolution of one or more images, such as images or video frames in a sequence or video stream, as part of a deep learning-based super sampling or super-resolution process”), comprising:
generating a current rendered image frame by performing jittered sampling on a three- dimensional (3D) scene, based on sub-pixels of low-resolution pixels for the current rendered image frame (¶ [0047] “…an upsampling process can consider a sub-pixel jitter that can be applied on a per-frame basis”; ¶ [0048] “…jitter-aware upsampling and accumulating samples at an upsampled resolution. In at least one embodiment, this jitter offset data can be provided, along with a current input video frame and a prior inferred frame, as input to an upscaler 108 including at least one neural network in order to infer a higher quality upsampled image 110 than would be produced by an upsampling algorithm alone. In at least one embodiment, this upsampling essentially shifts jitter offsets 122 and per-frame samples so that they are aligned with a history buffer that may be at a higher resolution”; ¶ [0066] “…jittering may be performed between frames or images in a sequence, wherein a center point of a color determination is shifted slightly to another point in this pixel…this can correspond to sample point offset from a pixel center by a sub-pixel offset.”; ¶ [0067] “…a process for generating an image of a sequence can be performed, wherein an image (or video frame) is rendered at a first resolution. In at least one embodiment…this resolution may be one that is native to a rendering engine or that provides desired performance. In at least one embodiment, this rendered image can be upsampled to a second, higher resolution using jitter-aware upsampling, where a determined sub-pixel offset is determined for this image that can be different from a prior offset for one or more prior images in this sequence”);
generating a current warped image frame by warping a previous output image frame, based on a motion vector map corresponding to a difference between the current rendered image frame and a previous rendered image frame (¶ [0048] “…this upsampling essentially shifts jitter offsets 122 and per-frame samples so that they are aligned with a history buffer that may be at a higher resolution”; ¶ [0049] “…this upscaled image 110 can be provided as input to a neural network 112 to determine one or more blending factors or blending weights. In at least one embodiment, neural network 112 also receives as input a prior high resolution image in this sequence that is warped and provided to neural network 112 along with this upscaled image 110”; ¶ [0051] “…generation of a frame using such an approach can involve an application providing to a reconstruction algorithm a low resolution jittered input image and associated jitter values, low resolution backward motion vectors per individual input image pixels, and other quantities, such as exposure value and a depth buffer. In at least one embodiment, these low resolution input (backward) motion vectors can be used to warp a previous frame output image to align with geometry in a current time step”;
While Kaskela further discloses: “this upsampling essentially shifts jitter offsets 122 and per-frame samples so that they are aligned with a history buffer that may be at a higher resolution” in ¶ [0048].
Kaskela does not specifically teach: generating a current shifted image frame by shifting pixels of the current warped image frame, based on a change in sampling positions based on the jittered sampling.
However, in a related field, Thomas, teaches: generating a current shifted image frame by shifting pixels of the current warped image frame, based on a change in sampling positions based on the jittered sampling (¶ [0003] “…The previously accumulated frame is warped using renderer generated velocity/motion vectors to align it with the current frame before accumulation”; ¶ [0049] “…to warp the previous output within the history data 4102 using motion vectors within the velocity data”; ¶ [0454] “…The warped history data is then aligned with the current frame data to generate aligned history data”); and
Kaskela further teaches: generating a current output image frame, based on the current rendered image frame and the current shifted image frame (¶ [0051] “…a high resolution output image for a current frame can be created as: output=w*(upsampled current frame input image)+(1−w)*(warped previous output image)”; ¶ [0049] “…jitter offset data 122 can be provided as input to this blending component 114 as well. In at least one embodiment, this blending of a current image with a prior (or historical) image of a sequence can help with temporal convergence to a nice, sharp, high-resolution output image 116, which can then be provided for presentation via a display 120 or other such presentation mechanism”; and Thomas similarly teaches similar technique in FIG. 40 and ¶ [0444] “Temporally amortized supersampling is performed by combining the current frame and the previous output frame warped with the current motion vectors”).
Therefore, it would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Kaskela to incorporate the teachings of Thomas by including: generating a current shifted image frame by shifting pixels of the current warped image frame, based on a change in sampling positions based on the jittered sampling in order to match the current frame’s jittered sampling positions. Both Kaskela and Thomas address temporal anti-aliasing and temporal supersampling using per-frame jitter and motion-vector warping of history to align with a current frame prior to accumulation/blending.
Regarding claim 2:
Kaskela further teaches: wherein the generating of the current rendered image frame comprises:
performing the jittered sampling by selectively sampling a plurality of sampling points of the 3D scene corresponding to corresponding sub-pixels of each of the low-resolution pixels for the current rendered image frame (¶ [0066] “…jittering may be performed between frames or images in a sequence, wherein a center point of a color determination is shifted slightly to another point in this pixel. In at least one embodiment, this can correspond to sample point offset from a pixel center by a sub-pixel offset”; ¶ [0067] “…where a determined sub-pixel offset is determined for this image that can be different from a prior offset for one or more prior images in this sequence. In at least one embodiment, this upsampled image can be provided to a neural network to determine one or more blending weights to be used to blend this upscaled image with a prior image”. Also see ¶ [0003] of Thomas).
Regarding claim 3:
Kaskela further teaches: further comprising: alternately sampling the plurality of sampling points, based on a predetermined period (¶ [0066] “…jittering may be performed between frames or images in a sequence, wherein a center point of a color determination is shifted slightly to another point in this pixel. In at least one embodiment, this can correspond to sample point offset from a pixel center by a sub-pixel offset”; ¶ [0067] “…where a determined sub-pixel offset is determined for this image that can be different from a prior offset for one or more prior images in this sequence. In at least one embodiment, this upsampled image can be provided to a neural network to determine one or more blending weights to be used to blend this upscaled image with a prior image” Also see ¶ [0003] of Thomas. Kaskela teaches that the jitter varies between frames and can do so according to a predetermined pattern or sequence, which reads on alternating sampling based on a predetermined period).
Regarding claim 4:
Kaskela further teaches: wherein the generating of the current shifted image frame comprises:
shifting the pixels of the current warped image frame based on a shift pattern synchronized to the change in the sampling positions based on the jittered sampling (Kaskela in ¶ [0048] “… upsampling essentially shifts jitter offsets 122 and per-frame samples so that they are aligned with a history buffer that may be at a higher resolution”; ¶ [0051] “these low resolution input (backward) motion vectors can be used to warp a previous frame output image to align with geometry in a current time step. In at least one embodiment, this low resolution current frame image is upsampled to a resolution of an output image 218 using an upsampling algorithm”; and Thomas in ¶ [0003] “Temporal Anti-aliasing (TAA) is an anti-aliasing technique in which the renderer jitters the camera every frame to sample different coordinates in screen space”; ¶ [0454] “…The history data is warped using the velocity data to generate warped history data. The warped history data is then aligned with the current frame data to generate aligned history data”. Thus Kaskela teaches jitter-synchronized shifting/alignment concept (shifting jitter offset/per-frame samples to align with high resolution history) and the warped-history context, while Thomas teaches explicit per-frame jitter changes plus warping and alignment of the history to the current frame prior to accumulation).
Regarding claim 7:
Kaskela further teaches: wherein the supersampling method further comprises generating, using a neural network-based neural supersampling model, the previous output image frame (Kaskela ¶ [0049] “…a copy of this high resolution output image 116 can also be stored to a history buffer 118, or other such storage location, for blending with a subsequently generated image in this sequence”; ¶ [0050] “…this data after any pre-processing is provided as input to a neural network, or other deep learning (DL)-based generator 212, which can analyze this data to determine pixel specific weightings for each pixel location in an image to be generated… this post-processor can also output information to be stored to high resolution color and historical buffer 214 for use in generating a subsequent image in a current sequence”. Thus, Kaskela teaches generating a previous output image frame using a neural network-bases supersampling pipeline) and
wherein the generating of the current output image frame comprises generating, using the neural network-based neural supersampling model, the current output image frame (¶ [0051] “…a high resolution output image for a current frame can be created as: output=w*(upsampled current frame input image)+(1−w)*(warped previous output image)”; ¶ [0053] “…upon current frame input image and warped previous frame output image. In at least one embodiment, wherever an upsampled current image has significantly different values from a warped previous frame output image, and thus would appear very different when displayed, a neural network can predict a high valued weighting factor w, giving more importance to an upsampled current frame input image. In at least one embodiment, when a current image has similar values to a warped previous frame output image, and thus would appear very similar when displayed, a neural network can predict a low valued weighting factor w, giving more importance to a warped previous frame output image”. Thus, Kaskela teaches generating the current output image frame using a neural-network-based supersampling model).
Regarding claim 8:
Kaskela further teaches: wherein the generating of the current output image frame comprises:
generating input data by combining the current rendered image frame and the current shifted image frame (Kaskela in ¶ [0049] “…neural network 112 also receives as input a prior high resolution image in this sequence that is warped and provided to neural network 112 along with this upscaled image 110”; ¶ [0050] “…this data after any pre-processing is provided as input to a neural network, or other deep learning (DL)-based generator 212, which can analyze this data to determine pixel specific weightings for each pixel location in an image to be generated.” Thus, Kaskela teaches generating neural-network input data by combining current frame with a warped prior output/history image to the current shifted image frame. Also see Thomas in ¶ [0449] “…The input block 4108 receives input including history data 4102, velocity data 4104, the current frame 4106, and the jitter offset 4107. The input block 4108 includes a warping unit 4202 to warp the previous output within the history data 4102 using motion vectors within the velocity data 4104. The input block 4108 also include an upscaling unit 4203 to upscale the current frame 4106”)).
inputting the input data into a neural network-based neural supersampling model (Kaskela in ¶ [0050] “…this data after any pre-processing is provided as input to a neural network, or other deep learning (DL)-based generator 212, which can analyze this data to determine pixel specific weightings for each pixel location in an image to be generated.” And Thomas in ¶¶ [0446] – [0447] teaches that the input block is part of a neural network model used for temporally amortized supersampling).
Regarding claim 12:
Kaskela further teaches: wherein the sub-pixels of the low-resolution pixels for the current rendered image frame have sizes corresponding to high-resolution pixels (¶ [0065] “…an upsampling process can be performed for each individual pixel of a lower resolution rendered image. In at least one embodiment, an upscaling process might result in color information from that pixel being applied to a corresponding pixel region in an upscaled image that is larger in size. In at least one embodiment, this pixel in this upscaled image can be segmented (or mapped) into a number of individual pixels. In at least one embodiment, upsampling can be 4× upsampling, where each pixel of an input image is segmented into four higher resolution pixels”);
wherein the previous rendered image frame and the current rendered image frame correspond to a low-resolution image based on the low-resolution pixels (¶ [0046] “…an upsampling process can be performed for each individual pixel of a lower resolution rendered image”; ¶ [0047] “…an upscaler system 108 (which can take a form of a service, system, module, or device) can be used to upscale individual frames of a video or animation sequence. In at least one embodiment, an amount of upscaling to be performed can depend upon an initial resolution of a rendered image and a target resolution of display, such as going from 1080p to 4 k resolution.”), and
wherein the previous output image frame and the previous output image frame correspond to a high-resolution image based on the high-resolution pixels (¶ [0049] “…this blending of a current image with a prior (or historical) image of a sequence can help with temporal convergence to a nice, sharp, high-resolution output image 116, which can then be provided for presentation via a display 120 or other such presentation mechanism. In at least one embodiment, a copy of this high resolution output image 116 can also be stored to a history buffer 118, or other such storage location, for blending with a subsequently generated image in this sequence”).
Regarding claim 14:
Kaskela further teaches: A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the supersampling method of claim 1 (¶ [0454] “…a computer-readable storage medium is a non-transitory computer-readable storage medium”).
Regarding claims 15-18: the claims limitations are similar to those of claims 1-4; therefore, rejected in the same manner as applied above. Kaskela further discloses a system in FIG. 10.
Claim(s) 9 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Kaskela (US 20230196662) in view of Thomas (US 20230148225) and Vemulapalli (US 20190206026).
Regarding claim 9:
While Kaskela is operating with a jitter-conditioned shifted/aligned (history-derived) image contribution that is combined with the current frame for reconstruction in ¶¶ [0048] – [0050].
An while Thomas teaches a space to channel/depth shuffle unit that shuffles pixels from spatial dimension into a channel/depth dimension to facilitate inferencing in ¶ [0449].
However, Kaskela in view Thomas does not specifically teach: wherein the generating of the input data comprises: dividing the current shifted image frame into pixel sets corresponding to a low-resolution image by performing sub-sampling based on the jittered sampling; and combining the current rendered image frame and the pixel sets.
However, in a related field, Vemulapalli: wherein the generating of the input data comprises: dividing the current shifted image frame into pixel sets corresponding to a low-resolution image by performing sub-sampling based on the jittered sampling (¶ [0087] “…extracts shifted low-resolution grids from the image and places them into the channel dimension to obtain S.sub.s(I).sub.i,j,k=I.sub.si+k%s,sj+(k/s)%s,k/s.sub.2, with zero-based indexing, modulus “%” and integer division “/”.”; ¶ [0093] “…a high-resolution image 590 can be mapped to a low-resolution space by extracting low-resolution grids from the high-resolution image 590 and shifting the low-resolution grids into a channel dimension, to obtain a low-resolution mapping 592. The high-resolution image 590 can include a plurality of portions from which one or more low-resolution grids can be extracted. Each portion can correspond to one or more pixels”); and
combining the current rendered image frame and the pixel sets (¶ [0088] “The computing system can concatenate the low-resolution mapping 318 of the warped previous estimated high-resolution image with the current low-resolution image frame 306, and input the concatenated result into a machine-learned frame estimation model 320”).
Therefore, it would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Kaskela in view of Thomas to incorporate the teachings of Vemulapalli by including: dividing the current shifted image frame into pixel sets corresponding to a low-resolution image by performing sub-sampling based on the jittered sampling; and combining the current rendered image frame and the pixel sets in order to efficiently represent the shifted/history image contribution as pixel sets corresponding to a low-resolution image and combine them with current rendered frame for neural inference.
Regarding claim 13:
While Kaskela teaches that a temporal reconstruction algorithm receives low resolution backward motion vectors for the input image pixels and used those motion vectors to warp a previous frame output image to align with the current time step in ¶ [0051]. Kaskela further teaches that motion vectors per individual input image pixels can be used to warp a previous frame output image to align with geometry in a current time step in ¶ [0054].
However, Kaskela does not explicitly teach: upscaling the motion vector map corresponding to the difference between the current rendered image frame and the previous rendered image frame, based on a resolution of the previous output image frame.
However, Vemulapalli further teaches: further comprising:
upscaling the motion vector map corresponding to the difference between the current rendered image frame and the previous rendered image frame, based on a resolution of the previous output image frame (FIG. 3, ¶ [0085] “…the computing system can upscale the estimated flow-map 312 to obtain a high-resolution flow-map 314. For example, the computing system can treat the flow-map F.sup.LR as an image, and upscale it using bilinear interpolation with scaling factor s which results in an HR flow-map F.sup.HR=bi(F.sup.LR)∈[−1,1].sup.sH×sW×2”; ¶ [0086] “…The computing system can use the high-resolution flow-map 314 to warp a previous estimated high-resolution image frame 304 and obtain a warped previous estimated high-resolution image frame 316.”).
Claim(s) 5 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Kaskela (US 20230196662) in view of Thomas (US 20230148225) and Stine (US 20160005344).
Regarding claim 5:
Kaskela further teaches: wherein the generating of the current shifted image frame comprises:
selecting, based on the jittered sampling, a sampling target from among the sub-pixels of each of the low-resolution pixels of the for the current rendered image frame (¶ [0066] “…jittering may be performed between frames or images in a sequence, wherein a center point of a color determination is shifted slightly to another point in this pixel. In at least one embodiment, this can correspond to sample point offset from a pixel center by a sub-pixel offset”; ¶ [0051] “…generation of a frame using such an approach can involve an application providing to a reconstruction algorithm a low resolution jittered input image and associated jitter values”; and see Thomas in ¶ [0448] – ¶ [0449]);
and generating the current shifted image frame based on a shift operation from among the plurality of shift operations corresponding to a position of the sampling target (¶ [0048]”… upsampling essentially shifts jitter offsets 122 and per-frame samples so that they are aligned with a history buffer that may be at a higher resolution”)
Kaskela in view of Thomas does not specifically teach: wherein a plurality of shift operations corresponds to positions of the sub-pixels of each of the low-resolution pixels for the current rendered image frame.
However, in the same field of endeavor, Stine teaches: wherein a plurality of shift operations corresponds to positions of the sub-pixels of each of the low-resolution pixels for the current rendered image frame (¶ [0024] “…In step 116, multiple shifted images are defined from the source image and in accordance with the uprendering matrix. In some embodiments, each of the multiple shifted images corresponds with one of the elements of the uprendering matrix. The shifted images are copies of the source image having a resolution that is the same as the source image. The pixels in each shifted image are also consistent with the pixels of the source image but are shifted from the source image in the X/Y and U/V sampling”; ¶ [0029] “…The first shifted image (B) corresponds to the element (1,2) of the uprendering matrix with the pixels of the first shifted image (B) being shifted a half pixel in the X direction. Similarly, the second and third shifted images (C and D, respectively) correspond to the elements (2,1) and (2,2), respectively, of the uprendering matrix with the pixels of the second shifted image (C) being shifted a half pixel in the Y direction and the pixels of the third shifted image (D) being shifted a half pixel in both the X and Y directions”)
Therefore, it would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Kaskela in view of Thomas to incorporate the teachings of Stine by including: wherein a plurality of shift operations corresponds to positions of the sub-pixels of each of the low-resolution pixels for the current rendered image frame in order to realize jitter-phase dependent alignment/shift for temporal accumulation and reconstruction.
Regarding claim 19: the claims limitations are similar to those of claim 5; therefore, rejected in the same manner as applied above.
Allowable Subject Matter
Claims 6, 10-11, and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Relevant Art not Relied Upon
Bastos (US 7403208) teaches Jittered sub-pixel samples that are used to reduce aliasing during rendering in a graphics pipeline. An alternate embodiment includes storage of additional sub-pixel offset values. As the number of sub-pixel offset values increases, the period of the jitter pattern across the image also increases over multiple pixels. In contrast to a method using a fixed number of sub-pixel offset values, a number of sub-pixel offset values may be stored dependent on the resolution of the image to be displayed so that the jitter pattern is effectively non-periodic (related to claim 6).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WASSIM MAHROUKA whose telephone number is (571)272-2945. The examiner can normally be reached Monday-Thursday 8:00-5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen Koziol can be reached at (408) 918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WASSIM MAHROUKA/Primary Examiner, Art Unit 2665