Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-2, 7, 9, 10, 12, 14-17, and 20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1-3 and 14-15 of copending Application No. 18/404,438 (reference application) in view of Kaskela et al (US 20230196662 A1, hereinafter Kaskela). This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Although the claims at issue are not identical, they are not patently distinct from each other because the copending claims recite each limitation (or a trivial variation) of the current claims, with the exception of the currently-claimed generating a warped image frame at a second resolution higher than the first resolution (current claims 1, 16, and 17), processing the current image frame and the position-adjusted warped image frame through a neural network model (current claims 1, 16, and 17), wherein the generating of the current image frame comprises: performing the jittered sampling based on a first jitter offset that is predetermined for the first-resolution pixel area of the 3D scene (current claim 2), wherein the generating of the output image frame comprises: matching the position-adjusted warped image frame and the current image frame in dimension; generating a concatenated image by concatenating the current image frame and the position-adjusted warped image frame matched in dimension; and outputting the output image frame by inputting the concatenated image into the neural network model (current claim 7 and 20), wherein the neural network model is configured to receive a first jitter offset and apply a predetermined value corresponding to the first jitter offset to the current output image frame (current claim 12), wherein the neural network model is configured to receive a first jitter offset, and when the neural network model uses a structure of a kernel prediction network, output the current output image frame by applying a predetermined value corresponding to the first jitter offset to a filter of the kernel prediction network (current claim 14) wherein the predetermined value corresponding to the first jitter offset comprises at least one of the first jitter offset, a formula calculated using a value of the first jitter offset, and a value obtained through separate learning using the first jitter offset value as input (current claim 15).
In the same art of supersampling, Kaskela teaches generating a warped image frame at a second resolution higher than the first resolution ([0051] “to warp a previous frame output image to align with geometry in a current time step. In at least one embodiment, this low resolution current frame image is upsampled to a resolution of an output image 218 using an upsampling algorithm”), processing the current image frame and the position-adjusted warped image frame through a neural network model ([0050] “pre-processor 208 can perform any relevant processing on current color data from color buffer 202 or warped prior color data from warper 210. In at least one embodiment, this data after any pre-processing is provided as input to a neural network … to determine pixel specific weightings for each pixel location in an image to be generated … this generated data goes to a post-processor 216 … which can output a final high resolution color image 218”)
It would have been obvious to one of ordinary skill in the art prior to the effective filing date to have modified Application No. 18/404,438 to include the teachings of Kaskela. Doing so would provide an image quality that is comparable or exceeds native resolution rendering ([0049] “a reconstructed image quality from such a process is comparable or even exceeds native resolution rendering, at least in terms of details, temporal stability, and lack of general artifacts such as ghosting or lag”)
Claim mapping between current application and reference application No. 18/404,438:
Current Application 19/076,572
1
2
7
9
10
12
14
15
16
17
20
Reference Application 18/404,438
1
1
1
2
3
1
1
1
14
15
15
Limitation mapping between claim 1 of the current application and claim 1 of the reference application:
19/076,572 Claim 1
18/404,438 Claim 1
An image processing method comprising:
generating a current image frame at a first resolution by performing jittered sampling on a first-resolution pixel area of a three-dimensional (3D) scene;
generating a warped image frame at a second resolution higher than the first resolution, by warping a feedback image frame based on a motion vector corresponding to a difference between the current image frame and a previous image frame;
obtaining a position-adjusted warped image by adjusting a position of the warped image frame based on a sampling position change corresponding to the jittered sampling;
and generating a current output image frame by processing the current image frame and the position-adjusted warped image frame through a neural network model.
A supersampling method, comprising:
generating a current rendered image frame by performing jittered sampling on a three-dimensional (3D) scene, based on sub-pixels of low-resolution pixels for the current rendered image frame
generating a current warped image frame by warping a previous output image frame, based on a motion vector map corresponding to a difference between the current rendered image frame and a previous rendered image frame
generating a current shifted image frame by shifting pixels of the current warped image frame, based on a change in sampling positions based on the jittered sampling;
and generating a current output image frame, based on the current rendered image frame and the current shifted image frame;
selecting target pixels in the current shifted image frame based on the sampling positions of the jittered sampling; and replacing the target pixels with pixels of the current rendered image frame.
Regarding claim 2, Kaskela teaches wherein the generating of the current image frame comprises: performing the jittered sampling based on a first jitter offset that is predetermined for the first-resolution pixel area of the 3D scene. ([0048] “this can include jitter-aware upsampling and accumulating samples at an upsampled resolution. In at least one embodiment, this jitter offset data can be provided, along with a current input video frame and a prior inferred frame, as input to an upscaler 108”)
The motivation to combine for claim 2 is the same as for claim 1 provided above.
Regarding claim 7, Kaskela teaches wherein the generating of the output image frame comprises: matching the position-adjusted warped image frame and the current image frame in dimension (Fig 3A, [0055] “Upsampling module 308, which can upsample this output to a higher resolution, such as may correspond to a resolution of inputs 302 or a target resolution.”); generating a concatenated image by concatenating the current image frame and the position-adjusted warped image frame matched in dimension (338 of Fig 3B, [0055] “these higher resolution results can then be passed to a refinement network 310, which also received inputs 302 that can be at this same higher resolution.”); and outputting the output image frame by inputting the concatenated image into the neural network model ([0056] “In at least one embodiment, this refinement network 310 can be a shallow convolutional neural network (CNN) operating at full resolution, or at least a higher resolution than U-Net 306.”).
The motivation to combine for claim 7 is the same as for claim 1 provided above.
Below is a Limitation Mapping between claim 9 of the current application and claim 2 of the reference application.
19/076,572 Claim 9
18/404,438 Claim 2
The image processing method of claim 1,
wherein the generating of the current image frame comprises
generating the current image frame by performing jittered sampling on subpixels included in the first-resolution pixel area.
The supersampling method of claim 1, wherein the generating of the current rendered image frame comprises:
performing the jittered sampling by selectively sampling a plurality of sampling points of the 3D scene corresponding to corresponding sub-pixels of each of the low-resolution pixels for the current rendered image frame.
Below is a Limitation Mapping between claim 10 of the current application and claim 3 of the reference application, which includes limitations of its base claims of 1 and 2.
19/076,572 Claim 10
No. 18/404,438 Claim 3
The image processing method of claim 9,
wherein the generating of the current image frame comprises
performing jittered sampling by selectively sampling respective sampling points corresponding to the subpixels,
and the sampling points are sampled alternately based on a predetermined period.
The supersampling method of claim 2, further comprising:
performing the jittered sampling by selectively sampling a plurality of sampling points of the 3D scene corresponding to corresponding sub-pixels (from claim 2)
alternately sampling the plurality of sampling points, based on a predetermined period.
Regarding claim 12, Kaskela teaches wherein the neural network model is configured to receive a first jitter offset and apply a predetermined value corresponding to the first jitter offset to the current output image frame ([0048] “this jitter offset data can be provided, along with a current input video frame and a prior inferred frame, as input to an upscaler 108 including at least one neural network in order to infer a higher quality upsampled image 110 than would be produced by an upsampling algorithm alone”)
The motivation to combine for claim 12 is the same as for claim 1 provided above.
Regarding claim 14, Kaskela teaches wherein the neural network model is configured to receive a first jitter offset, and when the neural network model uses a structure of a kernel prediction network ([0045] “such an algorithm can also utilize a filtering kernel to produce a new, higher resolution output image from a set of inputs”, [0057] “a number of convolutional layers, as well as a corresponding kernel size and numbers of features, can vary within different refinement networks”), output the current output image frame by applying a predetermined value corresponding to the first jitter offset to a filter of the kernel prediction network ([0058] “a number of channels (e.g., three) for filter parameters such as anisotropic Gaussian filter parameters, at least one channel for adaptive jitter-aware blending of data for at least one historical frame and a current frame”).
The motivation to combine for claim 14 is the same as for claim 1 provided above.
Regarding claim 15, Kaskela teaches wherein the predetermined value corresponding to the first jitter offset comprises at least one of the first jitter offset ([0048] “this upsampling essentially shifts jitter offsets 122 and per-frame samples so that they are aligned with a history buffer that may be at a higher resolution.”), a formula calculated using a value of the first jitter offset, and a value obtained through separate learning using the first jitter offset value as input ([0362] “wherein the one or more neural networks are further to adjust one or more additional parameters relating to at least one of historical pixel data, filter parameter data, pixel jitter data, filtered pixel data, or warped pixel data.”).
The motivation to combine for claim 15 is the same as for claim 1 provided above.
Below is a Limitation Mapping between claim 16 of the current application and claim 14 of the reference application.
19/076,572 Claim 16
No. 18/404,438 Claim 14
A non-transitory computer-readable storage medium storing instructions executable by a processor, to perform:
A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the supersampling method of claim 1.
Below is a Limitation Mapping between claim 16 of the current application and claim 1 of the reference application.
19/076,572 Claim 16
No. 18/404,438 Claim 1
generating a current image frame at a first resolution by performing jittered sampling on a first-resolution pixel area of a three-dimensional (3D) scene,
generating a warped image frame at a second resolution higher than the first resolution, by warping a feedback image frame based on a motion vector corresponding to a difference between the current image frame and a previous image frame,
obtaining a position-adjusted warped image by adjusting a position of the warped image frame based on a sampling position change corresponding to the jittered sampling,
and generating a current output image frame by processing the current image frame and the position-adjusted warped image frame through a neural network model.
generating a current rendered image frame by performing jittered sampling on a three-dimensional (3D) scene, based on sub-pixels of low-resolution pixels for the current rendered image frame
generating a current warped image frame by warping a previous output image frame, based on a motion vector map corresponding to a difference between the current rendered image frame and a previous rendered image frame
generating a current shifted image frame by shifting pixels of the current warped image frame, based on a change in sampling positions based on the jittered sampling;
and generating a current output image frame, based on the current rendered image frame and the current shifted image frame.
Below is a Limitation Mapping between claim 17 of the current application and claim 15 of the reference application.
19/076,572 Claim 17
18/404,438 Claim 15
An electronic device comprising:
a processor configured to:
generate a current image frame at a first resolution by performing jittered sampling on a first-resolution pixel area of a three-dimensional (3D) scene based on a first jitter offset,
generate a warped image frame at a second resolution higher than the first resolution, by warping a feedback image frame based on a motion vector corresponding to a difference between the current image frame and a previous image frame,
obtain a position-adjusted warped image by adjusting a position of the warped image frame based on a sampling position change corresponding to the jittered sampling,
and generate an output image frame by processing the jittered sample image frame and the position-adjusted warped image frame through a neural network model; and a display configured to display the current output image frame.
An electronic device, comprising:
a memory storing instructions;
and a processor communicatively coupled to the memory, wherein the processor is configured to execute the instructions to:
generate a current rendered image frame by performing jittered sampling on a three-dimensional (3D) scene, based on sub-pixels of low-resolution pixels of the current rendered image frame;
generate a current warped image frame by warping a previous output image frame, based on a motion vector map corresponding to a difference between the current rendered image frame and a previous rendered image frame;
generate a current shifted image frame by shifting pixels of the current warped image frame, based on a change in sampling positions based on the jittered sampling;
and generate a current output image frame, based on the current rendered image frame and the current shifted image frame;
and an output device configured to display the previous output image frame and the current output image frame.
Regarding claim 20, Kaskela teaches wherein the generating of the output image frame comprises: matching the position-adjusted warped image frame and the current image frame in dimension (Fig 3A, [0055] “Upsampling module 308, which can upsample this output to a higher resolution, such as may correspond to a resolution of inputs 302 or a target resolution.”)
The motivation to combine for claim 20 is the same as for claim 1 provided above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 7-9, 11-12, and 14-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kaskela et al (US 20230196662 A1, hereinafter Kaskela) and Thomas et al (US 20230148225 A1, hereinafter Thomas).
Regarding claim 1, Kaskela teaches an image processing method comprising:
generating a current image frame at a first resolution by performing jittered sampling ([0047] “In at least one embodiment, an upsampling process can consider a sub-pixel jitter that can be applied on a per-frame basis”) on a first-resolution pixel area of a three-dimensional (3D) scene ([0067] “a process for generating an image of a sequence can be performed, wherein an image (or video frame) is rendered at a first resolution. In at least one embodiment … this resolution may be one that is native to a rendering engine or that provides desired performance”); generating a warped image frame at a second resolution higher than the first resolution, by warping a feedback image frame based on a motion vector ([0049] “neural network 112 also receives as input a prior high resolution image in this sequence that is warped”, [0050] “this warper 210 receives as input motion vector information for a current frame as stored in motion vector buffer 204”, [0051] “to warp a previous frame output image to align with geometry in a current time step. In at least one embodiment, this low resolution current frame image is upsampled to a resolution of an output image 218 using an upsampling algorithm”) corresponding to a difference between the current image frame and a previous image frame ([0045] “this prior historical color data can be warped based upon motion detected between this historical frame and this current frame, such as may be indicated by a set of motion vectors”); and generating a current output image frame by processing the current image frame and the position-adjusted warped image frame through a neural network model ([0050] “pre-processor 208 can perform any relevant processing on current color data from color buffer 202 or warped prior color data from warper 210. In at least one embodiment, this data after any pre-processing is provided as input to a neural network … to determine pixel specific weightings for each pixel location in an image to be generated … this generated data goes to a post-processor 216 … which can output a final high resolution color image 218”).
Kaskela fails to explicitly teach obtaining a position-adjusted warped image by adjusting a position of the warped image frame based on a sampling position change corresponding to the jittered sampling, but in related field of endeavor Thomas teaches obtaining a position-adjusted warped image by adjusting a position of the warped image frame based on a sampling position change corresponding to the jittered sampling ([0003] “The previously accumulated frame is warped using renderer generated velocity/motion vectors to align it with the current frame before accumulation”, [0447] “The previously generated output includes at least the immediate previous frame (frame N−1), which is warped using the velocity data 4104 to align the frame with the current frame 4106”, [0449] “ warp the previous output within the history data 4102 using motion vectors within the velocity data 4104.”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have modified Kaskela to include obtaining a position-adjusted warped image by adjusting a position of the warped image frame based on a sampling position change corresponding to the jittered sampling as taught by Thomas. Doing so would allow processing at lower resolution and assist with processing costs ([0485] “the upscaled pipeline 4720 allows for processing at a lower resolution, thus assisting with processing costs”)
Regarding claim 2, Kaskela as modified by Thomas teaches the image processing method of claim 1, and Kaskela further teaches wherein the generating of the current image frame comprises: performing the jittered sampling based on a first jitter offset that is predetermined for the first-resolution pixel area of the 3D scene. ([0048] “this can include jitter-aware upsampling and accumulating samples at an upsampled resolution. In at least one embodiment, this jitter offset data can be provided, along with a current input video frame and a prior inferred frame, as input to an upscaler 108”)
Regarding claim 3, Kaskela as modified by Thomas teaches the image processing method of claim 1, and Thomas further teaches wherein the obtaining of the position-adjusted warped image comprises: adjusting positions of pixels of the warped image frame so that an area of pixels of the warped image frame corresponds to the current image frame ([0447] “the immediate previous frame (frame N−1), which is warped using the velocity data 4104 to align the frame with the current frame 4106”).
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Kaskela and Thomas to include wherein the obtaining of the position-adjusted warped image comprises: adjusting positions of pixels of the warped image frame so that an area of pixels of the warped image frame corresponds to the current image frame as taught by Thomas. Doing so would allow processing at lower resolution and assist with processing costs ([0485] “the upscaled pipeline 4720 allows for processing at a lower resolution, thus assisting with processing costs”)
Regarding claim 4, Kaskela as modified by Thomas teaches the image processing method of claim 3, and Kaskela further teaches wherein the obtaining of the position-adjusted warped image comprises: dividing the first-resolution pixel area of the current image frame into subpixels ([0066] “this can correspond to sample point offset from a pixel center by a sub-pixel offset. In at least one embodiment, a pixel analysis region (e.g., a 3×3 pixel analysis region) may still be used to determine color information for a given pixel”); obtaining a second jitter offset value corresponding to a sampling position adjusted so that positions of the subpixels of the current image frame are included in pixel areas of the warped image frame ([0066] “but a location of this 3×3 pixel analysis will shift slightly based on a jitter location used for centering that pixel analysis region”);
Thomas further teaches adjusting the position of the warped image frame based on the second jitter offset value ([0447] “The jitter offset 4107 is the camera offset that is applied to jitter the scene, with different jitter values being used for successive frames. The jitter offset 4107, in one embodiment, is a sub-pixel offset”).
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Kaskela and Thomas to include adjusting the position of the warped image frame based on the second jitter offset value as taught by Thomas. Doing so would allow processing at lower resolution and assist with processing costs ([0485] “the upscaled pipeline 4720 allows for processing at a lower resolution, thus assisting with processing costs”)
Regarding claim 5, Kaskela as modified by Thomas teaches the image processing method of claim 4, and Thomas further teaches wherein the adjusting of the position of the warped image frame based on the second jitter offset value comprises: adjusting the position of the warped image frame so that the warped image frame is matched to a same area as the current image frame whose position is adjusted based on the second jitter offset value ([0447] “The previously generated output includes at least the immediate previous frame (frame N−1), which is warped using the velocity data 4104 to align the frame with the current frame 4106 for temporal accumulation.”).
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Kaskela and Thomas to include wherein the adjusting of the position of the warped image frame based on the second jitter offset value comprises: adjusting the position of the warped image frame so that the warped image frame is matched to a same area as the current image frame whose position is adjusted based on the second jitter offset value as taught by Thomas. Doing so would allow processing at lower resolution and assist with processing costs ([0485] “the upscaled pipeline 4720 allows for processing at a lower resolution, thus assisting with processing costs”)
Regarding claim 7, Kaskela as modified by Thomas teaches the image processing method of claim 1, and Kaskela further teaches wherein the generating of the output image frame comprises: matching the position-adjusted warped image frame and the current image frame in dimension (Fig 3A, [0055] “Upsampling module 308, which can upsample this output to a higher resolution, such as may correspond to a resolution of inputs 302 or a target resolution.”);
PNG
media_image1.png
431
696
media_image1.png
Greyscale
generating a concatenated image by concatenating the current image frame and the position-adjusted warped image frame matched in dimension (338 of Fig 3B, [0055] “these higher resolution results can then be passed to a refinement network 310, which also received inputs 302 that can be at this same higher resolution.”); and
PNG
media_image2.png
411
764
media_image2.png
Greyscale
outputting the output image frame by inputting the concatenated image into the neural network model ([0056] “In at least one embodiment, this refinement network 310 can be a shallow convolutional neural network (CNN) operating at full resolution, or at least a higher resolution than U-Net 306.”).
Regarding claim 8, Kaskela as modified by Thomas teaches the information processing method of claim 7, and Thomas further teaches wherein the matching in dimension comprises rearranging the position-adjusted warped image frame to correspond to a depth or a channel of the neural network model by a space-to-depth operation ([0512] “the input block 5120 includes an upsample layer 5122, warping 5124, shuffling from space to depth 5126, and convolution and activation 5128.”).
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Kaskela and Thomas to include wherein the matching in dimension comprises rearranging the position-adjusted warped image frame to correspond to a depth or a channel of the neural network model by a space-to-depth operation as taught by Thomas. Doing so would provide faster inference performance ([0513] “For faster inference performance, pixels in the spatial dimensions are shuffled to channel dimension (shuffle from space to depth 5126)”)
Regarding claim 9, Kaskela as modified by Thomas teaches the image processing method of claim 1, and Kaskela further teaches wherein the generating of the current image frame comprises generating the current image frame by performing jittered sampling on subpixels ([0066] “jittering may be performed between frames or images in a sequence, wherein a center point of a color determination is shifted slightly to another point in this pixel. In at least one embodiment, this can correspond to sample point offset from a pixel center by a sub-pixel offset.”) included in the first-resolution pixel area ([0065] “an upsampling process can be performed for each individual pixel of a lower resolution rendered image … an upscaling process might result in color information from that pixel being applied to a corresponding pixel region in an upscaled image that is larger in size.”).
Regarding claim 11, Kaskela as modified by Thomas teaches the image processing method of claim 1, and Kaskela further teaches wherein the neural network model is configured to output the current output image frame ([0050] “Warper 210 can utilize this motion vector and depth data to warp pixel data or color data for specific features of a prior image to corresponding pixel locations in a current image frame, effectively using these motion vectors to map corresponding pixel locations of features in these two images”, where prior image corresponds to feedback image).
Thomas further teaches wherein the neural network model is configured to output a feature map corresponding to the current output image frame ([0208] “The computations for a CNN include applying the convolution mathematical operation to each filter to produce the output of that filter … The output may be referred to as the feature map. For example, the input to a convolution layer can be a multidimensional array of data that defines the various color components of an input image.”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Kaskela and Thomas to include wherein the neural network model is configured to output a feature map corresponding to the current output image frame as taught by Thomas. Doing so would enable high performance generation of high quality images ([0440] “A mixed low precision convolutional neural network is used that applied different computational precisions at different stages to enable the high performance generation of high quality images”)
Regarding claim 12, Kaskela as modified by Thomas teaches the image processing method of claim 1, and Kaskela further teaches wherein the neural network model is configured to receive a first jitter offset and apply a predetermined value corresponding to the first jitter offset to the current output image frame ([0048] “this jitter offset data can be provided, along with a current input video frame and a prior inferred frame, as input to an upscaler 108 including at least one neural network in order to infer a higher quality upsampled image 110 than would be produced by an upsampling algorithm alone”)
Regarding claim 14, Kaskela as modified by Thomas teaches the image processing method of claim 1, and Kaskela further teaches wherein the neural network model is configured to receive a first jitter offset, and when the neural network model uses a structure of a kernel prediction network ([0045] “such an algorithm can also utilize a filtering kernel to produce a new, higher resolution output image from a set of inputs”, [0057] “a number of convolutional layers, as well as a corresponding kernel size and numbers of features, can vary within different refinement networks”), output the current output image frame by applying a predetermined value corresponding to the first jitter offset to a filter of the kernel prediction network ([0058] “a number of channels (e.g., three) for filter parameters such as anisotropic Gaussian filter parameters, at least one channel for adaptive jitter-aware blending of data for at least one historical frame and a current frame”).
Regarding claim 15, Kaskela as modified by Thomas teaches the image processing method of claim 14, and Kaskela further teaches wherein the predetermined value corresponding to the first jitter offset comprises at least one of the first jitter offset ([0048] “this upsampling essentially shifts jitter offsets 122 and per-frame samples so that they are aligned with a history buffer that may be at a higher resolution.”), a formula calculated using a value of the first jitter offset, and a value obtained through separate learning using the first jitter offset value as input ([0362] “wherein the one or more neural networks are further to adjust one or more additional parameters relating to at least one of historical pixel data, filter parameter data, pixel jitter data, filtered pixel data, or warped pixel data.”).
Regarding claim 16, the non-transitory computer-readable storage medium (Kaskela [0454] “one or more non-transitory computer-readable storage media having stored thereon executable instructions”) claim 16 is similar in scope to the method claim 1, and is rejected under similar rationale.
Regarding claim 17, the device claim 17 (Kaskela [0050] “one or more processes running on one or more processors on one or more computing devices”) is similar in scope to the method claim 1, and is rejected under similar rationale.
Regarding claim 18, the device claim 18 is similar in scope to the method claim 3, and is rejected under similar rationale.
Regarding claim 19, the device claim 19 is similar in scope to the method claim 4, and is rejected under similar rationale.
Regarding claim 20, the device claim 20 is similar in scope to the method claim 7, and is rejected under similar rationale.
Claims 6 is rejected under 35 U.S.C. 103 as being unpatentable over Kaskela and Thomas as applied to claim 5 above, and further in view of Kaplanyan et al (US 20230142467 A1, hereinafter Kaplanyan).
Regarding claim 6, Kaskela as modified by Thomas teaches the image processing method of claim 5, and Thomas further teaches wherein the adjusting of the position of the warped image frame comprises at least one of: placing the warped image frame in the same area as the current image frame whose position is adjusted based on the second jitter offset value, by zero-padding and cropping the warped image frame ([0420] “During operation, scalable sparse matrix multiply accelerator 3400 is configurable to accept groups of only one element. Given Src2 input {B0, 0, B2, B3, 0, 0, 0, 0}, two groups ([B0,B2], [B3,0]) are made for the non-zero elements on Src2 for the third embodiment (e.g., scalable sparse matrix multiply accelerator 3300), with the second group including a zero padding”); but fails to explicitly teach placing the warped image frame in the same area as the current image frame whose position is adjusted based on the second jitter offset value, by flipping or reflecting the warped image frame.
In related field of endeavor, Kaplanyan teaches placing the warped image frame in the same area as the current image frame whose position is adjusted based on the second jitter offset value, by flipping or reflecting the warped image frame. ([0465] “the warped previous frame can be aligned with the currently rendered frame and processed via a feature extraction network of a machine learning model 4701 as in FIG. 47. The output block 4120 of the machine learning model 4701 can then output a temporally anti-aliased and upscaled frame. The output frame will be of a higher quality relative to a frame that is generated without motion vector augmentation for portions of the frame for which motion vectors would otherwise not be available, such as shadows, objects reflecting in mirrors, waves in water or other liquids, glossy surfaces, or objects visible through transparent and/or refractive glass.”)
It would have been obvious to one of ordinary skill in the art to have further modified Kaskela and Thomas to include placing the warped image frame in the same area as the current image frame whose position is adjusted based on the second jitter offset value, by flipping or reflecting the warped image frame as taught by Kaplanyan. Doing so would provide a higher quality output frame ([0465] “The output frame will be of a higher quality relative to a frame that is generated without motion vector augmentation”)
Claims 10 is rejected under 35 U.S.C. 103 as being unpatentable over Kaskela and Thomas as applied to claim 9 above, and further in view of Aoki et al (US 20230162468 A1, hereinafter Aoki).
Regarding claim 10, Kaskela as modified by Thomas teaches the image processing method of claim 9, and Kaskela further teaches wherein the generating of the current image frame comprises performing jittered sampling by selectively sampling respective sampling points corresponding to the subpixels ([0065] “this can include jitter-aware upsampling and accumulating of samples at an upsampled resolution … an upscaling process might result in color information from that pixel being applied to a corresponding pixel region in an upscaled image that is larger in size. In at least one embodiment, this pixel in this upscaled image can be segmented (or mapped) into a number of individual pixels”). Kaskela and Thomas fail to explicitly teach the sampling points are sampled alternately based on a predetermined period, but in related field of endeavor, Aoki teaches the sampling points are sampled alternately based on a predetermined period ([0326] “At this time, the preprocessing section 210 can set all the pixel positions in the region of interest as the pixel positions of the sampling pixels without performing thinning. Alternatively, the preprocessing section 210 may set, for the region of interest, the pixel positions of the sampling pixels at a sampling interval smaller than a sampling interval of the sampling pixels set in the original images 320Φ1 to 320Φ4”)
It would have been obvious to one of ordinary skill in the art to have further modified Kaskela and Thomas to include the sampling points are sampled alternately based on a predetermined period as taught by Aoki. Doing so would reduce the load and power consumption ([0327] “the load of the recognition section 220 is reduced, and power consumption can be suppressed”)
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Kaskela and Thomas as applied to claim 1 above, and further in view of Janis et al (US 20230206394 A1, hereinafter Janis).
Regarding claim 13, Kaskela and Thomas teach the image processing method of claim 1 Kaskela further teaches wherein the neural network model is configured to receive a first jitter offset, and output the current output image frame ([0048] “this jitter offset data can be provided, along with a current input video frame and a prior inferred frame, as input to an upscaler 108 including at least one neural network in order to infer a higher quality upsampled image 110 than would be produced by an upsampling algorithm alone”)
Kaskela and Thomas fail to explicitly teach applying a predetermined value corresponding to the first jitter offset to kernel weight or bias values of layers of the neural network model, but in related field of endeavor, Janis teaches ([0053] “j: Current input jitter vector, denoting sampling position within an input pixel ”, [0063] “an upsampling process can perform blending by upsampling an input color with one or more predicted anisotropic kernels, such as discussed previously. In at least one embodiment, this upsampling process can account for total kernel weight w.sub.UP, such as may be given by: w.sub.UP=Σ.sub.uK(u+j,p)”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Kaskela and Thomas to include applying a predetermined value corresponding to the first jitter offset to kernel weight or bias values of layers of the neural network model as taught by Janis. Doing so would help reduce a presence of artifacts during playback ([0045] “such warping can ensure that points, such as feature points, for various images are tracked over time and corresponding color values used for blending, which can help reduce a presence of artifacts such as noise or flickering during playback.”)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. O'Neil et al (US 20240037713 A1, hereinafter O'Neil) teaches an anti-aliasing method using jitter offsets and warped feedbacks with some similar features to the claimed invention (Fig. 3A).
PNG
media_image3.png
492
737
media_image3.png
Greyscale
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN PATRICK GOCO whose telephone number is (571)272-5872. The examiner can normally be reached M-Th, 7:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
/J.P.G./ Examiner, Art Unit 2611
Prosecution Insights