DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This communication is in response to Application No. 18/989,805 filed 12/20/2024. Claims 1-20 are pending.
Priority
Receipt is acknowledged of certified copies of papers submitted under 35. U.S.C 119(a)-(d), which papers have been placed of record in the file.
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 12/20/2024, 05/09/2025, 09/05/2025 and 03/05/2026 have been entered and considered. Initialed copies of the PTO-1449 by the examiner are attached.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Claim(s) 18 and 20 recite limitations that use words like “means” (or “step”) or similar terms with functional language and do invoke 35 U.S.C. 112(f):
Claims 18 and 20; recites the limitation, “processing module configured to …,”.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
After a careful analysis, as disclosed above, and a careful review of the specification the following limitations in claim(s) 18 and 20:
“processing module” (Fig. 3, #302 and Fig. 22, #2210. Paragraph [0233], [0345]- “The processing module 302 comprises temporal resampling logic 306 configured to perform the temporal resampling described herein. The temporal resampling logic 306 may be implemented in software, hardware or a combination thereof” and “FIG. 22 shows a computer system in which the processing modules or processing systems described herein may be implemented. The computer system comprises a CPU 2202, a GPU 2204, a memory 2206, a neural network accelerator (NNA) 2208 and other devices 2214, such as a display 2216, speakers 2218 and a camera 2222. A processing block 2210 (corresponding to the processing modules described herein) is implemented on the GPU 2204. In other examples, one or more of the depicted components may be omitted from the system, and/or the processing block 2210 may be implemented on the CPU 2202” thus, have sufficient structure or material wherein is any kind of GPU or CPU.).
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-3, 9, 10, and 18-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Panteleev et al. (US 20230138718 A1).
Regarding claim 1, Panteleev teaches a method of determining one or more pixel values at a respective one or more upsampled pixel locations for a current frame of a sequence of frames, the method comprising (“This blurred gradient image can then be upscaled 512, such as to a target output resolution that will include one gradient value per pixel location, such as may be determined using interpolation in the upscaling process” Panteleev, [0069], wherein the upscaling corresponds to that of upsampling; wherein the image is part of a sequence of images, “embodiments allow for rendering (or other generation) of images... In order to provide for smoothing between sequential frames of video” Panteleev, [0041]):
obtaining depth values for locations of pixels of a reference frame of the sequence of frames; and for each of the one or more upsampled pixel locations: obtaining a depth value of the current frame for the upsampled pixel location (“A light gradient can be generated 508 for each of the tiles (or other pixel regions) of the current image with respect to the corresponding tile of the prior image” Panteleev, [0069]; “Forward or backward projection can be applied to these samples to determine their screen space locations in the current frame, as illustrated in grid 204. The depth buffer of the current frame can be used to discard reprojected surface samples which are occluded in the current frame” Panteleev, [0058]),
obtaining a motion vector for the upsampled pixel location to indicate motion between the reference frame and the current frame for the upsampled pixel location (““temporal gradients” as differences in shading of the same surfaces on two consecutive frames using the same random numbers. These differences are zero if the lighting environment of the surface stays the same, and are nonzero if something has changed, such as when a light has moved relative to the surface or has come in (or out) of shadow” Panteleev, [0003]; wherein temporal gradients corresponds to motion vectors, see Panteleev [0054], Fig. 2 #204, “FIG. 2 illustrates an example approach to gradient sampling that can be used in accordance with at least one embodiment. Such an approach can attempt to determine an adaptive per-pixel weight computed from temporal gradients”),
using the motion vector for the upsampled pixel location to identify one or more of the pixels of the reference frame (“Forward or backward projection can be applied to these samples to determine their screen space locations in the current frame, as illustrated in grid 204” Panteleev, [0058], wherein “FIG. 2 illustrates an example approach to gradient sampling that can be used in accordance with at least one embodiment. Such an approach can attempt to determine an adaptive per-pixel weight computed from temporal gradients” Panteleev, [0054]),
determining a weight for each of the one or more identified pixels of the reference frame in dependence on: (i) the depth value of the current frame for the upsampled pixel location, and (ii) the depth value for the location of the identified pixel of the reference frame (“FIG. 2 illustrates an example approach to gradient sampling that can be used in accordance with at least one embodiment. Such an approach can attempt to determine an adaptive per-pixel weight computed from temporal gradients”, Panteleev, [0054], wherein the depth values of current frame and reference frame are included, as disclosed at [0069] and [0058], respectively, “A light gradient can be generated 508 for each of the tiles (or other pixel regions) of the current image with respect to the corresponding tile of the prior image”; “Forward or backward projection can be applied to these samples to determine their screen space locations in the current frame, as illustrated in grid 204. The depth buffer of the current frame can be used to discard reprojected surface samples which are occluded in the current frame”), and
determining the pixel value for the upsampled pixel location using the determined weight for each of the one or more identified pixels (“This blurred gradient image can then be upscaled 512, such as to a target output resolution that will include one gradient value per pixel location, such as may be determined using interpolation in the upscaling process. The gradients for these individual pixel locations can be transformed 514, during or after this upscaling, to pixel-specific confidence values” Panteleev, [0069], wherein upscaling corresponds to upsampling; and confidence values also correspond as weights when blending lighting information (e.g., pixel values) from the current frame and reference frame, see [0069] “For example, the weight may be a blending weight indicating how much the lighting data from the prior frame should be used versus the lighting data for a current frame for a given pixel location”).
Regarding claim 2, Panteleev teaches the method of claim 1, further comprising obtaining pixel values of the one or more identified pixels of the reference frame of the sequence of frames, wherein said determining the pixel value for the upsampled pixel location comprises performing a weighted sum of the pixel values of the one or more identified pixels of the reference frame using the determined weight for each of the one or more identified pixels in the weighted sum (“This blurred gradient image can then be upscaled 512, such as to a target output resolution that will include one gradient value per pixel location, such as may be determined using interpolation in the upscaling process. The gradients for these individual pixel locations can be transformed 514, during or after this upscaling, to pixel-specific confidence values” Panteleev, [0069], wherein upscaling corresponds to upsampling; and confidence values also correspond as weights when blending lighting information (e.g., pixel values) from the current frame and reference frame, see [0069] “For example, the weight may be a blending weight indicating how much the lighting data from the prior frame should be used versus the lighting data for a current frame for a given pixel location”; i.e., the blending weight corresponds to the weighted sum of pixel values).
Regarding claim 3, Panteleev teaches the method of claim 1, wherein for each of the one or more upsampled pixel locations, the weight for each of the one or more identified pixels of the reference frame is determined in dependence on a difference between the depth value of the current frame for the upsampled pixel location and the depth value for the location of the identified pixel of the reference frame (“Forward or backward projection can be applied to these samples to determine their screen space locations in the current frame, as illustrated in grid 204. The depth buffer of the current frame can be used to discard reprojected surface samples which are occluded in the current frame” such that “gradient reconstruction can be joint-bilateral, where the luminance is filtered and simultaneously used to derive filter weights used for reconstruction of the gradient and luminance samples” Panteleev, [0058]).
Regarding claim 9, Panteleev teaches the method claim 1, wherein said using the motion vector for the upsampled pixel location to identify one or more of the pixels of the reference frame comprises projecting the upsampled pixel location to a location in the reference frame based on the motion vector and identifying one or more of the pixels of the reference frame in the vicinity of the projected location in the reference frame (“Forward or backward projection can be used in different embodiments. Forward projection can carry a surface sample G.sub.i-1,j from the previous frame i−1 to the current frame i... Having access to the new world space location through {right arrow over (G)}.sub.i-1,j the coordinate transforms for frame i can yield the corresponding screen space location in the current frame. In particular, the index of pixel {right arrow over (j)} can be computed that covers the surface sample in the current frame, as illustrated in grid 204” Panteleev, [0056]).
Regarding claim 10, Panteleev teaches the method of claim 1, wherein, for each of the one or more upsampled pixel locations, said determining a weight for each of the one or more identified pixels of the reference frame comprises determining an initial weight and using the initial weight to determine the weight for the identified pixel of the reference frame (“Forward or backward projection can be applied to these samples to determine their screen space locations in the current frame, as illustrated in grid 204. The depth buffer of the current frame can be used to discard reprojected surface samples which are occluded in the current frame” wherein “Such gradient reconstruction can be joint-bilateral, where the luminance is filtered and simultaneously used to derive filter weights used for reconstruction of the gradient and luminance samples” Panteleev, [0058]-[0059]).
Regarding claim 18, Panteleev teaches a processing module (Panteleev, [0070]; Figs. 4 and 6) configured to determine one or more pixel values at a respective one or more upsampled pixel locations for a current frame of a sequence of frames, the processing module being configured to (“This blurred gradient image can then be upscaled 512, such as to a target output resolution that will include one gradient value per pixel location, such as may be determined using interpolation in the upscaling process” Panteleev, [0069], wherein the upscaling corresponds to that of upsampling; wherein the image is part of a sequence of images, “embodiments allow for rendering (or other generation) of images... In order to provide for smoothing between sequential frames of video” Panteleev, [0041]):
obtain depth values for locations of pixels of a reference frame of the sequence of frames; and for each of the one or more upsampled pixel locations: obtain a depth value of the current frame for the upsampled pixel location (“A light gradient can be generated 508 for each of the tiles (or other pixel regions) of the current image with respect to the corresponding tile of the prior image” Panteleev, [0069]; “Forward or backward projection can be applied to these samples to determine their screen space locations in the current frame, as illustrated in grid 204. The depth buffer of the current frame can be used to discard reprojected surface samples which are occluded in the current frame” Panteleev, [0058]),
obtain a motion vector for the upsampled pixel location to indicate motion between the reference frame and the current frame for the upsampled pixel location (““temporal gradients” as differences in shading of the same surfaces on two consecutive frames using the same random numbers. These differences are zero if the lighting environment of the surface stays the same, and are nonzero if something has changed, such as when a light has moved relative to the surface or has come in (or out) of shadow” Panteleev, [0003]; wherein temporal gradients corresponds to motion vectors, see Panteleev [0054], Fig. 2 #204, “FIG. 2 illustrates an example approach to gradient sampling that can be used in accordance with at least one embodiment. Such an approach can attempt to determine an adaptive per-pixel weight computed from temporal gradients”),
use the motion vector for the upsampled pixel location to identify one or more of the pixels of the reference frame (“Forward or backward projection can be applied to these samples to determine their screen space locations in the current frame, as illustrated in grid 204” Panteleev, [0058], wherein “FIG. 2 illustrates an example approach to gradient sampling that can be used in accordance with at least one embodiment. Such an approach can attempt to determine an adaptive per-pixel weight computed from temporal gradients” Panteleev, [0054]),
determine a weight for each of the one or more identified pixels of the reference frame in dependence on: (i) the depth value of the current frame for the upsampled pixel location, and (ii) the depth value for the location of the identified pixel of the reference frame (“FIG. 2 illustrates an example approach to gradient sampling that can be used in accordance with at least one embodiment. Such an approach can attempt to determine an adaptive per-pixel weight computed from temporal gradients”, Panteleev, [0054], wherein the depth values of current frame and reference frame are included, as disclosed at [0069] and [0058], respectively, “A light gradient can be generated 508 for each of the tiles (or other pixel regions) of the current image with respect to the corresponding tile of the prior image”; “Forward or backward projection can be applied to these samples to determine their screen space locations in the current frame, as illustrated in grid 204. The depth buffer of the current frame can be used to discard reprojected surface samples which are occluded in the current frame”), and
determine the pixel value for the upsampled pixel location using the determined weight for each of the one or more identified pixels (“This blurred gradient image can then be upscaled 512, such as to a target output resolution that will include one gradient value per pixel location, such as may be determined using interpolation in the upscaling process. The gradients for these individual pixel locations can be transformed 514, during or after this upscaling, to pixel-specific confidence values” Panteleev, [0069], wherein upscaling corresponds to upsampling; and confidence values also correspond as weights when blending lighting information (e.g., pixel values) from the current frame and reference frame, see [0069] “For example, the weight may be a blending weight indicating how much the lighting data from the prior frame should be used versus the lighting data for a current frame for a given pixel location”).
Regarding claim 19, Panteleev teaches a non-transitory computer readable storage medium having stored thereon computer readable code (“a non-transitory computer-readable storage” Panteleev, [0353]) configured to cause the method as set forth in claim 1 to be performed when the code is run on at least one processor (“a main central processing unit (“CPU”)” Panteleev, [0353]).
Regarding claim 20, Panteleev teaches a non-transitory computer readable storage medium having stored thereon an integrated circuit definition dataset that (“a non-transitory computer-readable storage” Panteleev, [0353]), when processed in an integrated circuit manufacturing system (“a main central processing unit (“CPU”)... computer system” Panteleev, [0353]), configures the integrated circuit manufacturing system to manufacture a processing module that is configured to determine one or more pixel values at a respective one or more upsampled pixel locations for a current frame of a sequence of frames (“This blurred gradient image can then be upscaled 512, such as to a target output resolution that will include one gradient value per pixel location, such as may be determined using interpolation in the upscaling process” Panteleev, [0069], wherein the upscaling corresponds to that of upsampling; wherein the image is part of a sequence of images, “embodiments allow for rendering (or other generation) of images... In order to provide for smoothing between sequential frames of video” Panteleev, [0041]), the processing module being configured to:
obtain depth values for locations of pixels of a reference frame of the sequence of frames; and for each of the one or more upsampled pixel locations: obtain a depth value of the current frame for the upsampled pixel location (“A light gradient can be generated 508 for each of the tiles (or other pixel regions) of the current image with respect to the corresponding tile of the prior image” Panteleev, [0069]; “Forward or backward projection can be applied to these samples to determine their screen space locations in the current frame, as illustrated in grid 204. The depth buffer of the current frame can be used to discard reprojected surface samples which are occluded in the current frame” Panteleev, [0058]),
obtain a motion vector for the upsampled pixel location to indicate motion between the reference frame and the current frame for the upsampled pixel location (““temporal gradients” as differences in shading of the same surfaces on two consecutive frames using the same random numbers. These differences are zero if the lighting environment of the surface stays the same, and are nonzero if something has changed, such as when a light has moved relative to the surface or has come in (or out) of shadow” Panteleev, [0003]; wherein temporal gradients corresponds to motion vectors, see Panteleev [0054], Fig. 2 #204, “FIG. 2 illustrates an example approach to gradient sampling that can be used in accordance with at least one embodiment. Such an approach can attempt to determine an adaptive per-pixel weight computed from temporal gradients”),
use the motion vector for the upsampled pixel location to identify one or more of the pixels of the reference frame (“Forward or backward projection can be applied to these samples to determine their screen space locations in the current frame, as illustrated in grid 204” Panteleev, [0058], wherein “FIG. 2 illustrates an example approach to gradient sampling that can be used in accordance with at least one embodiment. Such an approach can attempt to determine an adaptive per-pixel weight computed from temporal gradients” Panteleev, [0054]),
determine a weight for each of the one or more identified pixels of the reference frame in dependence on: (i) the depth value of the current frame for the upsampled pixel location, and (ii) the depth value for the location of the identified pixel of the reference frame (“FIG. 2 illustrates an example approach to gradient sampling that can be used in accordance with at least one embodiment. Such an approach can attempt to determine an adaptive per-pixel weight computed from temporal gradients”, Panteleev, [0054], wherein the depth values of current frame and reference frame are included, as disclosed at [0069] and [0058], respectively, “A light gradient can be generated 508 for each of the tiles (or other pixel regions) of the current image with respect to the corresponding tile of the prior image”; “Forward or backward projection can be applied to these samples to determine their screen space locations in the current frame, as illustrated in grid 204. The depth buffer of the current frame can be used to discard reprojected surface samples which are occluded in the current frame”), and
determine the pixel value for the upsampled pixel location using the determined weight for each of the one or more identified pixels (“This blurred gradient image can then be upscaled 512, such as to a target output resolution that will include one gradient value per pixel location, such as may be determined using interpolation in the upscaling process. The gradients for these individual pixel locations can be transformed 514, during or after this upscaling, to pixel-specific confidence values” Panteleev, [0069], wherein upscaling corresponds to upsampling; and confidence values also correspond as weights when blending lighting information (e.g., pixel values) from the current frame and reference frame, see [0069] “For example, the weight may be a blending weight indicating how much the lighting data from the prior frame should be used versus the lighting data for a current frame for a given pixel location”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 11-13 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Panteleev et al. in view of Tovey et al. (US 20230298133 A1).
Regarding claim 11, Panteleev teaches the method of claim 10, wherein said using the motion vector for the upsampled pixel location to identify one or more of the pixels of the reference frame comprises projecting the upsampled pixel location to a location in the reference frame based on the motion vector and identifying one or more of the pixels of the reference frame in the vicinity of the projected location in the reference frame (“Forward or backward projection can be used in different embodiments. Forward projection can carry a surface sample G.sub.i-1,j from the previous frame i−1 to the current frame i... Having access to the new world space location through {right arrow over (G)}.sub.i-1,j the coordinate transforms for frame i can yield the corresponding screen space location in the current frame. In particular, the index of pixel {right arrow over (j)} can be computed that covers the surface sample in the current frame, as illustrated in grid 204” Panteleev, [0056]).
Panteleev fail to explicitly teach wherein the initial weight for each of the one or more identified pixels of the reference frame is determined by: determining a distance between the projected location and the location of the identified pixel in the reference frame; and mapping the distance to an initial weight using a predetermined relationship.
However, Tovey explicitly teaches wherein the initial weight for each of the one or more identified pixels of the reference frame is determined by: determining a distance between the projected location and the location of the identified pixel in the reference frame (“FIG. 13 shows another example in which the rectifier component 910 reduces the influence of historical samples (S.sub.h) 1302 based on their distance (d) from the color bounding box 1102 of the upsampled pixels. In this example, the farther the distance (d), the more the influence of the historical sample 1302 is reduced by the color rectifier component 910” Tovey ending in 133, [0098]); and
mapping the distance to an initial weight using a predetermined relationship (“he reprojection component 906, in one or more implementations, uses bilinear interpolation at the translated pixel positions to reproject the locks. As such, the output of the reprojection sub-stage 905 is a presentation resolution image that includes all the data from the previous frame that is mappable into the current frame 502” Tovey ending in 133, [0091]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Panteleev of having a method of determining one or more pixel values at a respective one or more upsampled pixel locations for a current frame of a sequence of frames, with the teachings of Tovey of having wherein the initial weight for each of the one or more identified pixels of the reference frame is determined by: determining a distance between the projected location and the location of the identified pixel in the reference frame; and mapping the distance to an initial weight using a predetermined relationship.
Wherein having Panteleev’s image generation system includes wherein the initial weight for each of the one or more identified pixels of the reference frame is determined by: determining a distance between the projected location and the location of the identified pixel in the reference frame; and mapping the distance to an initial weight using a predetermined relationship.
The motivation behind the modification would have been to obtain an image rendering system that takes advantage of adaptive spatiotemporal variance of consecutive frames to provide smoothing between sequential frames and avoid introducing lag of lighting effects, since both Panteleev and Tovey are frame generation upscaling systems. Wherein Panteleev image generation system reduces computational workload by rendering gradients for a portion, fraction, or subset of pixel locations of a high resolution image, while Tovey image generation system improves upscaling by using temporal feedback to reconstruct high-resolution images while maintaining and improving image quality compared to native rendering. Please see Panteleev et al. (US 20230138718 A1), Paragraph [0049 and 0068] and Tovey et al. (US 20230298133 A1), Paragraph [0023].
Regarding claim 12, Panteleev teaches the method of claim 1, Panteleev fails to explicitly teach for each of the one or more upsampled pixel locations: obtaining a plurality of input pixel values of the current frame for locations within a region surrounding the upsampled pixel location; and determining a mean of the input pixel values of the current frame within the region surrounding the upsampled pixel location.
However, Tovey teaches for each of the one or more upsampled pixel locations: obtaining a plurality of input pixel values of the current frame for locations within a region surrounding the upsampled pixel location; and determining a mean of the input pixel values of the current frame within the region surrounding the upsampled pixel location (“In one or more implementations, the shading change detection component 902 compares the luminance of the pixel to the average luminance of a neighborhood (e.g., 32×32) of pixels in the current frame 502. If the difference between the luminance of the pixel and the average luminance of the neighborhood of pixels satisfies (e.g., is greater than or, alternatively, equal to) the shading change threshold, then the shading change detection component 902 determines that the shading of the pixel has changed” Tovey ending in 133, [0085]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Panteleev of having a method of determining one or more pixel values at a respective one or more upsampled pixel locations for a current frame of a sequence of frames, with the teachings of Tovey of having for each of the one or more upsampled pixel locations: obtaining a plurality of input pixel values of the current frame for locations within a region surrounding the upsampled pixel location; and determining a mean of the input pixel values of the current frame within the region surrounding the upsampled pixel location.
Wherein having Panteleev’s image generation system includes for each of the one or more upsampled pixel locations: obtaining a plurality of input pixel values of the current frame for locations within a region surrounding the upsampled pixel location; and determining a mean of the input pixel values of the current frame within the region surrounding the upsampled pixel location.
The motivation behind the modification would have been to obtain an image rendering system that takes advantage of adaptive spatiotemporal variance of consecutive frames to provide smoothing between sequential frames and avoid introducing lag of lighting effects, since both Panteleev and Tovey are frame generation upscaling systems. Wherein Panteleev image generation system reduces computational workload by rendering gradients for a portion, fraction, or subset of pixel locations of a high resolution image, while Tovey image generation system improves upscaling by using temporal feedback to reconstruct high-resolution images while maintaining and improving image quality compared to native rendering. Please see Panteleev et al. (US 20230138718 A1), Paragraph [0049 and 0068] and Tovey et al. (US 20230298133 A1), Paragraph [0023].
Regarding claim 13, Panteleev in view of Tovey teaches the method of claim 12, Panteleev fails to explicitly teach wherein said determining the pixel value for the upsampled pixel location comprises clamping the determined pixel value so that it does not differ from the determined mean of the input pixel values of the current frame within the region surrounding the upsampled pixel location by more than a threshold value.
However, Tovey teaches wherein said determining the pixel value for the upsampled pixel location comprises clamping the determined pixel value so that it does not differ from the determined mean of the input pixel values of the current frame within the region surrounding the upsampled pixel location by more than a threshold value (“In one or more implementations, the shading change detection component 902 compares the luminance of the pixel to the average luminance of a neighborhood (e.g., 32×32) of pixels in the current frame 502. If the difference between the luminance of the pixel and the average luminance of the neighborhood of pixels satisfies (e.g., is greater than or, alternatively, equal to) the shading change threshold, then the shading change detection component 902 determines that the shading of the pixel has changed” Tovey ending in 133, [0085]; i.e., the clamping here is interpreted as a form of “limiting” the determined pixel value, e.g., luminance of the pixel).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Panteleev of having a method of determining one or more pixel values at a respective one or more upsampled pixel locations for a current frame of a sequence of frames, with the teachings of Tovey of having wherein said determining the pixel value for the upsampled pixel location comprises clamping the determined pixel value so that it does not differ from the determined mean of the input pixel values of the current frame within the region surrounding the upsampled pixel location by more than a threshold value.
Wherein having Panteleev’s image generation system includes wherein said determining the pixel value for the upsampled pixel location comprises clamping the determined pixel value so that it does not differ from the determined mean of the input pixel values of the current frame within the region surrounding the upsampled pixel location by more than a threshold value.
The motivation behind the modification would have been to obtain an image rendering system that takes advantage of adaptive spatiotemporal variance of consecutive frames to provide smoothing between sequential frames and avoid introducing lag of lighting effects, since both Panteleev and Tovey are frame generation upscaling systems. Wherein Panteleev image generation system reduces computational workload by rendering gradients for a portion, fraction, or subset of pixel locations of a high resolution image, while Tovey image generation system improves upscaling by using temporal feedback to reconstruct high-resolution images while maintaining and improving image quality compared to native rendering. Please see Panteleev et al. (US 20230138718 A1), Paragraph [0049 and 0068] and Tovey et al. (US 20230298133 A1), Paragraph [0023].
Regarding claim 17, Panteleev teaches the method of claim 1, Panteleev further teaches determining an interpolated depth value for the upsampled pixel location (“This blurred gradient image can then be upscaled 512, such as to a target output resolution that will include one gradient value per pixel location, such as may be determined using interpolation in the upscaling process” Panteleev, [0069]) and performing a weighted sum of the determined interpolated depth values using the determined depth weights for the pairs of input pixels (“This blurred gradient image can then be upscaled 512, such as to a target output resolution that will include one gradient value per pixel location, such as may be determined using interpolation in the upscaling process”, Panteleev, [0069]; wherein confidences values include depth values (see Panteleev, [0058] “depth buffer of the current frame”) correspond to a weighted sum for pixels).
Panteleev fails to explicitly teach wherein said obtaining a depth value of the current frame for the upsampled pixel location comprises: receiving depth values of the current frame at locations of input pixels surrounding the upsampled pixel location; for each pair of input pixels of said input pixels for which depth values are received, based on the depth values for the pair of input pixels; determining depth weights for said pairs of input pixels based on depth gradients between the depth values for the pairs of input pixels and determining the depth value of the current frame for the upsampled pixel location.
However, Tovey teaches wherein said obtaining a depth value of the current frame for the upsampled pixel location comprises: receiving depth values of the current frame at locations of input pixels surrounding the upsampled pixel location; for each pair of input pixels of said input pixels for which depth values are received (“the upscaler 332 is temporal and takes data associated with a currently rendered (aliased) frame 502 (also referred to herein as “current frame 502” or “input frame 502”) and data associated with a previously presented (upscaled) frame as input... the input data associated with the previously presented frame includes an output buffer 506-1 that maintains data, such as a color buffer (e.g., output color), a depth buffer, and the like, for the previously presented frame” Tovey ending in 133, [0057]; wherein the upscaler corresponds to upsampler for pixels in the frame(s)), based on the depth values for the pair of input pixels (“as the depth values and motion vectors are dilated, they naturally follow the contours of the geometric edges present in the input depth buffer 322-1. In one or more implementations, the reconstruct and dilate component 516 computes the dilated depth values and motion vectors by considering the depth values of a 3×3 (or other kernel size) neighborhood for each pixel of the current frame 502. The reconstruct and dilate component 516 then selects the depth values and corresponding motion vectors in that neighborhood where the depth value is nearest to the camera. This process is illustrated in FIG. 8” Tovey ending in 133, [0072]);
determining depth weights for said pairs of input pixels based on depth gradients between the depth values for the pairs of input pixels (“As it may be difficult to determine from either color, depth, or motion vectors which pixels have been rendered using alpha blending, the application 206, in one or more implementations, provides a reactivity mask 602 as an input to the upscaler 332... the reactivity mask 602 guides the upscaler 332 on where it should reduce its reliance on historical information when compositing the current pixel and, instead, allow the samples of the current frame 502 to contribute more to the final result” Tovey ending in 133, [0060]; [0072]); and
determining the depth value of the current frame for the upsampled pixel location (“the upscaler 332 is temporal and takes data associated with a currently rendered (aliased) frame 502 (also referred to herein as “current frame 502” or “input frame 502”) and data associated with a previously presented (upscaled) frame as input... the input data associated with the previously presented frame includes an output buffer 506-1 that maintains data, such as a color buffer (e.g., output color), a depth buffer, and the like, for the previously presented frame” Tovey ending in 133, [0057]; wherein the upscaler corresponds to upsampler for pixels in the frame(s)).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Panteleev of having a method of determining one or more pixel values at a respective one or more upsampled pixel locations for a current frame of a sequence of frames, with the teachings of Tovey of having wherein said obtaining a depth value of the current frame for the upsampled pixel location comprises: receiving depth values of the current frame at locations of input pixels surrounding the upsampled pixel location; for each pair of input pixels of said input pixels for which depth values are received, based on the depth values for the pair of input pixels; determining depth weights for said pairs of input pixels based on depth gradients between the depth values for the pairs of input pixels and determining the depth value of the current frame for the upsampled pixel location.
Wherein having Panteleev’s image generation system includes wherein said obtaining a depth value of the current frame for the upsampled pixel location comprises: receiving depth values of the current frame at locations of input pixels surrounding the upsampled pixel location; for each pair of input pixels of said input pixels for which depth values are received, based on the depth values for the pair of input pixels; determining depth weights for said pairs of input pixels based on depth gradients between the depth values for the pairs of input pixels and determining the depth value of the current frame for the upsampled pixel location.
The motivation behind the modification would have been to obtain an image rendering system that takes advantage of adaptive spatiotemporal variance of consecutive frames to provide smoothing between sequential frames and avoid introducing lag of lighting effects, since both Panteleev and Tovey are frame generation upscaling systems. Wherein Panteleev image generation system reduces computational workload by rendering gradients for a portion, fraction, or subset of pixel locations of a high resolution image, while Tovey image generation system improves upscaling by using temporal feedback to reconstruct high-resolution images while maintaining and improving image quality compared to native rendering. Please see Panteleev et al. (US 20230138718 A1), Paragraph [0049 and 0068] and Tovey et al. (US 20230298133 A1), Paragraph [0023].
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Panteleev et al. in view of Ward (“Redundant discrete wavelet transform based super-resolution using sub-pixel image registration”, 2003).
Regarding claim 16, Panteleev teaches the method of claim 1, Panteleev fails to explicitly teach wherein the upsampled pixel locations are between the locations of diagonally adjacent input pixels of the current frame, such that the upsampled pixel locations and the locations of the input pixels form a repeating quincunx pattern.
However, Ward teaches wherein the upsampled pixel locations are between the locations of diagonally adjacent input pixels of the current frame, such that the upsampled pixel locations and the locations of the input pixels form a repeating quincunx pattern (“a registration algorithm exists which can take a video sequence and determine the pixel shift between adjacent frames at the sub-pixel level. The algorithm also assumes a specific sub-pixel shift between the frames... Quincunx sampling is best described as only taking pixels in an image with purely even, and purely odd, valued indices, as shown in Figure 3.1” Ward, pg. 3-1 or 33).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Panteleev of having a method of determining one or more pixel values at a respective one or more upsampled pixel locations for a current frame of a sequence of frames, with the teachings of Ward of having wherein the upsampled pixel locations are between the locations of diagonally adjacent input pixels of the current frame, such that the upsampled pixel locations and the locations of the input pixels form a repeating quincunx pattern.
Wherein having Panteleev’s image generation system includes wherein the upsampled pixel locations are between the locations of diagonally adjacent input pixels of the current frame, such that the upsampled pixel locations and the locations of the input pixels form a repeating quincunx pattern.
The motivation behind the modification would have been to obtain an image rendering system that takes advantage of adaptive spatiotemporal variance of consecutive frames to provide smoothing between sequential frames and avoid introducing lag of lighting effects, since both Panteleev and Ward are frame generation upscaling systems. Wherein Panteleev image generation system reduces computational workload by rendering gradients for a portion, fraction, or subset of pixel locations of a high resolution image, while Ward super-resolution algorithm method uses quincunx sampling as a starting point for effective half-pixel shift in the x and y direction relative to each other as it may be extended to other arbitrary sub-pixel shifts. Please see Panteleev et al. (US 20230138718 A1), Paragraph [0049 and 0068] and Ward (“Redundant discrete wavelet transform based super-resolution using sub-pixel image registration”, 2003), pg. 33 and/or section 3-1.
Allowable Subject Matter
Claim(s) 4-8, 14 and 15 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
ELAD et al. (US 20090110285 A1) – re-sampling a sequence of images in order to improve its resolution using upscaling; compares pixel vaues of areas of neighboring images within a predetermined range and a weight value is obtained for each comparison, including average of pixel values in each area.
Janis (US 12749146 B2) – reconstructs one or more images and determines one or more pixel blending weights; uses low resolution image data and motion vectors for a current image in a sequence and the image data can be upsampled to a higher resolution.
Prendergast et al. (“A block-based super-resolution for video sequences”, 2008) – uses reverse motion model mapping the locations of desired high-resolution pixels onto their corresponding locations in the observation frames. The reconstruction algorithm can work with any registration algorithm providing any mapping of pixels from neighboring frames onto the frame of interest.
Inquiries
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EMMANUEL SILVA-AVINA whose telephone number is (571)270-0729. The examiner can normally be reached Monday - Friday 11 AM - 8 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EMMANUEL SILVA-AVINA/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights