Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 10, 14, 16, and 17 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Lin et al. (U.S. Patent Publication No. 2021/0342984).
Regarding claim 1, Lin discloses a computer-implemented method comprising: receiving, from a client device, a digital image having a set of pixels to be replaced with a generated content portion [Lin: 0112 “pixel information in the one or more regions can be removed ( e.g. , for filling using pixel information determined during image inpainting )”](teaches pixels removed and then later filled which corresponds to pixels being replaced with generated content); determining a first set of tiles from the digital image (interpreted as the method forms a sub image portions (tiles) from the original image)[Lin: 0108 “set of most similar high - resolution feature patches from outside the designated portion”](teaches repeatedly operating on feature patches of the high-resolution input image, feature patches here correspond to tiles. Lin further teaches selecting/using sets of high-resolution feature patches from the high-resolution image corresponding to determining a first set of tiles from the digital image); determining a second set of tiles from a first modified digital image that corresponds to the digital image and includes the generated content portion at a first resolution (interpreted as the method forms a set of tiles from a modified version of the image that corresponds to the original image and already includes generated content in the target region, but at a first lower resolution)[Lin: 0045 “Image inpainting can be performed on the down sampled image to generate a downsampled image result ( e.g. , the downsampled image with the designated portion filled using inpainting )”](teaches downsampled image result which is a modified version corresponding to the input image, where the designated portion (hole) is filled using inpainting (generated content) at low resolution (first resolution), further teaches analyzing patches within that low resolution image result, these patches correspond to the claimed second set of tiles); generating, using a super resolution neural network and based on the first set of tiles and the second set of tiles, a second modified digital image that corresponds to the digital image and includes the generated content portion at a second resolution that is higher than the first resolution (interpreted as using a neural network that perform super resolution/upsampling and using information from both tile sets (from the original image and the modified low resolution image), the method generates another modified image where the generated content portion is at a higher resolution than before)[Lin: 0033 “enables generation of a high - resolution inpainting result from a lower - resolution image that has undergone inpainting”][Lin: 0071 “image upsampling is the process of increasing the resolution of an image”][Lin: 0101 “map information from the most similar feature patches identified in the low - resolution image result to corresponding feature patches in the high - resolution image .”](teaches generating a high resolution inpainting result from a lower resolution inpainted result using one or more neural networks (guided upsampling). Lin’s upsampling neural network processing is based on both feature patches from the low resolution image result (the modified image) and corresponding feature patches in the high resolution image (the original image). The output is a high-resolution image result (second resolution) that includes the inpainted/generated region at higher resolution than the low resolution inpainted result); and providing a super-resolved digital image generated from the second modified digital image for display on the client device (interpreted the method outputs the resulting image so it can be displayed) [Lin: 0058 “an upsampled image ( e.g. , without a hole ) can be provided to the user via the user device 102a”][Lin: 0109 “This output upsampled image can be presented”](teaches that after inpainting and upsampling, the system provides the upsampled image to the user via the user device and that the output image can be presented).
Regarding claim 14, Lin discloses the system of claim 10, wherein the operations further comprise generating, using a generative neural network and from the digital image, the first modified digital image having the generated content portion at the first resolution [Lin: 0106 “image inpainting is performed on the downsampled image to generate a downsampled image result”][Lin: 0003 “the image inpainting system can implement one or more neural net works based on a generative adversarial network architecture ( e.g. , comprised of a generator and a discriminator”][Lin: 0045 “the downsampled image with the designated portion filled using inpainting”], wherein the first resolution of the generated content portion is lower than a resolution of the digital image [Lin: 0035 “image can be downsampled to a low - resolution image”].
Regarding claim 17, Lin discloses the non-transitory computer-readable medium of claim 16, wherein the operations further comprise generating a super-resolved digital image by compositing the second modified digital image with the digital image having the set of pixels to be replaced with the generated content portion [Lin: 0116 “In particular , information from the high - resolution feature patches can be used to replace pixel information within the designated portion of the high - resolution image”][Lin: 0139 “Reconstructed image 714 can comprise a feature map including original pixels ( e.g. , outside of designated portion 702a ) and replaced pixels ( e.g. , from the high - resolution feature patches used to replace pixels within designated portion 702a ) .”](teaches taking the input image 702 and replacing pixel information in the designated portion with pixel information derived from the reconstruction upscaling process, while retaining original pixels outside the designated portion).
Claims 10 and 16 are system and non-transitory computer readable medium claims corresponding to method claim 1 without any additional limitations. Thus, claims 10 and 16 are rejected for the same reasons as claim 1 above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (U.S. Patent Publication No. 2021/0342984), in view of Ma et al. (U.S. Patent Publication No. 2020/0162789).
Regarding claim 2, Lin discloses the computer-implemented method of claim 1, further comprising:
determining a resolution ratio based on a third resolution of the digital image and the first resolution of the generated content portion from the first modified digital image (interpreted as compute a value representing how the original image resolution compares to the lower resolution of the generated content portion in the first modified image)[Lin: 0106 “a 512x512 image can be downsampled to 256x256”][Lin: 0021 “Such upsampling can increase the resolution of the low - resolution image result ( e.g. , upsample 256x256 to 512x512”](teaches a high resolution image (51x512) and a lower version (256x256) that is later upsampled, the ratio between the resolutions is a resolution ratio based on the high resolution image and the lower resolution inpainting); but fails to explicitly disclose and determining that the resolution ratio is above a low threshold, wherein generating the second modified digital image using the super resolution neural network comprises using the super resolution neural network to generate the second modified digital image based on determining that the resolution ratio is above the low threshold.
However, Ma discloses and determining that the resolution ratio is above a low threshold [Ma: 0010 “By setting specific threshold values of SI and TI for different resolutions , which can be derived from testing a range of different SI and TI values for different content , the down scaling factor that is oversampled can be screened out”](teaches resolution ratio in the form of downscaling factor and using threshold value to determine whether a candidate scaling/downscaling factor is acceptable), wherein generating the second modified digital image using the super resolution neural network comprises using the super resolution neural network to generate the second modified digital image based on determining that the resolution ratio is above the low threshold (interpreted as the system uses the super resolution neural network to generate the higher resolution output because the ratio/scale condition is satisfied) [Ma: 0012 “a deep learning based super resolution method is used in the learned resolution scaling module to restore the HR representation before display rendering at the client”][Ma: 0032 “the compressed video downscaled at downscaling factor roand encoded at bitrate R , will be upscaled using its default model M ( R1 , ro ) at the client”](teaches that when content has been downscaled by a particular downscaling factor, the client performs deep learning super resolution upscaling using a corresponding model. Because Ma also teaches thresholding used to screen/decide among downscaling factors (ratio related selection), Ma supports that the SR model is applied based on the outcome of the ratio/threshold related determination).
Lin and Ma are considered to be analogous to the claimed invention because they are in the same field of neural network-based image/video resolution scaling and reconstruction. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Lin to incorporate Ma’s teachings of using threshold-based determination of a resolution scaling factor. The motivation for such a combination would provide the benefit of performance and quality control.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (U.S. Patent Publication No. 2021/0342984), in view of Zhe Lin et al. (U.S. Patent Publication No. 2013/0034313).
Regarding claim 3, Lin discloses the computer-implemented method of claim 1, further comprising:
determining a resolution ratio based on a third resolution of the digital image and the first resolution of the generated content portion from the first modified digital image (interpreted as compute a ratio/scale factor comparing the resolution of the digital image (third resolution) and the resolution of the generated content portion in the first modified digital image) [Lin: 0044 “For example , a 512x512 image can be downsampled to 256x256 image”][Lin: 0021 “Such upsampling can increase the resolution of the low - resolution image result ( e.g. , upsample 256x256 to 512x512”](teaches a higher resolution and a corresponding lower resolution used for inpainting and then upsampling, the ratio between these two resolutions is the claimed resolution ratio based on the third resolution and the first resolution); but fails to explicitly disclose determining that the resolution ratio is above a high threshold; and
up-sampling the second modified digital image to generate a third modified digital image that corresponds to the digital image and includes the generated content portion at the third resolution of the digital image, wherein the third resolution is higher than the second resolution,
wherein providing the super-resolved digital image generated from the second modified digital image for display on the client device comprises providing the super-resolved digital image generated from the third modified digital image for display on the client device.
However, Zhe Lin discloses determining that the resolution ratio is above a high threshold [Zhe Lin: 0040 “Embodiments provide a technique for high-quality single image upscaling. Some embodiments recursively upscale the input image by a small scale factor at each iteration until a desired magnification factor is reached.”](factor here corresponds to threshold)[Zhe Lin: 0073 “For large upscaling factors, embodiments iteratively upscale the image by multiple times”](teaches scaling factor that trigger different handling); and up-sampling the second modified digital image to generate a third modified digital image that corresponds to the digital image and includes the generated content portion at the third resolution of the digital image, wherein the third resolution is higher than the second resolution (interpreted as perform an additional upsampling step on an already modified image to create another modified image at a higher resolution, third resolution > second resolution)[Zhe Lin: 40 “Embodiments provide a technique for high-quality single image upscaling. Some embodiments recursively upscale the input image by a small scale factor at each iteration until a desired magnification factor is reached”][Zhe Lin: 0071 “Some embodiments then Zoom 350 X using bicubic interpolation by the factor of s to get Y 340, an upsampled version of input image Xo310 which is missing high-frequency information.”](teaches multiple stage upsampling (recursively upscaling) until a desired factor is reached. This inherently includes an intermediate upscaled image and further upsampling step to reach the desired magnification (corresponding to the 3rd modified digital image), where the final third resolution is higher than the intermediate second resolution), wherein providing the super-resolved digital image generated from the second modified digital image for display on the client device comprises providing the super-resolved digital image generated from the third modified digital image for display on the client device (interpreted as the image ultimately provided for display is the one generated from the further upsampled image) [Zhe Lin: 0058 “Module 120 generates as output one or more modified (upscaled) images 130.”](teaches generating output modified images as the final result of the recursive upscaling process. That final output corresponds to the third modified digital image that the system provides for display).
Lin and Zhe Lin are considered to be analogous to the claimed invention because they are in the same field of neural network-based image/video resolution scaling and reconstruction. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Lin to incorporate Zhe Lin’s teachings of handling large upscaling factors by performing additional upsampling to reach desired final magnification. The motivation for such a combination would provide the benefit of quality control.
Claim 4 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (U.S. Patent Publication No. 2021/0342984), in view of Kreis et al. (U.S. Patent Publication No. 2023/0109379).
Regarding claim 4, Lin discloses the computer-implemented method of claim 1, but fails to explicitly disclose wherein generating the second modified digital image using the super resolution neural network comprises generating the second modified digital image using a generative adversarial network.
However, Kreis discloses wherein generating the second modified digital image using the super resolution neural network comprises generating the second modified digital image using a generative adversarial network (GAN)[Kreis: 0002 “super-resolution, image-to-image translation, and image editing. One of the most popular classes of generative models includes generative adversarial networks (GANs), which can generate high quality image or video content.”](teaches super resolution as an application area of generative modeling, and identifies GANs as a class of generative models, for a super resolution neural network implementation, Kreis teaches that the super resolution generation can be performed using a GAN).
Lin and Kreis are considered to be analogous to the claimed invention because they are in the same field of image processing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Lin to incorporate Kreis’s teachings of utilizing GAN. The motivation for such a combination would provide the benefit of output quality.
Claim 15 is a non-transitory computer readable medium claim corresponding to claim 4 with one additional limitation “using a diffusion neural network” [Kreis: 0023 “Various SGMs utilize a diffusion process”]. Thus, claim 15 is rejected for the same reasons as claim 4 above.
Claims 5 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (U.S. Patent Publication No. 2021/0342984), in view of Tatke et al. (U.S. Patent No. 8,744,213).
Regarding claim 5, Lin discloses the computer-implemented method of claim 1, but fails to explicitly disclose wherein: determining the first set of tiles from the digital image comprises determining, from the digital image, a first set of overlapping tiles having pairs of tiles that overlap with at least a threshold number of overlapping pixels; and determining the second set of tiles from the first modified digital image comprises determining, from the first modified digital image, a second set of overlapping tiles having additional pairs of tiles that overlap with at least the threshold number of overlapping pixels.
However, Tatke discloses wherein: determining the first set of tiles from the digital image comprises determining, from the digital image, a first set of overlapping tiles having pairs of tiles that overlap with at least a threshold number of overlapping pixels (tiles are interpreted as sub image portions and overlapping tiles means neighboring tiles share a common pixel region)(Tatke: Col. 1, Lines 39-44 “The present invention provides method for reconstituting an image comprising dividing an image into a number of overlapping tiles captured in consecutive Snapshots and reconstituting the image with a magnification without Substantial loss of accuracy.”)(Tatke: Col. 3, Lines 21-22 “images have a common overlap of at least N pixels between them”)(teaches dividing an image into overlapping tiles and adjacent tiles have a common overlap of at least certain amount of pixels which is a threshold number); and determining the second set of tiles from the first modified digital image comprises determining, from the first modified digital image, a second set of overlapping tiles having additional pairs of tiles that overlap with at least the threshold number of overlapping pixels (interpreted as the same overlapping tiling requirement as the previous limitation but applied to the first modified digital image)(Tatke: Col. 4, Lines 64-65 “dividing an image into a number of overlapping tiles”)(Tatke: Col. 5, Lines 3-4 “These images have a common overlap of at least N pixels between them. N is greater than 1 pixel.”)(teaches the operation of dividing an image into overlapping tiles with a common overlap of at least N pixels, Tatkes taught overlapping tiling operation applies to determining the second set of tiles from that first modified digital image, and the overlap condition satisfies the claimed threshold overlap requirement for those tile pairs).
Lin and Tatke are considered to be analogous to the claimed invention because they are in the same field of neural network-based image/video resolution scaling and reconstruction. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Lin to incorporate Tatke’s teachings of dividing an image into overlapping tiles having a common overlap of at least N pixels. The motivation for such a combination would provide the benefit of stitching quality.
Regarding claim 6, Lin discloses the computer-implemented method of claim 5, but fails to explicitly disclose wherein determining the first set of overlapping tiles comprises generating a grid of overlapping tiles positioned within boundaries of the digital image, causing each tile in the first set of overlapping tiles to contain image pixels and omit padding pixels.
However, Tatke discloses wherein determining the first set of overlapping tiles comprises generating a grid of overlapping tiles positioned within boundaries of the digital image (Tatke: Col. 10, Lines 32-33 “the slide is translated in XandY (the focus axis is in Z), pausing at each frame location while an image is acquired.”)(Tatke: Col. 10, Lines 37-39 “(This number allows for some overlap between adjacent frames, which is required to adaptively stitch the frames together.)”)(teaches a tiled image scan where images are acquired at successive frame locations as the stage is translated in X and Y which is a grid like acquisition pattern, further teaches an overlap between adjacent frames. Because the acquisition is performed to scan the area of interest on the slide, the resulting frame tiles are positioned within the boundaries of that captured image area), causing each tile in the first set of overlapping tiles to contain image pixels and omit padding pixels (Tatke: Col. 10, Line 34 “an image is acquired”)(teaches that tatkes tiles are acquired image frames at each frame location, an acquired frame necessarily consists of actual captured pixels).
Lin and Tatke are considered to be analogous to the claimed invention because they are in the same field of neural network-based image/video resolution scaling and reconstruction. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Lin to incorporate Tatke’s teachings of generating a grid of overlapping tiles positioned within the image boundaries so that each tile contains only image pixels. The motivation for such a combination would provide the benefit of improving efficiency.
Claims 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (U.S. Patent Publication No. 2021/0342984), in view of Yi et al. (U.S. Patent Publication No. 2021/0150678).
Regarding claim 9, Lin discloses the computer-implemented method of claim 1, but fails to explicitly disclose further comprising determining a third set of tiles from a soft or hard mask that corresponds to the digital image, wherein generating the second modified digital image based on the first set of tiles and the second set of tiles comprises generating the second modified digital image based on the first set of tiles, the second set of tiles, and the third set of tiles.
However, Yi discloses further comprising determining a third set of tiles from a soft or hard mask that corresponds to the digital image [Yi: 0074 “The inpainting mask is a 2D matrix containing binary data”](teaches that this inpainting mask is a binary 2d matrix (hard mask)), wherein generating the second modified digital image based on the first set of tiles and the second set of tiles comprises generating the second modified digital image based on the first set of tiles, the second set of tiles, and the third set of tiles [Yi: 0088 “Inputs to the inpainting generator 101 are the low - resolution image ( from the down - sampler 302 in FIG. 3 ) and the inpainting mask”](teaches that the generation module takes both the image content and the mask as input corresponding to the requirement that the generation is performed based on the image derived inputs and the mask derived input).
Lin and Yi are considered to be analogous to the claimed invention because they are in the same field of image processing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Lin to incorporate Yi’s teachings of using a hard mask as additional input. The motivation for such a combination would provide the benefit of quality control.
Regarding claim 18, Lin discloses the non-transitory computer-readable medium of claim 17, but fails to explicitly disclose wherein compositing the second modified digital image with the digital image comprises compositing the second modified digital image with the digital image using a mask that corresponds to the digital image.
However, Yi discloses wherein compositing the second modified digital image with the digital image comprises compositing the second modified digital image with the digital image using a mask that corresponds to the digital image [Yi: 0074 “The inpainting mask is a 2D matrix containing binary data ( e.g. , each entry in the matrix has a value of either “ 1 ” or “ O ” ) . The binary data of the inpainting mask may have a one - to - one mapping with a corresponding pixel in the original high - resolution image , such that the inpainting mask may have dimensions matching the pixel dimension of the original high - resolution image”][Yi: 0083 “The inpainting mask is applied to the low - fre quency inpainted image , at an apply mask operation 312 , to obtain the low - frequency inpainted area only for the inside mask area”][Yi: 0084 “The high - resolution inpainted area is added to the original high - resolution image , at a second addition opera tion 316.”](teaches a binary inpainting mask that has a one to one mapping with the original image pixels and matching dimensions and using that mask by applying the mask to obtain an inpainted area image only for the inside mask area, and then adding that inpainted area to the original image. Applying the mask and then adding the masked inpainted area to the original image is compositing the modified image with the original image using a corresponding mask).
Lin and Yi are considered to be analogous to the claimed invention because they are in the same field of image processing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Lin to incorporate Yi’s teachings of using a corresponding mask that controls where the inpainted pixels are applied. The motivation for such a combination would provide the benefit of improving visual quality.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (U.S. Patent Publication No. 2021/0342984), in view of Sidar et al. (U.S. Patent Publication No. 2016/0335751).
Regarding claim 13, Lin discloses the system of claim 10, but fails to explicitly disclose wherein generating the second modified digital image from the third set of overlapping tiles comprises: determining, from the third set of overlapping tiles, one or more overlapping portions where four tiles overlap; and generating the second modified digital image by blending the one or more overlapping portions using bilinear blending.
However, Sidar discloses generating the second modified digital image from the third set of overlapping tiles comprises: determining, from the third set of overlapping tiles, one or more overlapping portions where four tiles overlap [Sidar: 0098 “each tile overlaps with its neighboring tiles”][Sidar: 0108 “For each output Lab pixel, four or sixteen respective tiles may be selected. The Lab pixel may reside in a rectangle formed by center points of the selected four (or sixteen) tiles. In embodiments, the pixel’s Lab values feed the LUTs of its respective four (or sixteen) tiles.”] (teaches an image partitioned into tiles and the tiles may overlap); and generating the second modified digital image by blending the one or more overlapping portions using bilinear blending [Sidar: 0108 “The final Lab values of the output pixel may be a bi-linear interpolation of the four sets of Lab values (or bi-cubic interpolation of the sixteen sets of Lab values).”].
Lin and Sidar are considered to be analogous to the claimed invention because they are in the same field of image processing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Lin to incorporate Sidar’s teachings of utilizing bilinear blending. The motivation for such a combination would provide the benefit of smoother transitions.
Allowable Subject Matter
Claims 7-8, 11-12, and 19-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AHMED TAHA whose telephone number is (571)272-6805. The examiner can normally be reached 8:30 am - 5 pm, Mon - Fri.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, XIAO WU can be reached at (571)272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AHMED TAHA/Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613