DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
The current application claims foreign priority from the Korean application (KR10-2023-0088014). Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 04/30/2024 and 01/31/2025 are in compliance with the provisions of 37 CFR 1.97 and have been considered by the examiner.
Claim Objections
Claim 18 is objected to because of the following informalities: in claim 18 line 2, “one or more processor” should read “one or more processors”. Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 3, 8-9 and 17-18 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Milanfar et al. (US 2017/0206632 A1).
Regarding claim 1, Milanfar discloses a processor-implemented method with super resolution (Milanfar paragraph 0060: “The upscaling runtime systems 300a-b use the filters generated by the upscaling training system 200 to upscale low resolution images 302 into higher resolution images, e.g., output images 312”), the method comprising: determining a direction type of an input image based on a gradient of the input image (Milanfar paragraph 0062: “determine a property of each of the patches of pixels from the upscaled low resolution image, e.g., an angle of a gradient”); acquiring a first intermediate image that is a super-resolution image corresponding to the input image based on a residual look-up table (LUT) corresponding to a kernel set mapped to the determined direction type (Milanfar paragraphs 0063-0064: “uses the property of the patches in the same bucket to access a lookup table and determine a filter to apply to each of the patches in the bucket…aggregate 310 the filtered patches”); determining a LUT strength map of the input image (Milanfar paragraph 0065: “determine randomness weights 314 that reflect a degree to which a pixel patch contains noise or structured content, e.g., a geometric shape”) based on the gradient of the input image (Milanfar paragraph 0066: “may analyze the low resolution image 302, the upscaled low resolution image, the filtered patches, or a combination of two or more of these, to determine whether a patch or area of the image contains noise or structured content”) and a preset tuning parameter (Milanfar paragraph 0065: “weight values between zero and one, inclusive”); and generating an output image that is a super-resolution image of the input image based on the LUT strength map and the first intermediate image (Milanfar FIG. 3A, paragraphs 0066-0067: “uses the randomness weights to generate a weighted average 316 of the upscaled low resolution image and the aggregated filtered patches…aggregates the selected patches to generate the output image 312”).
Regarding claim 3, Milanfar discloses the method of claim 1, wherein the generating of the output image comprises: acquiring a second intermediate image based on the LUT strength map and the first intermediate image (Milanfar paragraph 0066: “uses the randomness weights to generate a weighted average 316 of the upscaled low resolution image and the aggregated filtered patches”); and generating the output image based on a baseline image acquired by applying interpolation for super resolution to the input image (Milanfar FIG. 3A, paragraph 0033: “may perform a lightweight upscaling process on the low resolution image, e.g., to create an upscaled low resolution image 102. For example, the upscaling training system may perform bilinear interpolation on the low resolution image”) and the second intermediate image (Milanfar FIG. 3A, paragraph 0067: “aggregates the selected patches to generate the output image 312”).
Regarding claim 8, Milanfar discloses the method of claim 1, wherein the determining of the LUT strength map of the input image comprises determining a LUT strength value corresponding to a pixel detected as a corner in the input image (Milanfar paragraph 0035: “when analyzing a patch 106c from the upper left side of the low resolution image, e.g., a patch that is ten by ten pixels, the upscaling training system may determine that an edge represented in the patch has approximately a forty-five degree angle or that the derivative of the angle the edge is ten”) based on a gradient of the pixel detected as the corner in the input image and the tuning parameter (Milanfar paragraph 0065: “determine randomness weights 314 that reflect a degree to which a pixel patch contains noise or structured content, e.g., a geometric shape”), and the generating of the output image comprises generating the output image based on the LUT strength value corresponding to the pixel detected as the corner in the input image and the first intermediate image, in response to a first pixel in the first intermediate image corresponding to the pixel detected as the corner in the input image (Milanfar FIG. 3A, paragraphs 0066-0067: “uses the randomness weights to generate a weighted average 316 of the upscaled low resolution image and the aggregated filtered patches…aggregates the selected patches to generate the output image 312”).
Regarding claim 9, Milanfar discloses the method of claim 1, further comprising setting the output image as the input image and repeating the determining of the direction type of the input image, the acquiring of the first intermediate image, the determining of the LUT strength map, and the generating of the output image (Milanfar paragraph 0115: “may repeat the application of filters to patches from the filtered images until determining that a threshold has been reached”).
Regarding claim 17, it is the corresponding non-transitory computer-readable storage medium configured to execute the method claimed in claim 1. Therefore, Milanfar discloses the limitations of claim 17 as it does the limitations of claim 1.
Regarding claim 18, it is the corresponding apparatus configured to execute the method claimed in claim 1. Therefore, Milanfar discloses the limitations of claim 18 as it does the limitations of claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 2 is rejected under 35 U.S.C. 103 as being unpatentable over Milanfar in view of Zhu et al. (NPL "A Range and Velocity Ambiguity Resolution Method Based on Ambiguity Matrix Completion and Elimination with Low SNR").
Regarding claim 2, Milanfar discloses the method of claim 1, wherein the acquiring of the first intermediate image comprises: determining a receptive field corresponding to a reference pixel in the input image based on a kernel included in the kernel set (Milanfar paragraphs 0042, 0044: “may generate a patch for each pixel that is centered on the respective pixel…may generate NxN patches of pixels, or NxM patches of pixels”). However, Milanfar fails to disclose updating values of pixels in the receptive field with values obtained by subtracting a value of the reference pixel from the values of the pixels in the receptive field; and obtaining values stored in a residual LUT with respect to a combination of the updated values of the pixels other than the reference pixel in the receptive field. In the related art of residual look-up table, Zhu discloses updating values of pixels in the receptive field with values obtained by subtracting a value of the reference pixel from the values of the pixels in the receptive field (Zhu page 2, right-hand column [RHC], last paragraph: “we choose a reference PRF, and put the differences between the other measurement range values and the reference range value in the look-up table”); and obtaining values stored in a residual LUT with respect to a combination of the updated values of the pixels other than the reference pixel in the receptive field (Zhu page 2, RHC, last paragraph: “Look up the table according to the differences between the measurement range values which come from the radar measurement result, and by using the least mean-square error rule, we can get the true range value”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Milanfar to incorporate the teachings of Zhu to resolve range ambiguity (Zhu page 2, RHC, last paragraph).
Claim(s) 4 is rejected under 35 U.S.C. 103 as being unpatentable over Milanfar in view of Ichihashi et al. (US 2010/0119176 A1).
Regarding claim 4, Milanfar discloses the method of claim 1, wherein the LUT strength map of the input image comprises a LUT strength value of each pixel in the input image (Milanfar paragraph 0005: “The upscaling system may use different weights for each pixel”). However, Milanfar fails to disclose the LUT strength value is determined based on a product of a gradient of each pixel in the input image and the tuning parameter. In the related art of super-resolution, Ichihashi discloses the LUT strength value is determined based on a product of a gradient of each pixel in the input image and the tuning parameter (Ichihashi equation 35, paragraph 0428: “the statistical diagonal interpolation pixel multiplied by the weight”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Milanfar to incorporate the teachings of Ichihashi to increase the image quality of an image having a higher resolution that is generated from a plurality of images (Ichihashi paragraph 0020).
Claim(s) 5 is rejected under 35 U.S.C. 103 as being unpatentable over Milanfar and Ichihashi in view of Winzell et al. (US 2024/0153050 A1).
Regarding claim 5, Milanfar, modified by Ichihashi, discloses the method of claim 4. However, Milanfar and Ichihashi fail to disclose a value of a first pixel included in the input image is determined to be a predetermined maximum value in response to the product being greater than the predetermined maximum value, is determined to be a predetermined minimum value in response to the product being less than the predetermined minimum value, and is determined to be the product in response to the product of the magnitude of the gradient of the first pixel and the tuning parameter being less than or equal to the maximum value and greater than or equal to the minimum value. In the related art of image processing, Winzell discloses a value of a first pixel included in the input image is determined to be a predetermined maximum value in response to the product being greater than the predetermined maximum value (Winzell paragraph 0077: “if the addition of a pixel value difference to a pixel value in the transformed second image 512 would cause it to be larger than the maximum value, it will be set to be equal to the maximum value”), is determined to be a predetermined minimum value in response to the product being less than the predetermined minimum value (Winzell paragraph 0077: “if the addition of a pixel value difference to a pixel value in the transformed second image 512 would cause it to be smaller than the minimum value, it will be set to be equal to the minimum value”), and is determined to be the product in response to the product of the magnitude of the gradient of the first pixel and the tuning parameter being less than or equal to the maximum value and greater than or equal to the minimum value (Winzell paragraph 0077: “After the pixel value differences in the image buffer 504 have been added to the pixel values of the transformed second image 512, the pixel values in the resulting image 514 may be restricted such that they are between a minimum and a maximum value”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Milanfar and Ichihashi to incorporate the teachings of Winzell to prevent artifacts in the processed images (Winzell paragraph 0026).
Claim(s) 6 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Milanfar in view of Yin et al. (NPL "Online Video Streaming Super-Resolution with Adaptive Look-Up Table Fusion").
Regarding claim 6, Milanfar discloses the method of claim 1, wherein the kernel set comprises a plurality of kernels (Milanfar paragraph 0004: “The upscaling system may learn multiple filters and apply the filters to patches or groups of pixels in the initial upscaled image”). However, Milanfar fails to disclose acquiring the first intermediate image based on a weighted sum of super-resolution operation results respectively corresponding to the kernels based on weights determined respectively for the kernels. In the related art of super-resolution, Yin discloses acquiring the first intermediate image based on a weighted sum of super-resolution operation results respectively corresponding to the kernels based on weights determined respectively for the kernels (Yin equation 1, Abstract: “a set of LUT bases are built upon state-of-the-art SR networks pre-trained on different degraded data, and those LUT bases are combined with extracted weights from lightweight convolution modules”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Milanfar to incorporate the teachings of Yin to adapt to any degradation (Yin page 3, RHC, last paragraph).
Regarding claim 19, it is the corresponding apparatus configured to execute the method claimed in claim 6. Therefore, Milanfar, modified by Yin, discloses the limitations of claim 19 as it does the limitations of claim 6.
Claim(s) 7 is rejected under 35 U.S.C. 103 as being unpatentable over Milanfar in view of Toutounchi et al. (NPL "An Efficient Video Super-Resolution Approach based on Sparse Representation").
Regarding claim 7, Milanfar discloses the method of claim 1, wherein the acquiring of the first intermediate image comprises updating the first intermediate image with an image acquired by overlapping patches corresponding to different regions of the first intermediate image, each of the patches of the first intermediate image is a set of pixels corresponding to super-resolution results of each pixel in the input image (Milanfar paragraphs 0018-0019: “Creating the pixel subsets of the upscaled image may include creating overlapping pixel subsets…applying, for each of the groups of subsets, a second filter to each of the final pixel subsets in the group to create a second final pixel subset, a combination of all of the second final pixel subsets representing a second higher resolution image of the content with the third resolution”). However, Milanfar fails to disclose a pixel value of a region where the patches are overlapped in the updated first intermediate image is determined to be an average of pixel values of the overlapped patches. In the related art of super-resolution, Toutounchi discloses a pixel value of a region where the patches are overlapped in the updated first intermediate image is determined to be an average of pixel values of the overlapped patches (Toutounchi page 2, left-hand column [LHC], first full paragraph: “overlapping blocks are employed. As a simple approach, it is possible to average the feature values in overlapped regions between adjacent blocks”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Milanfar to incorporate the teachings of Toutounchi to avoid artifacts at the block boundaries (Toutounchi page 2, LHC, first full paragraph).
Allowable Subject Matter
Claims 10-16 and 20 are allowed. The following is a statement of reasons for the indication of allowable subject matter:
Independent claim 10, along with its dependent claims 11-16, and independent claim 20 are allowable because independent claims 10 and 20 comprise of allowable subject matter. The cited prior art either alone or in combination fails to disclose, teach, or suggest: the residual LUTs comprise super-resolution operation results of a combination of pixel values, the number of which is "1" smaller than sizes of corresponding kernels.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Gottscho et al. (US 2024/0378147 A1) discloses an offset LUT omitting entries corresponding to elements of the kernel that do not correspond to an element of the input or would be multiplied by a zero value of the input for the convolution operation.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRISTINE ZHAO whose telephone number is (703)756-5986. The examiner can normally be reached Monday - Friday 9:00am - 5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571)270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.Z./Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights