DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-6, 8-13, 15-16 and 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. US Patent Pub. No.: 2017/0347110, hereinafter, ‘Wang’ in view of El- Khamy et al. US Patent Pub. No.: 2018/0293707 A1, hereinafter, ‘El- Khamy’.
Consider Claim 10 and as applied to the method of Claim 1, Wang teaches a computing apparatus comprising: one or more processors; and memory storing instructions (e.g., this is met based on the computer program product claimed in at least Claim 20 ) configured to cause the one or more processors to: determine values of respective portions of an image(e.g., values noted in at least 0266, 0344 and 0387); according to the values: upscale first portions, among the portions, by interpolation of the first portions performed according to a target (e.g., see at least 0202 -In some embodiments, resampling into the higher dimension space of visual data from a low-quality to high-quality domain happens before being processed through a super resolution network for enhancement, for example in some of these embodiments being enhanced using bicubic interpolation – see also 0409-0410 and 0414); upscale second portions, among the portions, by processing the second portions with a neural network that upsamples the second portions according to the target; and generating a version of the image upscaled according to the upscale parameter based on the upscaled first portions and the upscaled second portions(e.g., see at least 0414 – “the convolutional neural network consists of a plurality of non-linear mapping layers followed by a sub-pixel convolution layer. The benefit is the reduced computational complexity compared to a three-layer convolutional network used to represent the patch extraction and representation, non-linear mapping and reconstruction stages in the conventional sparse-coding-based image reconstruction methods and using an upscaled and interpolated version of the low-resolution input. The convolutional neural network of the embodiment uses learned filters on the feature maps to super resolved the low-resolution data into high-resolution data (instead of using hand crafted interpolation or a single filter on the input images/frames”).
Wang further teaches downsampling by a upscaling ratio r. - 0410, but does not specifically teach a target upscale parameter.
In analogous art, El- Khamy teaches a method for super resolution imaging, the method includes: receiving, by a processor, a low resolution image; generating, by the processor, an intermediate high resolution image having an improved resolution compared to the low resolution image; generating, by the processor, a final high resolution image based on the intermediate high resolution image and the low resolution image; and transmitting, by the processor, the final high resolution image to a display device for display thereby. In paragraph 0058, El- Khamy specifically teaches the network receives a LR input image, or alternatively, a bicubic upsampled version of the LR input image, where the upsampling ratio is according to a target upsampling ratio.
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date to try where the upsampling ratio is according to a target upsampling ratio for the purpose of improving resolution.
Consider Claim 18, Wang teaches a method performed by a computing device, the method comprising: based on first image-quality measures of respective first regions of an image (i.e., low -quality resolution ), upscaling the first regions according to a target by interpolating the first regions(e.g., see at least 0202 -In some embodiments, resampling into the higher dimension space of visual data from a low-quality to high-quality domain happens before being processed through a super resolution network for enhancement, for example in some of these embodiments being enhanced using bicubic interpolation – see also 0409-0410 and 0414 ); based on second image-quality measures of respective second regions of the image, upscaling the second regions according to a target by passing the second regions through an upsampling neural network that performs upsampling the second regions; and forming an upscaled version of the image based on the upscaled first regions and the upscaled second regions(e.g., see at least 0414 – “the convolutional neural network consists of a plurality of non-linear mapping layers followed by a sub-pixel convolution layer. The benefit is the reduced computational complexity compared to a three-layer convolutional network used to represent the patch extraction and representation, non-linear mapping and reconstruction stages in the conventional sparse-coding-based image reconstruction methods and using an upscaled and interpolated version of the low-resolution input. The convolutional neural network of the embodiment uses learned filters on the feature maps to super resolved the low-resolution data into high-resolution data (instead of using hand crafted interpolation or a single filter on the input images/frames”).
Wang further teaches downsampling by a upscaling ratio r. - 0410, but does not specifically teach a target upscale parameter.
In analogous art, El- Khamy teaches a method for super resolution imaging, the method includes: receiving, by a processor, a low resolution image; generating, by the processor, an intermediate high resolution image having an improved resolution compared to the low resolution image; generating, by the processor, a final high resolution image based on the intermediate high resolution image and the low resolution image; and transmitting, by the processor, the final high resolution image to a display device for display thereby. In paragraph 0058, El- Khamy specifically teaches the network receives a LR input image, or alternatively, a bicubic upsampled version of the LR input image, where the upsampling ratio is according to a target upsampling ratio.
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date to try where the upsampling ratio is according to a target upsampling ratio for the purpose of improving resolution.
Consider Claim 2, Wang teaches the claimed invention except wherein the upscale parameter is an upscale ratio that is not an integer.
Wang further teaches downsampling by a upscaling ratio r. - 0410, but does not specifically teach a target upscale parameter.
In analogous art, El- Khamy teaches a method for super resolution imaging, the method includes: receiving, by a processor, a low resolution image; generating, by the processor, an intermediate high resolution image having an improved resolution compared to the low resolution image; generating, by the processor, a final high resolution image based on the intermediate high resolution image and the low resolution image; and transmitting, by the processor, the final high resolution image to a display device for display thereby. In paragraph 0058, El- Khamy specifically teaches the network receives a LR input image, or alternatively, a bicubic upsampled version of the LR input image, where the upsampling ratio is according to a target upsampling ratio (i.e., the ratio reflects the teachings of a non-integer).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date to try wherein the upscale parameter is an upscale ratio that is not an integer for the purpose of improving resolution.
Consider Claim 3, Wang teaches wherein the neural network is configured to upscale in successive layers, each layer increasing the size of an input thereto to a size less than twice the size of the input (i.e., this reflects the teachings of a CNN, e.g., 0290 teaches - there is provided a method for enhancing at least a section of lower-quality visual data using a hierarchical algorithm, the method comprising steps of: receiving at least one section of lower-quality visual data; extracting a subset of features, from the at least one section of lower-quality visual data; forming a plurality of layers of reduced-dimension visual data from the extracted features; and enhancing the plurality of layers of reduced dimension visual data to form at least one section of higher-quality visual data).
Consider Claim 4, Wang teaches wherein the successive layers are, respectively, upsampling layers that perform upsampling(e.g., 0209 teaches “ in some embodiments, the final layer can be a sub-pixel convolution layer that is capable of upscaling low resolution feature maps into high resolution output, thus avoiding using a non-adaptive generic or handcrafted bicubic filter, and allowing for a plurality of more complex upscaling filters for each feature map. Further, in some of these embodiments, having the final layer perform super resolution from the lower resolution space can reduce the complexity of the super resolution operation”.)
Consider Claim 5, Wang teaches wherein the values correspond to complexities of the respective portions (e.g., values noted in at least 0266, 0344 and 0387. The Examiner suggest explicitly defining “complexities”. Applicant is remined although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Consider Claim 6, Wang teaches wherein the values are determined by a feature extractor neural network (e.g., see at least 0413 – “the high-resolution data is super resolved from the low-resolution feature maps by a sub-pixel convolution layer, to learn the upscaling operation for image and video super resolution. If the last or final layer performs upscaling, feature extraction occurs through non-linear convolutions in the low-resolution dimensions”).
Consider Claim 8, Wang teaches wherein the neural network performs gradual super-resolution upscaling(e.g., see at least 0433 –“In some and other embodiments, a neural network can have multi-stage upscaling (or other function) where an earlier layer upscales and then a later layer upscales, for example a middle layer upscales by 2× and then the last layer upscales by 2×. This type of “chained” approach can allow for neural networks to be trained in a long network (or “chain”) of functional layers, for example having a range of upscaling factors (e.g. 2×, 3× and 4×) with multiple upscaling layers and output layers. One of more of these layers can be sub-pixel convolution layers of the described embodiment” ).
Consider Claim 9, Wang teaches wherein the values are determined from image content of the respective images such that each portion's value depends on its image content(e.g., values noted in at least 0266, 0344 and 0387. “extracted standardised features are used to produce a value or series of values based on a metric from the input data.”).
Consider Claim 11, Wang teaches the claimed invention except wherein the neural network is configured to upscale to arbitrary ratios including a ratio between 1 and 2.
Wang teaches downsampling by a upscaling ratio r. - 0410, but does not specifically teach a target upscale parameter.
In analogous art, El- Khamy teaches a method for super resolution imaging, the method includes: receiving, by a processor, a low resolution image; generating, by the processor, an intermediate high resolution image having an improved resolution compared to the low resolution image; generating, by the processor, a final high resolution image based on the intermediate high resolution image and the low resolution image; and transmitting, by the processor, the final high resolution image to a display device for display thereby. In paragraph 0058, El- Khamy specifically teaches the network receives a LR input image, or alternatively, a bicubic upsampled version of the LR input image, where the upsampling ratio is according to a target upsampling ratio (i.e., the ratio reflects the teachings of a non-integer).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date to try wherein the neural network is configured to upscale to arbitrary ratios including a ratio between 1 and 2 for the purpose of improving resolution.
Consider Claims 12 and 19, Wang teaches wherein a first portion among the first portions, does not overlap a second portion among the second portions (e.g., this is met based on at least claim 1 –“ receiving at least a plurality of neighbouring sections of lower-quality visual data; selecting a plurality of input sections from the received plurality of neighbouring sections of lower quality visual data; extracting features from the plurality of input sections of lower-quality visual data; and enhancing a target section based on the extracted features from the plurality of input sections of lower-quality visual data” .)
Consider Claim 13, Wang teaches wherein the instructions are further configured to cause the one or more processors to determine the first portions and the second portions using first neural network (e.g., see at least figure 3 - FIG. 3 illustrates the encoding process to generate for transmission, from high-resolution visual data, a combination of low-resolution visual data and convolution neural network able to use super resolution to increase the resolution of the low-resolution data).
Consider Claim 15, Wang teaches wherein first portions and the second portions are passed to the upsampling neural network, wherein the neural network comprises upsampling layers and an interpolation layer, wherein the upsampling layers perform the upsampling of the second portions, and wherein the interpolation layer performs the interpolation of the first portions (e.g., this teaching is suggested in at least 0202 –“resampling into the higher dimension space of visual data from a low-quality to high-quality domain happens before being processed through a super resolution network for enhancement, for example in some of these embodiments being enhanced using bicubic interpolation” ).
Consider Claims 16 and 20, Wang teaches wherein the first portions are not processed by the upsampling layers and the second portions are not processed by the interpolation layer (e.g., this is met by paragraph 0202 which suggest the implementation- resampling into the higher dimension space of visual data from a low-quality to high-quality domain happens before being processed through a super resolution network for enhancement, for example in some of these embodiments being enhanced using bicubic interpolation, but this enhancing does not add information useful for super resolution and forces the network to perform subsequent computation in a higher-quality domain when for example extracting feature maps and performing non-linear mapping. This “initial upscaling” makes the computation of the network/algorithm computationally expensive and increases memory requirements. In some embodiments, the low quality domain is a low resolution domain while the high quality domain is a high resolution domain).
Claim(s) 7 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. US Patent Pub. No.: 2017/0347110, hereinafter, ‘Wang’ in view of El- Khamy et al. US Patent Pub. No.: 2018/0293707 A1, hereinafter, ‘El- Khamy’ and further in view of Chang et al. US Patent Pub. No.: 2019/0378242 A1, hereinafter, ‘Chang’.
Consider Claim 7, Wang as modified by El-Khamy teaches the claimed invention except wherein the values correspond to textures of the respective portions.
In analogous art, Chang teaches super-resolution is the process of recovering a high-resolution (HR) image from a corresponding low-resolution (LR) image. Because a LR image lacks texture features (e.g., high-frequency details such as fur, hair, eyelashes, patterns, small items, and the like), super-resolution methods recover lost pixels of texture features, such as by interpolation or constraining neighborhood similarity. However, though these super-resolution methods may enhance edges, such as an edge of a building in an image, these methods tend to blur high-frequency texture features. Paragraph 0004 teaches- super-resolve a LR image based on reference images. Reference images are not constrained to have similar or same content as the LR image. Content features (e.g., low-frequency details such as an outline, color, shape, and the like) of a LR image are extracted into a content feature map of the LR image. Texture features, such as representing details of hair, needles of a tree, eyelashes, small items, and the like, are extracted from a reference image and a version of the LR image upscaled to a same or similar scale as the reference image into respective texture feature maps.
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date to try wherein the values correspond to textures of the respective portions for the purpose of improving super resolution.
Consider Claim 14, Wang as modified by El-Khamy teaches the claimed invention except wherein the first neural network comprises residual blocks connected in series.
In analogous art, Chang teaches wherein the first neural network comprises residual blocks connected in series (e.g., see figure 4 -414).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date to try wherein the first neural network comprises residual blocks connected in series for the purpose of improving super resolution.
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. US Patent Pub. No.: 2017/0347110, hereinafter, ‘Wang’ in view of El- Khamy et al. US Patent Pub. No.: 2018/0293707 A1, hereinafter, ‘El- Khamy’ and further in view of Netravali US Patent No.: 4,488,175.
Consider Claim 17, Wang as modified by El-Khamy teaches the claimed invention except wherein the values are predicted errors of the regions, wherein the first regions are selected for interpolation based on having predicted errors below a threshold, and wherein the second regions are selected for upsampling based on having predicted errors above the threshold.
In analogous art, Netravali teaches in regions where prediction errors are low, the predicted value of a nontransmitted element is instead used as a basis for the reconstruction. Increased efficiency is thus achieved at little increase in complexity, since the same prediction required for DPCM encoding is used for reconstruction of nontransmitted pels – abstract – “while the interpolative reconstruction of nontransmitted pels provided by the encoder of FIG. 2 may be adequate in areas of the picture in which spatial detail is low, noticeable blurring occurs in many other instances, which can be highly objectionable. This is avoided, in accordance with the present invention, by adapting the technique used to reconstruct the value of nontransmitted pels as a function of the quantized prediction error value of the pel being processed, and, if desired, of other nearby pels. In particular, interpolative reconstruction of the value of nontransmitted pels as described above is used in areas of the picture in which the prediction error exceeds a predetermined threshold…” – col. 5 lines 26-53.
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date to try wherein the values are predicted errors of the regions, wherein the first regions are selected for interpolation based on having predicted errors below a threshold, and wherein the second regions are selected for upsampling based on having predicted errors above the threshold for the purpose of improving the picture quality.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
WO 2020000333 A1 teaches image processing method involves determining (201) the upsample region based on a region excluding a region of interest (ROI) in an image. An upsampling operation in the upsample region is performed (202) without performing the upsampling operation in the ROI.
US 20210012459 A1 teaches a method for image processing includes determining an upsample region based on a region excluding a region of interest in an image; and performing an upsampling operation in the upsample region without performing the upsampling operation in the region of interest.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHARLES TERRELL SHEDRICK whose telephone number is (571)272-8621. The examiner can normally be reached 8A-5P.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew D Anderson can be reached at 571 272 4177. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHARLES T SHEDRICK/Primary Examiner, Art Unit 2646
Prosecution Insights