DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The Amendment filed on June 15, 2026 has been entered. Claims 1, 3–6, and 9–10 are currently pending. Claims 2 and 7–8 have been canceled. Claims 1, 3, 5, and 9 have been amended.
Response to Arguments
Applicant's arguments filed 06/15/2026 have been fully considered. Some arguments are persuasive with respect to the previous 35 U.S.C. § 112(b) rejections, the previous 35 U.S.C. § 101 rejection of claim 9, and the previous rejections under 35 U.S.C. § 102 and § 103 over Tao and Tao in view of Zhang, as explained below.
Applicant's arguments filed 06/15/2026 are persuasive with respect to the previous rejections under 35 U.S.C. § 112(b). Applicant's arguments, see pages 5–6 of the Remarks, state that original claim 2 has been incorporated into independent claim 1 with clarifying amendments defining the permissible range of index i and expressly defining the i-th module in each of the image restoration branch and the image super-resolution branch, and that claims 7 and 8 have been canceled. Upon reconsideration, the Examiner agrees that the amended claim 1 language resolves the indefiniteness previously identified regarding the scope of the index i and the composition of the i-th module. Applicant's arguments, see page 6 of the Remarks, further state that claim 3 has been amended to correct the erroneous duplicate reference to "an encoder" in connection with the up-sampling modules, clarifying that the up-sampling modules are recited as part of a decoder. The Examiner agrees that this amendment resolves the indefiniteness previously identified in claim 3. Therefore, the prior rejections of claims 2, 3, and 8 under 35 U.S.C. § 112(b) have been withdrawn.
Applicant's arguments filed 06/15/2026 are also persuasive with respect to the previous rejection of claim 9 under 35 U.S.C. § 101. Applicant's arguments, see page 6 of the Remarks, state that claim 9 has been amended to recite a "non-transitory computer-readable storage medium." The Examiner agrees that this amendment excludes transitory signal embodiments from the scope of claim 9. Therefore, the prior rejection of claim 9 under 35 U.S.C. § 101 has been withdrawn.
Applicant's arguments filed 06/15/2026 are also persuasive with respect to the previous rejections under 35 U.S.C. § 102(a)(1) and § 103. Applicant's arguments, see pages 7–9 of the Remarks, state that amended independent claim 1 now requires that each of the image restoration branch and the image super-resolution branch be divided into a plurality of cascaded, stage-wise indexed modules, that an i-th fusion module fuse the stage-specific output features of the i-th module of each branch together with the output of the i−1-th fusion module, and that the output of the i-th fusion module be fed back to drive the next-stage processing of the i+1-th module in both the image restoration branch and the image super-resolution branch. Applicant argues that Tao's MSFNet employs three separate single-pass branches (deblurring, denoising, and super-resolution) that are not divided into cascaded stages, and that Zhang's restoration branch and base feature-extraction branch each execute once to produce a single fixed feature, with recursion confined to the internal gate blocks and with the fused output flowing forward only to the reconstruction module, never back into either branch. The Examiner agrees that the cited portions of Tao and Zhang do not disclose or suggest stage-wise division of the restoration and super-resolution branches into cascaded, indexed modules, nor feedback of the fusion result into subsequent stages of both branches. Therefore, the previous rejection of claim 1 under 35 U.S.C. § 102(a)(1) as anticipated by Tao, and the previous rejections of claims 2–10 under 35 U.S.C. § 103 as unpatentable over Tao in view of Zhang, have been withdrawn.
However, a new ground of rejection under 35 U.S.C. § 103 is made in this Office Action applying Zhang in view of newly cited prior art Mao, Guo, and Ma to address the newly added stage-wise, feedback-coupled dual-branch architecture of amended claim 1. This modification to the rejection is directly necessitated by Applicant's amendment adding new limitations to independent claim 1. Applicant's amendments to claims 1, 3, and 9 have been fully considered. The newly added limitations of amended claim 1 have been considered and addressed in the updated rejections utilizing newly cited prior art. Because the necessity to apply new references to independent claim 1, and the withdrawal and replacement of the prior grounds of rejection, were directly necessitated by Applicant's substantive amendments adding new limitations, this action is properly made final in accordance with MPEP § 706.07(a).
Based on these facts, this action is made FINAL.
Claim Objections
Claim 1 is objected to because of the following informalities:
The phrase “low resolution degraded image” / “high-resolution clear image” should be made consistent in hyphenation, for example “low-resolution degraded image” and “high-resolution clear image”.
The phrase “pre trained processing model” should be corrected to “pre-trained processing model”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 3-6, and 9-10 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation “an index i satisfies 1≤i<2N+1”, but later recites “an i-th fusion module concatenates … output features of an i-1-th fusion module …”. That still creates ambiguity because i=1 is included in the stated range. If i=1, there is no i-1-th fusion module, i.e., no 0-th fusion module. Applicant separately recites “a first fusion module” but the i-th fusion-module clause still appears to include i=1. Therefore, the scope of the indexed fusion-module relationship is unclear. A cleaner claim would say something like 2≤i<2N+1 for the i-th fusion module, or otherwise expressly exclude the first fusion module from the i-th/ i−1-th clause.
Claim 1 further recites the limitation “an i−N−1-th restoration convolution module of the decoder” in claim. There is insufficient antecedent basis for this limitation in the claim. Claim 1 never introduces an "encoder" or "decoder" anywhere earlier in its own text. The encoder/ decoder structure is only introduced later in dependent claim 3 (“the image restoration branch is an encoding and decoding structure”).
Claim 3 recites that “a decoder consists of the N restoration convolution modules and the N−1 2× up-sampling modules”, but then states that those modules “are alternately connected in the encoder”. It is unclear whether the up-sampling modules are connected in the decoder or in the encoder. Therefore, the scope of the claimed encoding/ decoding structure remains unclear in claim itself.
Because claims 4-6 depend from claims 1 and 3, they inherit this ambiguity, fail to cure the deficiency.
Regarding claims 9-10, the rationale provided in the rejection of claim 1 is incorporated herein. Accordingly, the method for processing a low resolution degraded image of claim 1 corresponds to the non-transitory computer-readable storage medium of claim 9, as well as the computing device of claim 10, and performs the steps disclosed herein. Therefore, the claims are all rejected.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 3-4 and 9-10 are rejected rejected under 35 U.S.C. §103 as being unpatentable over Zhang in view of Mao (Mao et al, Image restoration using very deep convolutional Encoder-Decoder networks with symmetric skip connections. NIPS, 2016), further in view of Guo (Guo et al, Closed-loop Matters: Dual Regression Networks for Single Image Super-Resolution. arXiv.Org, 2020), further in view of Ma (Ma et al, Deep Face Super-Resolution with Iterative Collaboration between Attentive Recovery and Landmark Estimation. arXiv.Org, 2020).
Regarding claim 1, Zhang teaches a method for processing a low resolution degraded image ( [Abstract]: a method for degraded image super-resolution. ), comprising:
obtaining a low resolution degraded image to be processed;
inputting the low resolution degraded image into a pre-trained processing model to obtain a high-resolution clear image;
wherein, the processing model comprises an image restoration branch, an image super-resolution branch, and a plurality of fusion modules;
( [Sec. 3.1 “Network Architecture”], [Sec. 3.3 “Implementation Details”], [Fig. 2]: Zhang discloses a method for degraded image super-resolution, including processing a degraded low-resolution image using a proposed Gated Fusion Network (GFN) model to recover a sharp high-resolution image; and teaches that “given a degraded LR image Ldeg as the input”, the goal is to recover a sharp HR image. Zhang further teaches that the proposed model has a dual-branch architecture including a restoration module, a base/SR feature-extraction stream, a gate module, and a reconstruction module; the gate module includes recursive gate blocks that operate as a plurality of feature-fusion modules. Zhang also discloses training/pre-training the proposed network before evaluation/inference. )
the image restoration branch is used to restore the low resolution degraded image into a corresponding low-resolution clear image;
( [Sec. 3.1.1 “Restoration Module”], [Sec. 3.2 “Loss Functions”]: Zhang further teaches that the proposed network generates two output images, including a recovered LR image and a sharp HR image, and uses a recovering loss to guide the restoration branch to extract recovered features for the restoration task. )
the image super-resolution branch is used to generate a corresponding high-resolution clear image from the low resolution degraded image;
( [Sec. 3.1.2 “Base Feature Extraction Module”], [Sec. 3.1.4 “Reconstruction Module”], [Fig. 2]: Zhang teaches that, after the base feature extraction module extracts base/SR features from the degraded input Ldeg, then in the final stage, the fused features are fed into residual blocks and pixel-shuffling layers to enlarge spatial resolution and reconstruct the HR output image. )
the fusion module is used to fuse image features generated during a restoration task processed by the image restoration branch and corresponding image features generated during a super-resolution task processed by the image super-resolution branch to obtain fused features, and the fused features assist in generating the high-resolution clear image;
( [Sec. 3.1.3 “Gate Module”], [Sec. 3.1.4 “Reconstruction Module”], [Fig. 3-4]: Zhang teaches that recovered/restoration features and base/SR features complement each other and are adaptively merged by a gate module. Zhang further teaches a recursive merging strategy in which N gate blocks progressively fuse the recovered features into the main feature stream, and that the fused features are then fed into the image reconstruction module to generate the sharp HR output. )
wherein the image restoration branch has N+M+1 sequentially-indexed modules comprising N restoration convolution modules, a connection module, and another M restoration convolution modules, and the N restoration convolution modules, the connection module, and the M restoration convolution modules are sequentially connected, and an index i satisfies 1≤i<N+M+1;
( [Sec. 3.1 “Network Architecture”], [Sec. 3.1.1 "Restoration Module"], [Sec. 3.3 "Implementation Details"], [Fig. 2]: Zhang teaches that the restoration branch Gres adopts an asymmetric residual encoder-decoder architecture. Zhang discloses that the encoder consists of three scales, wherein each scale includes a residual group of six residual blocks (N=3 restoration convolution modules), with the first two residual groups followed by a strided convolutional down-sampling layer. Zhang further teaches that the output of the final encoder scale is passed to a decoder comprising two deconvolutional layers followed by two additional convolutional layers to reconstruct the restored low-resolution image (M=2 restoration convolution modules), wherein the point of transition between the encoder's final residual group and the decoder's first deconvolutional layer constitutes a connection module joining the encoder and decoder stages. Accordingly, Zhang discloses N+M+1 sequentially-indexed stages, with an index i satisfying 1≤i<N+M+1, corresponding to the encoder, connection, and decoder stages of Gres. )
the image super-resolution branch comprises 2via a fusion module, wherein the restoration convolution module is a basic operation module that uses convolution operations for restoration tasks, and the super-resolution convolution module is a basic operation module that uses convolution operations for super-resolution tasks;
( [Sec. 3.1 “Network Architecture”], [Sec. 3.1.2 "Base Feature Extraction Module"], [Sec. 3.1.3 "Gate Module"], [Sec. 3.1.4 "Reconstruction Module"], [Fig. 2]: Zhang teaches that the GFN model includes a base feature extraction module Gbase, a gate module Ggate, and a reconstruction module Grecon. The base feature extraction module extracts base features from the degraded LR input using eight residual blocks, and the reconstruction module receives fused features and reconstructs the final sharp HR image using residual blocks and pixel-shuffling layers. Zhang further teaches that the restoration module uses residual/ convolutional and deconvolutional operations for the restoration task, while the base feature extraction and reconstruction modules use convolutional/ residual and pixel-shuffling operations for the super-resolution task. Therefore, Zhang teaches SR-side processing modules connected through a gate/ fusion module, with restoration and SR modules implemented using convolutional operations for their respective tasks. )
a first fusion module concatenates output features of a first restoration convolution module, output features of a first super-resolution convolution module, and features of the low-resolution degraded image;
( [Sec. 3.1.3 "Gate Module"], [Fig. 3-4]: Zhang teaches that the features extracted by the restoration module Gresand base feature extraction module Gbase are fused by the gate module Ggate and then fed into the reconstruction module. Zhang further teaches that the first recursive gate block Ggate1 receives recovered/restoration features φRF, base/SR features φBF, and the degraded LR input Ldeg as inputs. These inputs correspond to output features of a restoration convolution module, output features of a super-resolution convolution module, and features of the low-resolution degraded image, respectively. )
an i-th fusion module concatenates output features of the restoration convolution module in the image restoration branch, output features of the the super-resolution convolution module in the image super-resolution branch, and output features of an i−1-th fusion module;
( [Sec. 3.1.3 "Gate Module"], [Fig. 3-4]: Zhang teaches that for i>1, the i-th recursive gate block computes φ^i_fusion = G^i_gate(φRF, φ^(i-1)_fusion) ⊗ φRF ⊕ φ^(i-1)_fusion, wherein φRF (the output feature of the restoration convolution module in the image restoration branch) and φBF (the output feature of the super-resolution convolution module in the image super-resolution branch, incorporated into the fusion stream via the first gate block) are concatenated together with φ^(i-1)_fusion, the output features of the i−1-th fusion module, at each successive gate block. )
concatenated features are processed by repeating a fusion operation a preset number of times, and a concatenation result is used as the input of an i+1-th fusion module,
( [Sec. 3.1.3 "Gate Module"], [Sec. 3.3 "Implementation Details"], [Fig. 3-4], [Table 4]: Zhang teaches that the gate module comprises N recursive gate blocks (Zhang reports N=3 as the best default configuration, i.e., a preset number of times), wherein each gate block sequentially passes the concatenated features through a 3×3 convolutional layer (first convolutional layer), a Leaky ReLU activation layer (activation layer), and a 1×1 convolutional layer (second convolutional layer) to generate a gating map. Zhang further teaches that the output φ^i_fusion of the i-th gate block is used as an input to the next, i+1-th gate block, consistent with the claimed repeated fusion operation and forward propagation of the concatenation result. )
wherein the the restoration convolution module; comprising an encoder, a a restoration convolution module of the decoder; and the the super-resolution convolution module.
( [Sec. 3.1 “Network Architecture”], [Sec. 3.1.1 “Restoration Module”], [Sec. 3.1.2 “Base Feature Extraction Module”], [Sec. 3.1.4 “Reconstruction Module”], [Fig. 2]: Zhang teaches that the restoration branch/ module Gres includes encoder-side residual/ convolutional processing, an intermediate encoder-decoder representation, decoder-side deconvolutional processing, and final convolutional reconstruction of a sharp LR image. The output features of the decoder φRF are fed into the gate module for feature fusion. Zhang further teaches that the SR-side processing includes the base feature extraction module Gbase, which extracts base/ SR features φBF from the degraded input, and the reconstruction module Grecon, which reconstructs the final HR image from fused features.)
While Zhang teaches an image restoration branch comprising a connection module positioned between an asymmetric N-module encoder and M-module decoder, Zhang does not explicitly disclose a symmetric restoration branch wherein the number of encoding-side restoration convolution modules directly equals the number of decoding-side restoration convolution modules (2N sequentially-indexed modules). Mao, however, teaches this feature:
wherein the image restoration branch has 2N sequentially-indexed modules comprising N restoration convolution modules, directly connected to another N restoration convolution modules, and the N restoration convolution modules, and the N restoration convolution modules are sequentially connected, and an index i satisfies 1≤i<2N;
( [Sec. 1 “Introduction”], [Sec. 3.1 "Architecture"], [Sec. 5.1 "Network parameters"], [Fig. 1]: Mao discloses a very deep, fully convolutional encoding-decoding framework ("RED-Net") that "contains a chain of convolutional layers and symmetric deconvolutional layers" [Sec. 3.1], wherein "the convolutional layers act as a feature extractor" and "the deconvolutional layers are then combined to recover the details of image contents" [Sec. 3.1]. Mao further discloses specific implemented embodiments (RED20, RED30) in which "RED20 contains 10 convolutional and deconvolutional layers" and "RED30 contains 15 convolutional and deconvolutional layers" [Sec. 5.1]; i.e., an equal number (N) of convolutional layers (N restoration convolution modules) directly and sequentially connected to an equal number (N) of deconvolutional layers (another N restoration convolution modules), for a total of 2N sequentially-indexed layers, with an index i satisfying 1≤i<2N as shown in Fig. 1. )
wherein the i-th module in the image restoration branch collectively refers to, for 1≤i≤N, an i-th restoration convolution module; for i=N+1, the connection module; and for N+1≤i≤2N, an i−N-th restoration convolution module of the decoder; and the i-th module in the image super-resolution branch refers to an i-th super-resolution convolution module.
( [Sec. 3.1 "Architecture"], [Sec. 3.2 "Deconvolution decoder"], [Sec. 4.1 "Analysis on the architecture"], [Fig. 1]: Mao teaches, per its architectural analysis and Equation (8), that "the output of the i-th layer" of the network is expressed as Fc(Xi−1) for i<L/2 and as XL−i + Fd(Xi−1) for i≥L/2, where L is the total number of layers [Sec. 4.1]. Mapping Mao's L/2 to the claimed N, this confirms that for 1≤i≤N, the i-th module corresponds to the i-th convolutional (encoding) layer [Sec. 3.1: "the convolutional layers successively down-sample the input image content"], and for N+1≤i≤2N (i.e., i≥L/2), the i-th module corresponds to the (i−N)-th deconvolutional (decoding) layer, which Mao describes as "up-sampl[ing] the abstraction back into its original resolution" and mirroring its corresponding convolutional layer via a symmetric skip connection [Sec. 3.2]. Mao's Fig. 1 depicts this direct, uninterrupted transition from the final convolutional layer to the first deconvolutional layer, without any separately-identified connection-module position intervening between the two-part encoder/decoder indexing scheme. )
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to implement Zhang’s restoration module using Mao’s very deep convolutional encoder-decoder restoration architecture because both Zhang and Mao are directed to CNN-based image restoration of degraded/corrupted images, and Zhang already uses an encoder-decoder-style restoration module to recover a clean LR image before feature fusion. Mao teaches that symmetric convolution/deconvolution encoder-decoder restoration networks with skip connections improve restoration quality by preserving image-detail information from encoding layers and passing it to corresponding decoding layers. Substituting or refining Zhang’s restoration branch with Mao’s known RED-Net encoder-decoder structure would have been a predictable use of a known image-restoration architecture to obtain improved recovered/restoration features for Zhang’s gated fusion and final super-resolution reconstruction.
While Zhang [as modified by Mao] teaches an image super-resolution branch comprising a base feature extraction module and a reconstruction module connected via a gate/ fusion module, Zhang [as modified by Mao] does not explicitly disclose a super-resolution branch comprising a fine-grained, sequentially-connected chain of 2N+1 super-resolution convolution modules matching the depth of the restoration branch's own module count. Guo, however, teaches:
the image super-resolution branch comprises 2N+1 super-resolution convolution modules sequentially connected, wherein the restoration convolution module is a basic operation module that uses convolution operations for restoration tasks, and the super-resolution convolution module is a basic operation module that uses convolution operations for super-resolution tasks;
( [Sec. 3.1 "Dual Regression Scheme for Paired Data"], [Sec. 4.1 "Architecture Design of DRN"], [Sec. B "Model Details of Dual Regression Network" (Supplementary)], [Fig. 3], [Table A]: Guo discloses that the primal network of DRN (the image super-resolution branch) is built upon a U-Net-style downsampling-upsampling design, wherein "both the downsampling...and upsampling...modules contain log2(s) basic blocks, where s denotes the scale factor" [Sec. 4.1], and each basic block is built using B residual channel attention blocks (RCAB) to improve model capacity [Sec. 4.1]. Guo's Table A discloses, for the 8× DRN model, a Head module, three sequentially-connected downsampling modules (Down 1, Down 2, Down 3), and three sequentially-connected upsampling modules (Up 1, Up 2, Up 3), each upsampling module comprising B RCABs; i.e., a Head module plus N=3 downsampling modules directly and sequentially connected to N=3 upsampling modules, for a total of 2N+1=7 sequentially-connected super-resolution convolution modules, as shown in Fig. 3 and Table A. Guo further teaches that the dual network (the restoration/dual regression branch) is "designed with only two convolution layers and a LeakyReLU activation layer" [Sec. 4.1] dedicated to reconstructing the downsampled LR image (a basic operation module using convolution operations for the restoration/ reconstruction task), while the primal network's Down/Up modules, built from convolutional and RCAB layers, are basic operation modules using convolution operations dedicated to the super-resolution task of reconstructing the HR image [Sec. 4.1, Table A]. )
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to further modify Zhang [as modified by Mao]’s super-resolution/ base feature stream with Guo’s U-Net-style primal super-resolution network because Zhang [as modified by Mao] and Guo are both directed to CNN-based single-image super-resolution from low-resolution inputs, and Zhang [as modified by Mao] already teaches using SR/ base features together with restored features to reconstruct a sharp high-resolution image. Guo teaches that arranging the SR network as a head module followed by paired downsampling and upsampling modules improves super-resolution performance by extracting hierarchical features and reconstructing HR details through multi-scale SR processing. Incorporating Guo’s known 2N+1 stage-indexed SR structure into Zhang [as modified by Mao]’s SR-side stream would have predictably provided stage-corresponding SR features for Zhang [as modified by Mao]’s gated fusion while improving final high-resolution reconstruction quality.
While Zhang [as modified by Mao and Guo] teaches a first fusion module and an i-th fusion module that repeatedly fuse the restoration branch's and super-resolution branch's static, once-computed output features together with the prior fusion module's output, Zhang [as modified by Mao and Guo] does not explicitly disclose feeding the fusion result back as an input to drive further processing at a subsequent iteration of both the image restoration branch and the image super-resolution branch. Ma, however, teaches this feature:
an i-th iteration fusion module concatenates output features of an i-th iteration module in the image restoration branch, output features of the i-th iteration module in the image super-resolution branch, and output features of an i−1-th iteration fusion module;
( [Sec. 3.1 "Deep Iterative Collaboration"], [Eq. (1)], [Fig. 2]: Ma discloses that the recurrent SR branch computes, at the n-th step, f^GR_n = GR(G1(ILR), f^GR_(n-1), L_(n-1)) [Eq. (1)], wherein GR is a recursive block comprising an attentive fusion module and a recurrent SR module. Ma teaches that the attentive fusion module, which corresponds to the claimed i-th iteration fusion module, concatenates/ fuses G1(ILR) features together with L_(n-1) (the landmark/ alignment output produced by the image restoration-analogous alignment branch at the (n-1)-th step) and f^GR_(n-1) (the recurrent SR branch's own feedback feature from the (n-1)-th step). Ma further discloses at Sec. 3.2, Eq. (8) that the attentive fusion module computes f_Fusion = Σ M_p · f_p, wherein the component-specific features f_p are generated under the guidance of attention maps M_p derived from the landmark output L_(n-1), and expressly states that "the attentive fusion module is a part of the recurrent SR branch, so that the gradients can be back-propagated to both the SR and alignment branches in a recursive manner" [Sec. 3.2], consistent with the claimed i-th iteration fusion module concatenating output features of the i-th iteration module in each of the two branches together with the output of the i-1-th iteration fusion module. )
concatenated features are processed by repeating a fusion operation a preset number of times, and a concatenation result is used as the input of an i+1-th iteration fusion module, the input of the i+1-th iteration module in the image restoration branch, and the input of the i+1-th iteration module in the image super-resolution branch, wherein each repetition of the fusion operation comprises sequentially passing the features through a first convolutional layer, an activation layer, and a second convolutional layer; and
( [Sec. 3.1 "Deep Iterative Collaboration"], [Eqs. (1)-(4)], [Sec. 4.2 "Implementation Details"], [Fig. 2], [Fig. 3]: Ma teaches that "for the n-th step where n = 1, ..., N, the SR branch recovers SR images I^SR_n by using the alignment results and the feedback information from the previous step n-1" [Sec. 3.1], and that this recurrent process is repeated for N steps (Ma's Implementation Details, Sec. 4.2, discloses "we...set...the number of steps to 4," i.e., a preset number of times). Ma further discloses that the fused output of step n, f^GR_n, is fed forward as the input to the SR branch's processing at the next step (n+1) via Eq. (1)'s recursive f^GR_(n-1) term, and is simultaneously used to generate I^SR_n [Eq. (2)] which is fed into the alignment branch A1 at the next step via Eq. (3)'s A1(I^SR_n) term; i.e., the concatenation/ fusion result at step n is used as an input to both the SR branch and the alignment (restoration-analogous) branch at step n+1, consistent with the claimed feedback into the i+1-th iteration module of each branch. Ma additionally discloses, per Sec. 3.2 and Fig. 3, that the attentive fusion module's feature processing includes "an input feature...expanded by a convolutional layer" followed by "a series of group convolutional layers" and a softmax/ attention-weighting operation (a first convolutional layer, an activation-type operation, and a second convolutional/ group-convolutional layer), consistent with the claimed sequential first-convolutional-layer/ activation-layer/ second-convolutional-layer structure of each fusion repetition. )
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to further modify Zhang [as modified by Mao and Guo]’s recursive gated-fusion framework, to use Ma’s iterative collaboration between branch modules because Zhang [as modified by Mao and Guo] already recognizes that restoration and super-resolution are interdependent and uses recursive gated fusion to progressively merge restoration features with SR features. Ma teaches that, in a super-resolution network, feeding information from a current recurrent step into later branch processing allows the recovery/ SR branch and the auxiliary branch to progressively improve each other over a preset number of iterations. Applying Ma’s known iterative feedback/ collaboration scheme to Zhang [as modified by Mao and Guo]’s gated restoration/ SR fusion would have predictably improved feature refinement by causing later restoration and SR modules to receive progressively enhanced fused information, thereby improving the final high-resolution reconstruction.
Regarding claim 3, Zhang [as modified by Mao, Guo and Ma] teaches the method for processing a low resolution degraded image according to claim 1, wherein the image restoration branch is an encoding and decoding structure, further comprising N−1 2x down-sampling modules and N−1 2x up-sampling modules;
an encoder consists of the N restoration convolution modules and the N−1 2x down-sampling modules, and the N restoration convolution modules and the N−1 2x down-sampling modules are alternately connected in the encoder;
( Zhang, [Sec. 3.1.1 “Restoration Module”], [Sec. 3.3 “Implementation Details”], [Appendix, Table 6]: Zhang discloses that the encoder of restoration branch Gres consists of three scales/residual groups corresponding to N=3 restoration convolution modules. Zhang teaches that the first two residual groups are followed by strided convolutional layers that down-sample the feature maps by 1/2, corresponding to N−1=2 down-sampling operations alternately interposed between the three residual-group modules, reducing the feature-map resolution from h×w to h/2×w/2 and then to h/4×w/4. )
a decoder consists of the N restoration convolution modules and the N−1 2x up-sampling modules, and the N restoration convolution modules and the N−1 2x up-sampling modules are alternately connected in the encoder;
( Zhang, [Sec. 3.1.1 “Restoration Module”], [Sec. 3.3 “Implementation Details”]; Mao, [Sec. 3.1 “Architecture”], [Sec. 5.1 “Network Parameters”], [Fig. 1]: Zhang teaches that the decoder of the restoration branch enlarges the feature maps from h/4×w/4 back to h×w using deconvolutional up-sampling stages. To the extent Zhang’s decoder includes two deconvolutional layers and additional convolutional reconstruction layers rather than a strictly symmetric N-module decoder matching the N-module encoder, Mao teaches such a symmetric decoder arrangement. Mao discloses a very deep fully convolutional encoding-decoding restoration network having a chain of convolutional layers and symmetric deconvolutional layers, in which the deconvolutional layers recover image details and mirror the convolutional encoding layers. Thus, Zhang in view of Mao teaches a decoder including N decoder-side restoration/deconvolution modules and N−1 2x up-sampling modules alternately connected in the decoder. )
and during a decoding process, each decoding layer is connected to the feature map of the corresponding encoding layer.
( Mao, [Sec. 3.1 “Architecture”], [Sec. 4.1 “Analysis on the architecture”], [Fig. 1]: Mao teaches symmetric skip connections between corresponding convolutional and deconvolutional layers. Mao teaches that image-detail information from convolutional encoding layers is directly propagated to corresponding mirrored deconvolutional decoding layers, and that the passed convolutional feature maps are combined with the deconvolutional feature maps. Accordingly, Mao teaches that, during decoding, each decoding layer is connected to the feature map of the corresponding encoding layer. It would have been obvious to incorporate Mao’s symmetric skip-connection scheme into Zhang’s restoration-branch encoder-decoder structure because Mao teaches that corresponding-layer skip connections help back-propagate gradients, make training deeper networks easier, and pass image details useful for recovering the original image, thereby predictably improving Zhang’s recovered low-resolution restoration features used in gated fusion and final high-resolution reconstruction. )
Regarding claim 4, Zhang [as modified by Mao, Guo and Ma] teaches the method for processing a low resolution degraded image according to claim 3, wherein the restoration convolution module comprises N residual convolution modules sequentially connected.
( [Sec. 3.1.1 “Restoration Module”], [Appendix, Table 6]: Zhang teaches that the restoration module includes residual groups in the encoder portion of the restoration branch. Each residual group includes six residual blocks, and Table 6 lists restoration-module residual blocks arranged as Resblock 1–6, Resblock 7–12, and Resblock 13–18. Accordling, each residual group/ restoration convolution module is composed of a sequence of N=6 residual blocks, corresponding to the claimed restoration convolution module comprising N residual convolution modules sequentially connected. )
Regarding claims 9-10, the rationale provided in the rejection of claim 1 is incorporated herein. In addition, Zhang [as modified by Mao, Guo and Ma] teaches the processing of a low resolution degraded image is implemented within a computer system. ( Zhang, [Sec. 3.3 “Implementation Details”]; Guo, [Sec. 5 "Experiments", 5.1]; Ma, [Sec. 4 "Experiments"]). Accordingly, the method for processing a low resolution degraded image of claim 1 corresponds to the non-transitory computer-readable storage medium of claim 9, as well as the computing device of claim 10, and performs the steps disclosed herein. Therefore, the claims are all rejected.
Claims 5-6 are rejected rejected under 35 U.S.C. §103 as being unpatentable over Zhang [as modified by Mao, Guo and Ma] in view of Dong (Dong et al, “Accelerating the Super-Resolution Convolutional Neural Network.” ArXiv.org, 2016).
Regarding claim 5, Zhang [as modified by Mao, Guo and Ma] teaches the method for processing a low resolution degraded image according to claim 1,
Zhang [as modified by Mao, Guo and Ma] teaches the claimed degraded-image restoration/super-resolution framework, stage-indexed restoration and super-resolution branches, and iterative fusion of branch features, but does not expressly characterize the super-resolution branch using the conventional classical SRCNN/FSRCNN organization of feature extraction, nonlinear mapping, and reconstruction. Dong teaches this classical super-resolution structure:
wherein the image super-resolution branch is a classical super-resolution structure, comprising a feature extraction module, a nonlinear mapping learning module, and a reconstruction module that are sequentially connected;
the feature extraction module comprises a convolutional layer; 2N+1 super-resolution convolution modules sequentially connected form the nonlinear mapping learning module; and the reconstruction module comprises N−1 2x up-sampling modules sequentially connected.
( Dong, [Sec. 2 “Related Work”], [Sec. 3 “Fast Super-Resolution Convolutional Neural Networks”], [Fig. 1]: Dong teaches a classical CNN-based super-resolution structure based on SRCNN/FSRCNN. Dong explains that SRCNN performs super-resolution using sequential stages including feature extraction, nonlinear mapping, and reconstruction. Dong further teaches FSRCNN as a fast super-resolution convolutional neural network including a feature extraction part, a shrinking part, multiple mapping layers, an expanding part, and a deconvolution/reconstruction part. The feature extraction part includes a convolutional layer that extracts features from the low-resolution input image; the mapping portion includes a sequence of convolutional mapping layers corresponding to the claimed nonlinear mapping learning module formed by sequential super-resolution convolution modules; and the deconvolution/reconstruction part upsamples the feature maps and reconstructs the high-resolution image. Thus, Dong teaches a classical super-resolution branch comprising sequentially connected feature extraction, nonlinear mapping, and reconstruction modules, with convolutional feature extraction, sequential super-resolution convolution/mapping modules, and reconstruction/upsampling for generating the high-resolution image.
Zhang [as modified by Mao, Guo and Ma] can use Dong’s classical SRCNN/FSRCNN organization because Dong teaches that feature extraction, nonlinear mapping, and reconstruction are the conventional functional stages of CNN-based super-resolution, and Zhang [as modified by Mao, Guo and Ma] already use convolutional SR feature extraction and HR reconstruction. Applying Dong’s known organization would have predictably structured the SR branch into standard SR processing stages while preserving Zhang’s gated restoration/SR fusion and Guo’s stage-indexed SR processing. )
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to implement the SR-side stream of Zhang [as modified by Mao, Guo and Ma] according to Dong’s classical SRCNN/FSRCNN organization because Zhang [as modified by Mao, Guo and Ma] already performs CNN-based super-resolution using convolutional SR feature extraction, feature processing, and HR reconstruction, and Dong teaches that feature extraction, nonlinear mapping, and reconstruction are conventional functional stages for CNN-based super-resolution. Applying Dong’s known SR organization would have predictably structured the SR branch into standard feature-extraction, mapping, and reconstruction stages while preserving Zhang’s gated restoration/SR fusion and Guo’s stage-indexed SR processing.
Regarding claim 6, Zhang [as modified by Mao, Guo, Ma and Dong] teaches the method for processing a low resolution degraded image according to claim 5, wherein the super-resolution convolution module comprises N residual convolution modules sequentially connected.
( Zhang, [Sec. 3.1.2 “Base Feature Extraction Module”], [Sec. 3.1.4 “Reconstruction Module”], [Fig. 2]; Dong, [Sec. 3.2 “Shrinking, Mapping, and Expanding”]: Dong teaches that the nonlinear mapping learning module of the classical super-resolution structure, as mapped with respect to claim 5, is formed by sequentially connected super-resolution convolution/mapping layers. Zhang further teaches that its SR-side base feature extraction module Gbase is built using eight residual blocks, and that its reconstruction module Grecon also uses residual-block-based processing before pixel-shuffling reconstruction. Zhang’s residual blocks are residual convolution modules because they perform convolutional feature processing with residual/skip connections. Thus, Zhang teaches that SR-side convolutional processing modules may comprise a plurality of residual convolution modules sequentially connected. It would have been obvious to implement Dong’s sequential nonlinear mapping / SR convolution modules using Zhang’s residual-block SR feature-processing design because Zhang and Dong are both directed to CNN-based super-resolution, and Zhang teaches that residual blocks are suitable SR-side feature-processing units. Replacing or refining Dong’s plain convolutional mapping layers with Zhang’s known residual convolution modules would have predictably improved feature learning and gradient flow in the nonlinear mapping portion while preserving Dong’s classical feature-extraction, nonlinear-mapping, and reconstruction organization. )
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KEN KUDO whose telephone number is (571)272-4498. The examiner can normally be reached M-F 8am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at 571-272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
KEN KUDO
Examiner
Art Unit 2671
/KEN KUDO/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671
Prosecution Insights