DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filling date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 5-6 and 9-10 are rejected under 35 U.S.C. 103 as being unpatentable by Karam et al. (U.S. 2019/0303720 A1) in view of Van Aert et al. (U.S. 2025/0078216 A1).
Regarding Claim 1, Karam discloses a method of training an image restoration model, the method being performed by an image restoration apparatus (Karam, [0002] “systems and methods for feature corrections and regeneration for robust sensing” and [0004] “the prediction performance of a DNN trained on high quality images devoid of distortions” Karam teaches a method of training (a DNN trained ) an image (feature) restoration correction and regeneration model performed by systems to get high quality images (devoid of distortions), the method comprising:
pre-training an image restoration model by generating training images by randomly applying a plurality of synthetic degradation functions to a clean image (Karam, [0002] “methods for feature corrections and regeneration” and [0005] “Testing distorted images with a pre-trained DNN model” and [0007] “FIG. 2. a pre-trained model are discriminative enough to generate a concise clustering of high-quality images from the same class” [0043] “It is important for a DNN to perform well on both clean and distortion-affected images and [0046] “Computing the output predictions by swapping a distortion affected filter output with its corresponding clean output for each of the ranked filters and its contribution to the associated performance degradation” Karam teaches pre-training an image corrections and regeneration model by generating training images by randomly applying a plurality of performance degradation functions to a clean image; and
fine-tuning parameters of the pre-trained image restoration fine-tuned model through contribution-based low-rank adaptation for an image restoration task (Karam, [0059] “The CIFAR-100 pre-trained model was fine-tuned as shown in FIG. 3A on a mix of distortion affected images and clean images” and [0060] “For Gaussian blur affected images, fine-tuning parameters in all layers (Finetune-9) achieves the highest accuracy among all fine-tuned models” Karan teaches fine-tuning parameters in all layers (Finetune-9) of the pre-trained model to clean images;
wherein fine-tuning the parameters comprises fine-tuning parameters of each layer of the image restoration model based on a ratio of learnable network parameters (Karam, [0059] “The CIFAR-100 pre-trained model was fine-tuned as shown in FIG. 3A. Starting with an initial learning rate that is ten times lower than the initial learning rate used to generate the pre-trained model, i.e., 0.01, and momentum equal to 0.9, network parameters in various layers are updated” and [0060] “For Gaussian blur affected images, fine-tuning parameters in all layers (Finetune-9) achieves the highest accuracy among all fine-tuned models, whereas Finetune-6, Finetune-3 and Finetune-1 can achieve ≈98%, 87% and 60% of this accuracy” Karan teaches fine-tuning the parameters in each layer to achieve the high quality images based on a leaning rate (ratio) used to generate network parameters.
Karam teaches using the restoration accuracy formular to clean images (Karam, [0085] Restoration Accuracy=acc(lp+c)/acc(lc), where lc is a set containing the clean images… and acc(.) is the top-1 accuracy.
However, Karam does not explicitly teach an image restoration model;
image restoration model through contribution-based low-rank adaptation for an image restoration task;
determined according to a contribution of each layer of the pre-trained image restoration model, and low-rank adaptation for the image restoration task.
Van Aert teaches an image restoration model (Van Aert, [0007] “the art for image restoration algorithms to process images without (or with minimal) imposing constraints on the operational parameters, e.g. for high-quality restoration algorithms that can be easily applied to (e.g.) single-shot EM images” Van Aert teaches an image restoration model (algorithm) for providing high-quality images.
image restoration model through contribution-based low-rank adaptation for an image restoration task (Van Aert, [0004] “The signal-to-noise ratio (SNR) of the recorded image can be reduced by these distortions” and [0011] “an acquisition strategy for neural network based image restoration of images with a low SNR” [0015] “construct an artificial neural network that is adapted to provide such image restoration” Van Aert teaches image restoration model through contribution-based low-rank adaptation for an image restoration task (neural network is adapted to contribute image restoration of images with a low SNR);
determined according to a contribution of each layer of the pre-trained image restoration model, and low-rank adaptation for the image restoration task (Van Aert, [0011] “an acquisition strategy for neural network based image restoration of images with a low SNR” and [0015] “construct an artificial neural network that is adapted to provide such image restoration” and [0191] “The discriminator network is adapted (i.e. trained) to evaluate the quality of the output data generated by the generator network” and [0187] “The generator receives, as input, a distorted electron microscopy image 10 (e.g. in training, the input image of a training pair, and in use after training, the image to restore). The input is provided to a down-convolutional layer 11, which corresponds to an up-convolutional layer 12 to determine the correction to apply (e.g. additively 13) to obtain the output image, i.e. the restored electron microscopy image 14 (e.g. in training, the output image of the training pair, and, in use, the desired output with reduced distortions)” Van Aert teaches determined a contribution of each layer (construction of AN network each layer (convolutional layers) of the pre-trained image restoration model, and low-rank (a low SNR) adaptation for the image restoration task.
Karam and Van Aert are combinable because they are from the same field of endeavor, system and method for image processing and try to solve similar problems. It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was made for modifying the method of Karam to combine with an image restoration model (as taught by Van Aert ) in order to apply an image restoration model to provide high quality image because Van Aert can provide an image restoration model (algorithm) for providing high-quality images (Van Aert, [0007]). Doing so, it may provide image restoration algorithm would have a relatively low computational cost, to allow fast execution and to reduce the costs associated with memory, processing power and other hardware requirements (Van Aert , [0014]).
Regarding Claim 2, a combination of Karam and Van Aert discloses the method of claim 1, wherein pre-training the image restoration model comprises randomly selecting the plurality of synthetic degradation functions from among a plurality of different synthetic degradation functions, and sequentially applying the plurality of synthetic degradation functions to the clean image one by one in a randomly determined order (Karam, [0043] DeepCorrect Framework. It is important for a DNN to perform well on both clean and distortion-affected images” and [0045] “to the performance degradation…By focusing on restoring the activations of only select filters which are most susceptible to input distortions” and [0016] “FIG. 7, a correction unit based on a residual function acting on outputs of βiNi filters out of Ni total filters in the ith convolutional layer of a pre-trained DNN” and [0072] “whereas decreasing Di makes the correction unit thinner. A natural choice for Di would be to make it equal to the number of distortion susceptible filters that need correction (βiNi), in a DNN layer i; this is also the default parameter setting used in the correction units for Deepcorr-5” and [0091] “FIG. 17 recognizing with a reasonable accuracy objects in low-resolution blurred images or in low SNR noisy images, DNNs that are trained on pristine images predict incorrect class labels even at a relatively low-level of perceivable degradation” Karam teaches randomly selecting a synthetic degradation function (e.g., a residual function of convolutional filter of a pre-trained DNN) performs decreasing parameter Di makes the correction unit thinner (degradation) to clean and distortion-affected images. Fig. 17 shows a result of sequentially applying the plurality of synthetic degradation functions to the clean image one by one in a randomly determined order.
Regarding Claim 5, a combination of Karam and Van Aert discloses an image restoration apparatus (Karam [0002] “systems and methods for feature corrections and regeneration for robust sensing” and [0004] “the prediction performance of a DNN trained on high quality images devoid of distortions” Karam teaches an image corrections and regeneration system, comprising:
memory (Karam, [0108] “a main memory 104. Mass storage device 107 can be used to store information and instructions”) configured to store an image restoration model and a program required for training the image restoration model; and
a controller including at least one processor (Karam, [0107] “one processor 102”), and configured to train the image restoration model;
wherein the controller pre-trains the image restoration model by generating training images by randomly applying a plurality of synthetic degradation functions to a clean image, and
fine-tunes parameters of the pre-trained image restoration model for an image restoration task, in which case parameters of each layer of the image restoration model are fine-tuned based on low-rank adaptation according to a contribution for each layer of the pre-trained image restoration model for the image restoration task.
Claim 5 is substantially similar to claim 1 and is rejected based on similar analyses.
Regarding Claim 6, a combination of Karam and Van Aert discloses the image restoration apparatus of claim 5, wherein the controller randomly selects the plurality of synthetic degradation functions from among a plurality of different synthetic degradation functions, and sequentially applies the plurality of synthetic degradation functions to the clean image one by one in a randomly determined order.
Claim 6 is substantially similar to claim 2 and is rejected based on similar analyses.
Regarding Claim 9, a combination of Karam and Van Aert discloses a computer program that is executed by an image restoration apparatus and stored in a non-transitory computer-readable storage medium to perform the method set forth in claim 1 (Karam, [0110] “a computer program product, which may include a machine-readable medium having stored thereon instructions to perform a process” and [0111] “processor(s) 102 accesses main memory 104 via the use of bus 101 in order to launch, run, execute, interpret or otherwise perform processes, such as through logic instructions” Karam teaches a computer program is executed by a processor accesses main memory).
Regarding Claim 10, a combination of Karam and Van Aert discloses a non-transitory computer-readable storage medium having stored thereon a program that, when executed by a processor, causes the processor to execute the method set forth in claim 1 (Karam, [0110] “a computer program product, which may include a machine-readable medium having stored thereon instructions” and [0111] “processor(s) 102 accesses main memory 104 via the use of bus 101 in order to launch, run, execute, interpret or otherwise perform processes, such as through logic instructions” Karam teaches a machine-readable medium stores the program instructions and is executed by a processor accesses main memory).
Claims 3, 7 are rejected under 35 U.S.C. 103 as being unpatentable by Karam et al. (U.S. 2019/0303720 A1) in view of Van Aert et al. (U.S. 2025/0078216 A1) and further in view of Ding et al. (U.S. 2023/0306239 A1).
Regarding Claim 3, a combination of Karam and Van Aert discloses the method of claim 1, wherein fine-tuning the parameters comprises also fine-tuning parameters of bias layers and normalization layers included in the pre-trained image restoration model (Van Aert, [0121] “The numerical parameter ranges that are applied for the data generation may be fine-tuned based on analyzing a large number of high quality simulations of (S)TEM images for microscope settings” and Fig.3, [0187] “The generator receives, as input, a distorted electron microscopy image 10 (e.g. in training, the input image of a training pair…which corresponds to an up-convolutional layer 12 to determine the correction to apply…the desired output with reduced distortions” [0192] “the zero-padding layers were removed and the batch normalization layers were replaced by instance normalization layers IN. This results in a serial layer group (or stack) structure, in which in each layer group a convolutional layer, an instance normalization layer and a leaky rectified linear unit are applied (except for the first and last layers, which are structured as shown in FIG. 6” Van Aert teaches the fine-tuning parameters include instance normalization layer IN (Fig. 6) in the pre-trained image restoration model (reduced distortions).
Karam and Van Aert are combinable see rationale in claim 1.
However, a combination of Karam and Van Aert does not explicitly teach fine-tuning parameters of bias layers included in the pre-trained image restoration.
Ding teaches fine-tuning parameters of bias layers included in the pre-trained image restoration (Ding, [0098] “a specific pretrained neural network, during the online training process, the decoding portion is fixed, and modules that only in the encoding portion can be tuned based on one or more input images to optimize a rate-distortion performance” and [0077] “FIG. 2, the main encoder network (111) includes four convolution layers where each convolution layer has a convolution kernel of 5×5 and 192 channels. The parameters used in the main encoder network (111) include the 19200 weights and optional biases. Additional parameter(s) can be included when biases and/or additional NN(s) are used in the main encoder network (111)” Ding teach during the training process, perform fine-tuning encoding portion, bias parameters are added to layers (referred to as bias layers) in the main encoder network (Fig. 2) to optimize (reduce) a rate distortion of input images (image restoration).
Karam and Van Aert and Ding are combinable because they are from the same field of endeavor, system and method for image processing and try to solve similar problems. It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was made for modifying the method of Karam to combine with fine-tuning parameters of bias layers (as taught by Ding) in order to include fine-tuning parameters of bias layers in the pre-trained image restoration because Ding can provide during the training process, perform fine-tuning encoding portion, bias parameters are added to layers (referred to as bias layers) in the main encoder network (Fig. 2) to optimize (reduce) a rate distortion of input images (image restoration) (Ding, Fig. 2, [0077], [0098]). Doing so, it may provide the online training is performed with parameters of a layer in a main encoder network or a hyper encoder network of the NIC framework being tunable (Ding, [0010]).
Regarding Claim 7, a combination of Karam and Van Aert discloses the image restoration apparatus of claim 5, wherein the controller also fine-tunes parameters of bias layers and normalization layers included in the pre-trained image restoration model.
Claim 7 is substantially similar to claim 3 and is rejected based on similar analyses.
Allowable Subject Matter
Dependent claims 4, 8 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding to independent claims 1, 5, the closest prior art references the examiner found are Karam et al. (U.S. 2019/0303720 A1) in view of Van Aert et al. (U.S. 2025/0078216 A1) have been made of record as teaching: pre-training an image restoration model by generating training images by randomly applying a plurality of synthetic degradation functions to a clean image (Karam, [0002], Fig. [0007]); wherein fine-tuning the parameters comprises fine-tuning parameters of each layer of the image restoration model based on a ratio of learnable network parameters (Karam, [0059], [0060]); an image restoration model (Van Aert, [0007]); image restoration model through contribution-based low-rank adaptation for an image restoration task (Van Aert, [0004], [0011]); determined according to a contribution of each layer of the pre-trained image restoration model, and low-rank adaptation for the image restoration task (Van Aert, [0011], [0015]), recited in claims 1, 5.
However, the art of record did not teach or suggest the claim taken as a whole and particular the limitation pertaining
wherein fine-tuning the parameters comprises computing a FAIG score for each layer during re-training of the pre-trained image restoration model for the image restoration task, in order to determine the contribution of each layer recited in dependent claims 4, 8.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance”.
Conclusion
The prior arts made of record and not relied upon are considered pertinent to applicant's disclosure Li et al. (U.S. 2025/0225700A1) and Chen et al. (U.S. 2025/0252301 A1).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KHOA VU whose telephone number is (571)272-5994. The examiner can normally be reached 8:00- 4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at 571-272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
/KHOA VU/Examiner, Art Unit 2611
Prosecution Insights