Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 05/13/2026 has been entered.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 3-8, 10-15 and 17-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-5, 7, 8, 10-12, 14-15, and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Arefin et. al (“Multi-Image Super-Resolution for Remote Sensing using Deep Recurrent Networks”) in view of Varol et. al (“A Constrained Latent Variable Model”), further in view of Bora et. al (“Compressed Sensing using Generative Models”).
Regarding Claim 1, Arefin teaches a computer-implemented method comprising:
Abstract: “In this work, we present a data-driven, multi-image super resolution approach to alleviate these problems. Our approach is based on an end-to-end deep neural network that consists of an encoder, a fusion module, and a decoder.”
selecting, by one or more computer processors, a latent vector associated with a high-resolution image from a plurality of latent vectors of a generative neural network model, wherein the generative neural network model is trained using a set of rules based on physical knowledge and wherein the set of rules includes a rule that is selected from the group consisting of: a multiple timestamp consistency, and an overlapping consistency;
Abstract: “In this work, we present a data-driven, multi-image super resolution approach to alleviate these problems. Our approach is based on an end-to-end deep neural network that consists of an encoder, a fusion module, and a decoder.”
Introduction, pg. 1 and 2: “On the other hand, low-resolution data are plentiful and sometimes even publicly available at no cost. However, these may involve different acquisition sources, locations, or times, and may thus require special care in the way they are combined for super-resolution purposes… satellite imaging systems typically orbit the earth with pre-defined speeds and paths. Pixel-level inconsistencies can however still occur even with a well-calibrated system, and sub-pixel registration is often a necessary processing step for applications using several images at once… A simple solution to this problem is to increase the amount of input information by instead using multiple low-resolution images at once. This technique is called multi-image super-resolution (MISR). The challenge for MISR approaches then becomes information fusion (or registration) due to the noisy nature of the imaging process. In general, MISR is capable of more accurate high-resolution reconstructions than SISR as it aggregates more information extracted from multiple views of the target region.”
3.1 Problem Formulation, pg. 3: “An overview of our model is shown in Figure 1. It can be split into three modules (left to right): 1) Encoder, which encodes relevant features from the low-resolution images into latent representations…”
3.2 Encoder, pg. 4: “Given the input tensors (li)Ki=1, and q, the network is trained to produce feature representations, denoted by (ri)Ki=1.”
Fig. 3 caption: “Example of a candidate region for super-resolution: a) overlapping low-resolution input images, b) reconstructed super resolution image by MISR-GRU, c) target high-resolution image.”
Explanation: The reference discloses latent representations (feature vectors) with a plurality of latent vectors (r1, r2, rK), where selection occurs during processing of each latent representation. The reference also discloses training and model-based reconstruction, where the rules based on physical knowledge are constraints from sensor acquisition, temporal variation, and spatial consistency. The multiple timestamp consistency corresponds to the images acquired at different times, and the overlapping consistency corresponds to the overlapping views/regions.
generating, by one or more computer processors, a super resolution image from the selected latent vector;
Abstract: “Finally, a decoder reconstructs the super-resolved image.”
3.1 Problem Formulation, pg. 3: “Decoder, which reconstructs the target high-resolution image.”
downscaling, by one or more computer processors, the super resolution image to match a size of a plurality of low-resolution images, wherein the plurality of low-resolution images are acquired from the group consisting of: at different times, and by different sensors over at least partially overlapping areas;
Introduction, pg. 1: “The processes behind the acquisition pipelines that determine the quality of the imagery rely heavily on the quality of the sensors themselves, whether electro-optical, radar, or laser-based. We approach the super-resolution problem from the image reconstruction perspective which aims at generating a high-resolution image based on one or more low-resolution images…On the other hand, low-resolution data are plentiful and sometimes even publicly available at no cost. However, these may involve different acquisition sources, locations, or times, and may thus require special care in the way they are combined for super-resolution purposes. In general, MISR is capable of more accurate high-resolution reconstructions than SISR as it aggregates more information extracted from multiple views of the target region.”
Related Work, pg. 3: “As detailed earlier, MISR approaches aim to reconstruct hidden high-resolution details using multiple low-resolution observations of the same scene…Many modern optimization-based approaches to MISR build a generative model that, given a high-resolution image, simulates the acquisition of low-resolution images.”
Fig. 3 caption: “Example of a candidate region for super-resolution: a) overlapping low-resolution input images, b) reconstructed super resolution image by MISR-GRU, c) target high-resolution image.”
Explanation: The reverse relationship is explicitly disclosed, and the simulation corresponds to downscaling HR [Wingdings font/0xE0] LR. They also disclose that the plurality of the LR images are acquired at different times and different sensors over overlapping areas.
computing, by one or more computer processors, a difference between the down-scaled super resolution image and each of the plurality of low-resolution images;
Related Work, pg. 3: “An initial guess for the high-resolution image is then improved by minimization of the error between simulated and ground-truth low-resolution images.”Explanation: Error = difference between downscaled HR and LR images.
determining, by one or more computer processors, a multi-image minimum difference of the difference between the down-scaled super resolution image and each of the plurality of low-resolution images;
Related Work, pg. 3: “An initial guess for the high-resolution image is then improved by minimization of the error between simulated and ground-truth low-resolution images.”3.6 Loss Function, pg. 5: “The typical way to formulate the super-resolution training objective is to minimize the reconstruction error between the target high-resolution image and the model’s prediction.”
Explanation: Optimization across multiple images = multi-image minimum difference.
determining, by one or more computer processors, whether the multi-image minimum difference meets a pre-defined stopping criterion, wherein the pre-defined stopping criteria is selected from the group consisting of: a threshold on the multi-image minimum difference, a satisfaction of the multiple timestamp consistency, and a satisfaction of the overlapping consistency;
3.6 Loss Function, pg. 5: “The typical way to formulate the super-resolution training objective is to minimize the reconstruction error between the target high-resolution image and the model’s pre diction. In this spirit, the Mean Squared Error (MSE) is commonly used in practice due to its interpretability and effectiveness.”
4.2 Experimental Setup: “The model was optimized end-to-end using Adam [24] starting with an initial learning rate of 0.0007 and gradual learning rate decay with a factor of 0.97 whenever the validation score plateaued for more than 2 epochs.”
Explanation: Plateau/convergence = stopping criterion. Threshold/metric = MSE. Temporal and spatial consistency shown above.
responsive to the pre-defined stopping criteria being met, transmitting, by one or more computer processors, the super resolution image to a user;
3.1 Problem Formulation, pg. 3: “The predicted output of our model is denoted by H…”
Explanation: Final output corresponds to transmission/output.
and storing, by one or more computer processors, the super resolution image (Fig. 3 (shown below)).
PNG
media_image1.png
363
729
media_image1.png
Greyscale
Explanation: Generated outputs are stored/displayed for evaluation.
Arefin fails to teach that selecting the latent vector includes not using latent vectors that violate the set of rules based on physical knowledge.
However, Varol teaches constraining latent-variable representations based on known physical constraints. Specifically, Varol teaches that “nothing prevents commonly-employed latent variable models from generating configurations that violate known constraints” (Introduction, pg. 1) and further teaches “a constrained latent variable model whose generated output inherently accounts for such knowledge” (Abstract) where “the generated outputs comply with equality and in equality constraints expressed in terms of the problem variables” (Introduction, pg. 1). Varol additionally teaches that existing latent variable models may generate outputs that “violate known constraints” (Introduction, pg. 1), and that it is preferable to exploit such constraints during learning, thereby yielding “a latent variable model that only generates physically-plausible deformations” (Related Work, pg. 2). Accordingly, Varol teaches preventing the use of latent-variable configurations that would result in outputs violating known physical constraints and restricting latent-variable solutions to those satisfying the physical constraints.
Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Arefin with the constrained latent-variable techniques of Varol. Arefin recognizes that multi-image super-resolution must address “pixel-level inconsistencies” (3. Model, pg. 3), “geometric distortions, blur, and pixel noise” (2. Related Work, pg. 3), and further teaches that reconstruction quality can be improved through improved handling of such inconsistencies (2. Related Work, pg. 3). One of ordinary skill in the art would have been motivated to incorporate Varol’s constraint-based latent-vector selection into Arefin’s latent-representation framework in order to prevent physically inconsistent latent representations from being used during image reconstruction, thereby improving reconstruction fidelity and reducing errors caused by inconsistent observations. Such modification merely applies Varol’s known latent-space constraint technique to Arefin’s latent-representation super-resolution system for its known purpose of preventing latent configurations that violate physical constraints and would have yielded the predictable result of producing physically consistent reconstructed outputs.
Arefin and Varol fail to teach the limitation: “responsive to the pre-defined stopping criteria not being met, the method selects a different latent vector and repeats the generating, the downscaling, the computing, and the determining the multi-image minimum difference until the pre-defined stopping criteria is determined to be met.”
However, Bora explicitly teaches iterative latent vector selection and repetition until error minimization. Bora states that “our approach is to find a vector in representation space such that the corresponding vector in the sample space matches the observed measurements” (2 Our Algorithm, pg. 3), which directly corresponds to selecting a latent vector. Bora further teaches generating G(z) and computing a measurement difference, stating that “we thus define the objective to be loss(z) = || AG(z)−y||2” (2 Our Algorithm, pg. 3). Bora also discloses repeatedly updating (i.e., selecting a different latent vector z), repeating generation and computation, and continuing until convergence (i.e., stopping criteria satisfied), stating that “by using any optimization procedure, we can minimize loss(z) with respect to z…in particular, if the generative model G is differentiable, we can evaluate the gradients of the loss with respect to z using backpropagation and use standard gradient based optimizers” (2 Our Algorithm, pg. 3). Lastly, Bora states “if the optimization procedure terminates at ˆz, our reconstruction for x∗ is G(ˆz)” (2 Our Algorithm, pg. 3), which directly corresponds to stopping criteria and final latent vector selection.
Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate iterative latent vector optimization into Arefin and Varol’s multi-image super resolution method. Arefin and Varol already perform reconstruction using latent representations, and Bora provides a known method to improve reconstruction accuracy via iterative latent vector optimization. Thus, there existed a recognized problem (minimizing reconstruction error) and a finite number of predictable solutions (optimization techniques such as gradient descent). Thus, it would have been obvious to apply Bora’s iterative latent vector optimization within Arefin and Varol’s reconstruction framework to improve reconstruction accuracy, with a reasonable expectation of success. Additionally, a person of ordinary skill in the art would have been motivated to apply Bora’s iterative optimization technique to Arefin and Varol’s system because both operate in the same field (image reconstruction using learned representations), and such application would predictably improve reconstruction accuracy.
Regarding Claim 3, Arefin in view of Varol and Bora teaches the computer-implemented method of claim 1, and Arefin further teaches that it comprises:
retrieving, by one or more computer processors, the plurality of low-resolution images;
Introduction, pg. 2: “A simple solution to this problem is to increase the amount of input information by instead using multiple low-resolution images at once. This technique is called multi-image super-resolution (MISR). The challenge for MISR approaches then becomes information fusion (or registration) due to the noisy nature of the imaging process. In general, MISR is capable of more accurate high-resolution reconstructions than SISR as it aggregates more information extracted from multiple views of the target region.”
Related Work, pg. 3: “As detailed earlier, MISR approaches aim to reconstruct hidden high-resolution details using multiple low-resolution observations of the same scene.”
and determining, by one or more computer processors, the size of each of the plurality of low-resolution images.
3.1 Problem Formulation, pg. 3: “We define the ith LR image of a scene as li ∈ Rc×h×w. Here, c, h, and w are the (channel-wise) depth, height, and width of the input LR image, respectively.”
Regarding Claim 4, Arefin in view of Varol and Bora teaches the computer-implemented method of claim 1 and Arefin further teaches that the size of an image is selected from the group consisting of: a number of pixels, a spatial resolution, and a physical size.
Results, pg. 6: “The low-resolution images were prepared with a shape of 128×128 pixels…”
4.2 Experimental Setup, pg. 6: “Because of the memory constraint and to improve generalization by data augmentation, we trained our model with randomly cropped 64 × 64 LR and corresponding 192 × 192 HR patches. As our model is fully convolutional, at test time we feed full LR images of spatial size 128 × 128 as input.”
Regarding Claim 5, Arefin in view of Varol and Bora teaches the computer-implemented method of claim 1 and Arefin further teaches that the difference between the down-scaled super resolution image and each of the plurality of low resolution images is selected from the group consisting of: an average difference between the down-scaled super resolution image and the plurality of low resolution images meets the pre-defined stopping criteria, and a percentage of difference values between the down-scaled super resolution image and the plurality of low resolution images meets the pre- defined stopping criteria.
Related Work, pg. 3: “An initial guess for the high-resolution image is then improved by minimization of the error between simulated and ground-truth low-resolution images.”
3.6 Loss Function, pg. 5: “The typical way to formulate the super-resolution training objective is to minimize the reconstruction error between the target high-resolution image and the model’s pre diction. In this spirit, the Mean Squared Error (MSE) is commonly used in practice due to its interpretability and effectiveness. In concordance with the evaluation guidelines of the challenge dataset (detailed in the next section), we opt to use a corrected metric for our loss function. We settle on a variant of the MSE called the corrected MSE (cMSE) which equalizes the brightness in both predicted and target images.”
Explanation: MSE = average difference. Loss thresholding corresponds to percentage/aggregate difference criteria.
Regarding Claim 7, Arefin in view of Varol and Bora teaches the computer-implemented method of claim 1, and Arefin further teaches that it comprises downscaling the super resolution image to match the size of the plurality of low-resolution images further comprises:
reducing, by one or more computer processors, a number of pixels associated with the super resolution image to match a number of pixels associated with the plurality of low-resolution images.
Related Work, pg. 3: “Many modern optimization-based approaches to MISR build a generative model that, given a high-resolution image, simulates the acquisition of low-resolution images.”
Results, pg. 6: “The low-resolution images were prepared with a shape of 128×128 pixels while the high-resolution (target) images contained 384 × 384 pixels…”
Explanation: This describes HR [Wingdings font/0xE0] LR transformation (downscaling), where HR has more pixels and LR has fewer pixels, requiring pixel reduction for matching.
Regarding Claim 8, Arefin in view of Varol and Bora teaches all of the limitations with respect to claim 1 above. Arefin further teaches the one or more computer readable storage media that perform substantially the same steps as claim 1 as the reference discloses the implementation of a neural network model in PyTorch which requires program instructions stored in a computer-readable medium for execution.
4.2 Experimental Setup, pg. 6: “Our model was implemented in PyTorch [35] and made publicly available3.”
Regarding Claim 10, Arefin in view of Varol and Bora teaches the computer program product of claim 8, and additional limitations are met as in the consideration of claim 3 above.
Regarding Claim 11, Arefin in view of Varol and Bora teaches the computer program product of claim 8, and additional limitations are met as in the consideration of claim 4 above.
Regarding Claim 12, Arefin in view of Varol and Bora teaches the computer program product of claim 8, and additional limitations are met as in the consideration of claim 5 above.
Regarding Claim 14, Arefin in view of Varol and Bora teaches the computer program product of claim 8, and additional limitations are met as in the consideration of claim 7 above.
Regarding Claim 15, Arefin in view of Varol and Bora teaches all of the limitations with respect to claim 1 above. Arefin further teaches the one or more computer readable storage media (see claim 8 above), one or more computer processors, and the one or more computer readable memories that perform substantially the same steps as claim 1.
4.2 Experimental Setup, pg. 6: “The training process took roughly 13 hours on a NVIDIA Titan RTX with memory of 24GB. During inference our model can super-resolve around 14 scenes per second in the same GPU if each scene is processed individually without batching.”
Explanation: GPU = processor
Regarding Claim 17, Arefin in view of Varol and Bora teaches the computer system of claim 15, and additional limitations are met as in the consideration of claim 3 above.
Regarding Claim 18, Arefin in view of Varol and Bora teaches the computer system of claim 15, and additional limitations are met as in the consideration of claim 4 above.
Regarding Claim 19, Arefin in view of Varol and Bora teaches the computer system of claim 15, and additional limitations are met as in the consideration of claim 5 above.
Regarding Claim 20, Arefin in view of Varol and Bora teaches the computer system of claim 15, and additional limitations are met as in the consideration of claim 7 above.
Claims 6 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Arefin et. al in view of Varol et. al and Bora et. al, further in view of El-Khamy (US10489887B2).
Regarding Claim 6, Arefin in view of Varol and Bora teaches the computer-implemented method of claim 1, but fails to teach that, responsive to transmitting the super resolution image to the user, the method provides, by one or more computer processors, an opportunity to the user to accept the super resolution image via a user interface.
However, El-Khamy teaches user interaction with a super-resolution system, including explicit UI-based acceptance/capture, stating that “according to some example embodiments, once the user is satisfied with a current frame and wants to capture the image, the user may transmit a signal (e.g., by selecting a button or prompt in a user interface for interacting with the progressive fusion SR imaging system 200) to the progressive fusion SR imaging system 200 to generate the high resolution image 206” (paragraph [0060]).
Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the multi-image super resolution method of Arefin, Varol, and Bora to include providing an opportunity for a user to accept the generated super-resolution image via a user interface. El-Khamy teaches a known technique of allowing a user to accept/capture a generated super-resolution image via a UI, which would improve usability by enabling user control over output selection and ensuring only satisfactory images are finalized. Incorporating such a user-interface mechanism into the system of Arefin, Varol, and Bora would have been a predictable use of prior art elements according to their established functions to improve usability and user control, yielding predictable results.
Regarding Claim 13, Arefin in view of Varol and Bora teaches the computer program product of claim 8, and additional limitations are met as in the consideration of claim 6 above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Park (US 20250124544 A1) teaches systems and methods for upsampling low-resolution content within a high-resolution image include obtaining a composite image and a mask.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM ADU-JAMFI whose telephone number is (571)272-9298. The examiner can normally be reached M-T 8:00-6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM ADU-JAMFI/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights