DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/31/2025 has been entered.
Response to Arguments
Applicant’s arguments, see Remarks page 9-10, filed 12/31/2025, with respect to the rejections of amended independent claim(s) 1 and 11 under 35 U.S.C. 103 have been fully considered and are moot in view of the new grounds of rejections (detailed in the rejections below) necessitated by Applicant’s amendment to the claim(s).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 4, 14, and 22 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 4, the recited limitation “obtaining the compensated image based on down-sampling the compensated image,” is indefinite because it is unclear how the compensated image is obtained, since the compensation image is needed for obtaining itself. For the purposes of examination, the limitation is interpreted as “wherein, the processing of the compensated image includes down-sampling.”
As per claim(s) 14 and 22, arguments made in rejecting claim(s) 4 are analogous.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-11, and 13-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ren et al. (Medical Video Super-Resolution Based on Asymmetric Back-Projection Network With Multilevel Error Feedback) hereinafter referenced as Ren, in view of Tao et al. (Detail-revealing Deep Video Super-resolution) hereinafter referenced as Tao, and Satou et al. (US2005232493A1) hereinafter referenced as Satou.
Regarding claim 1, Ren discloses: A method for microscope-based super-resolution, comprising: acquiring, using a microscope imaging system including a microscope and an image acquisition device, a to-be-processed image and at least an auxiliary image (Ren: Figure 10; Section V.-V.A: “The image dataset includes the DIV2K, Flickr, and ImageNet datasets, and the video dataset includes the Videoset4, Myanmar and some gastroscope and colonoscopy medical videos…four types of medical videos were selected, including 2 gastroscope videos (500 frames and 300 frames), 1 colonoscopy video (867 frames), 1 retinopathy video (268 frames), and 1 melanoma video (148 frames). The resolution of the video was low due to the limitations of the imaging environment…”; Wherein the retinopathy video frames constitute microscope images acquired using a microscopic imaging system), the to-be-processed image including a target area, the auxiliary image including an overlapping portion with the target area, the to-be-processed image and the auxiliary image being both microscope images of a first resolution (Ren: Figures 1 & 2; Section III.B: “First, we estimate the optical flow between video frames, and the warp operation is used to compensate for the motion of adjacent video frames. The centre frame is denoted as It, and the adjacent frames are denoted as It-1 and It+1.”; Wherein the center frame constitutes the to-be-processed image and the adjacent frames constitute the auxiliary images and wherein the contents of frames as shown in Figure 2 overlap.);
calculating an optical flow prediction between the to-be-processed image and the auxiliary image, the optical flow prediction being used to predict an optical flow change between the to-be-processed image and the auxiliary image; obtaining a compensated image with motion compensation based on the optical flow prediction and the auxiliary image; and obtaining a registered image based on the compensated image (Ren: Figure 1: Motion Estimation and Compensation; Section III.A: “The video SR method includes three steps: motion estimation, motion compensation, and SR reconstruction. Motion estimation is used to estimate motion between the frames of the LR image. Motion compensation can predict and compensate for the current partial image by using the previous partial image to register the adjacent frame image with the current image in the same coordinate system.”;
Section III.B: “We calculate the optical flow between It, It-1 and It, It+1, as shown in formulas 1 and 2. Based on the motion estimation, the motion compensation frames I’t-1 and I’t+1 of the adjacent frames It-1 and It+1 are calculated using the warp operation (w)”);
extracting one or more high-resolution features from the registered image, the one or more high-resolution features representing image features of the target area at a second resolution, the second resolution being greater than the first resolution (Ren: Figure 1: Patch Extraction and Nonlinear Mapping; Section III.C: “the input three frames I’LRt-1, ILRt, and I’LRt+1 are combined using a suitable fusion strategy to extract the initial features Finitial…asymmetric iterative up-and-down sampling units are used to realize nonlinear mapping from LR to SR, as shown in formula 8”; Wherein the extracted features are mapped from a low resolution to a super resolution constituting the extraction of high-resolution features.);
reconstructing, based on the one or more high-resolution features, a target image of the second resolution corresponding to the to-be-processed image of the first resolution (Ren: Figure 1: Reconstruction; Section III.C: “Finally, the SR frames of all the upper sampling units are concatenated together, and a convolution layer conv (1 x 1) is used to reconstruct the central frame.”); and
outputting the target image for display (Ren: Figure 2; Section VI: “In telemedicine systems, high-quality medical videos are an important basis for doctors to diagnose early diseases. However, some medical videos have lower resolution due to the current hardware manufacturing processes, hardware costs, available storage space, and system transmission conditions. We propose a medical video SR method based on an asymmetric back-projection network with multilevel error feedback that can reconstruct low-resolution medical videos into high-resolution medical videos.”; Wherein the up-sampled frames are output for display and disease diagnosis).
Ren does not disclose expressly: calculating an optical flow prediction map between the to-be-processed image and the auxiliary image, the optical flow prediction map being used to predict an optical flow change between the to-be-processed image and the auxiliary image; obtaining a compensated image with motion compensation based on the optical flow prediction map and the auxiliary image.
Tao discloses: calculating an optical flow prediction map between the to-be-processed image and the auxiliary image, the optical flow prediction map being used to predict an optical flow change between the to-be-processed image and the auxiliary image (Tao: Figure 2: Motion Estimation; Section 3.1: “The motion estimation module takes two LR frames as input and produces a LR motion field as
PNG
media_image1.png
28
259
media_image1.png
Greyscale
where Fi→j = (ui→j, vi→j) is the motion field from frame ILi to ILj. ΘME is the set of module parameters.”); obtaining a compensated image with motion compensation based on the optical flow prediction map and the auxiliary image (Tao: Figure 2: SPMC Layer; Figure 3(a); Section 3.2: “As indicated in Eq. (5), the SPMC layer takes one LR image JL and one flow field F = (u, v) as input, without other trainable parameters.”; Section 3.3: “The SPMC layer produces a series of motion compensated frames {JHi} expressed as
PNG
media_image2.png
25
275
media_image2.png
Greyscale
”; Wherein JL is an auxiliary image and JHi is a motion compensation map); and encoding and decoding the compensated image to obtain the registered image (Tao: Figure 2: Detail Fusion Net; Section 3.3: “The network structure includes
PNG
media_image3.png
63
246
media_image3.png
Greyscale
where NetE and NetD are encoder and decoder CNNs with parameters ΘE and ΘD. fi is the output of encoder net. gi is the input of decoder net. si is the hidden state for LSTM at the ith step. SEi for all i are intermediate feature maps of NetE, used for skip-connection. IL↑0 is the bicubic upsampled IL0. I(i)0 is the ith time step output.”; Wherein the motion compensation map is applied to the to-be-processed image to create I(i)0).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the algorithms for creating the motion compensated images I’LRt-1 and I’LRt+1 disclosed in Ren with the network disclosed in Tao. The suggestion/motivation for doing so would have been “In this paper, we show that proper frame alignment and motion compensation is crucial for achieving high quality results…Our implementation can generate visually and quantitatively high-quality results, superior to current state-of-the-arts, without the need of parameter tuning.” (Tao: Abstract). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results.
Ren in view of Tao does not disclose expressly: determining a ratio of (i) an overlap area of the overlapping portion of the auxiliary image to (ii) an overall area of the auxiliary image;
when the ratio is greater than a reference value: calculating an optical flow prediction map between the to-be-processed image and the auxiliary image.
Thus, Ren in view of Tao does not disclose expressly: the determining of a ratio of (i) an overlap area of the overlapping portion of the auxiliary image to (ii) an overall area of the auxiliary image, wherein the images are registered if the determined ratio is greater than a reference value.
Satou discloses: acquiring a detection image region and at least an observation image, the detection image region image including a target area, the observation image including an overlapping portion with the target area, the detection image region and the observation image being both microscope images of a first resolution (Satou: 0025: “ A pattern matching is performed with respect to the detection image region 101 and the observation image 102 by the pattern matching scheme. Namely, the detection image region 101 and the observation image 102 are superimposed while gradually shifting the relative positions there between in a scanning direction 103. This makes it possible to calculate degree of similarity in the range of a common region 104 which is common to both of the images 101 and 102.”);
determining a ratio of (i) an overlap area of the overlapping portion of the observation image to (ii) an overall area of the observation image, wherein it is determined whether the ratio is greater than a reference value (Satou: 0035-0037: “As threshold values, there exist the following categories. A threshold value for correlation values for performing a judgment on the correlation values, a threshold value for area for performing a judgment on the area of the common region 104…As the threshold value for the area, there exist the following categories. The threshold value for the area of the common region 104, a threshold value for the ratio of the area of the common region 104 relative to the observation image 102, threshold values for a longitudinal width and a transverse width of the common region 104…The threshold value for the area is used for making a judgment as to whether or not to calculate the correlation values. If the area is larger than the threshold value, the correlation values will be calculated.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the algorithm for determining whether to further process images based on an area threshold taught by Satou by determining whether to process the images disclosed by Ren in view of Tao based on whether the ratio of the area of the overlapping patch relative to the auxiliary image satisfies a threshold condition. The suggestion/motivation for doing so would have been “In the case of the present embodiment, the area of the observation image 102 and that of the detection image region 101 are substantially the same. Consequently, if the area of the common region 104 is small, the detection image region 101 can be considered to have overlapped with the observation image 102 along the peripheral portion of the image 102. In the case like this, the calculation of the correlation values will not be performed, since accuracy of the correlation values can be considered low.” (Satou: 0037). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ren in view of Tao with Satou to obtain the invention as specified in claim 1.
Regarding claim 3, Ren in view of Tao and Satou discloses: The method according to claim 1, wherein the calculating the optical flow prediction map comprises: invoking an optical flow prediction network configured to calculate the optical flow prediction map according to a first optical flow field of the to-be-processed image and a second optical flow field of the auxiliary image (Tao: Figure 2: Motion Estimation; Section 3.1: “The motion estimation module takes two LR frames as input and produces a LR motion field as
PNG
media_image4.png
50
457
media_image4.png
Greyscale
where Fi→j = (ui→j, vi→j) is the motion field from frame ILi to ILj. ΘME is the set of module parameters. Using neural networks for motion estimation is not a new Idea…We choose MCT because it has less parameters and accordingly less computation cost”; Wherein the Motion estimation calculates the flow field from i to 0 using the to-be-processed image and an adjacent frame).
Regarding claim 4, Ren in view of Tao and Satou discloses: The method according to claim 1, wherein the obtaining the compensated image comprises: invoking a super-resolution network to up-sample the optical flow prediction map to obtain an up-sampled map (Tao: Figure 3(a); Section 3.2: “In this step, transformed coordinates are first calculated according to estimated flow F = (u, v)…We denote transform of coordinates as operator WF:a, which depends on flow field F and scale factor a. xsp and ysp are the transformed coordinates in an enlarged image space, as shown in Fig. 3.”; Wherein the flow map is up-sampled); performing interpolation on the up-sampled map based on the auxiliary image to obtain an up-sampled compensated image with motion compensation information (Tao: Section 3.2: “Output image is constructed in the enlarged image space according to xsp and ysp. The resulting image JHq is
PNG
media_image5.png
89
512
media_image5.png
Greyscale
…M(·) is the sampling kernel, which defines the image interpolation methods (e.g. bicubic, bilinear, and nearest-neighbor) …the SPMC layer takes one LR image JL and one flow field F = (u, v) as input”); and obtaining the compensated image based on down-sampling the compensated image (Limitation is interpreted according to the Rejection of claim 4 under 35 U.S.C. 112(b) disclosed above) (Tao: Figure 2: Detail Fusion Net; Section 3.3: “The network structure includes
PNG
media_image6.png
129
508
media_image6.png
Greyscale
…The first layer of NetE and the last layer of NetD have kernel size 5 × 5. All other convolution layers use kernel size 3 × 3, including those inside ConvLSTM. Deconvolution layers are with kernel size 4 × 4 and stride 2.”; Wherein passing the image JHi through the encoder NetE constitutes the down-sampling of the compensated image.).
Regarding claim 5, Ren in view of Tao and Satou discloses: The method according to claim 1, wherein the obtaining the registered image comprises: invoking a deconvolution network on the compensated image to obtain an image residual (Tao: Figure 2: Detail Fusion Net; Section 3.3: “The network structure includes
PNG
media_image6.png
129
508
media_image6.png
Greyscale
…The first layer of NetE and the last layer of NetD have kernel size 5 × 5. All other convolution layers use kernel size 3 × 3, including those inside ConvLSTM. Deconvolution layers are with kernel size 4 × 4 and stride 2.”; Wherein passing the image JHi through the detail fusion network constitutes obtaining an image residual.); and fusing the image residual with the to-be-processed image to obtain the registered image (Tao: Figure 2: Detail Fusion Net; Section 3.3: “The network structure includes
PNG
media_image6.png
129
508
media_image6.png
Greyscale
where NetE and NetD are encoder and decoder CNNs with parameters ΘE and ΘD. fi is the output of encoder net. gi is the input of decoder net. si is the hidden state for LSTM at the ith step. SEi for all i are intermediate feature maps of NetE, used for skip-connection. IL↑0 is the bicubic upsampled IL0. I(i)0 is the ith time step output.”; Wherein the JHi passed through the network is added with an upsampled IL0, which constitutes fusing the image residual with the to-be-processed image).
Regarding claim 6, Ren in view of Tao and Satou discloses: The method according to claim 1, wherein the extracting the one or more high-resolution features comprises: extracting low-resolution features from the registered image, the low-resolution features representing first image features of the target area in the first resolution (Ren: Figure 1: Patch Extraction and Nonlinear Mapping; Section III.C: “the input three frames I’LRt-1, ILRt, and I’LRt+1 are combined using a suitable fusion strategy to extract the initial features Finitial , as shown in formula 7, and a convolution layer conv(1 x 1) is used to reduce the initial feature dimension and improve the reconstruction efficiency.”; Wherein the registered images are fused and processed by the convolutional layer to extract initial features); and mapping the low-resolution features in the first resolution to the one or more high-resolution features in the second resolution (Ren: Figure 1: Reconstruction; Section III.C: “in this paper, the ABPN structure is used, and asymmetric iterative up-down sampling units, multilevel error feedback, dense connections and depth cascading techniques are adopted. The ABPN can use a shallow network structure (12 up-down sampling units and 1 shared down-down sampling unit) to obtain high-quality SR reconstruction…asymmetric iterative up-and-down sampling units are used to realize nonlinear mapping from LR to SR, as shown in formula 8, and dense connections are used to improve the reconstruction quality. Finally, the SR frames of all the upper sampling units are concatenated together, and a convolution layer conv (1 x 1) is used to reconstruct the central frame.”).
Regarding claim 7, Ren in view of Tao and Satou discloses: The method according to claim 1, wherein the extracting the one or more high-resolution features from the registered image, comprises: fusing the registered image and the auxiliary image to obtain a fused image; and extracting the one or more high-resolution features from the fused image (Ren: Figure 1: Patch Extraction and Nonlinear Mapping; Section III.C: “the input three frames I’LRt-1, ILRt, and I’LRt+1 are combined using a suitable fusion strategy to extract the initial features Finitial…asymmetric iterative up-and-down sampling units are used to realize nonlinear mapping from LR to SR, as shown in formula 8, and dense connections are used to improve the reconstruction quality.”; Wherein the fusion of registered images consisting of auxiliary images constitutes fusing the registered image and the auxiliary image).
Regarding claim 8, Ren in view of Tao and Satou discloses: The method according to claim 1, wherein the reconstructing the target image of the second resolution comprises: converting the one or more high-resolution features into pixel values of pixel points in the target image through an image reconstruction network (Ren: Figure 1: Reconstruction; Section III.C: “asymmetric iterative up-and-down sampling units are used to realize nonlinear mapping from LR to SR, as shown in formula 8, and dense connections are used to improve the reconstruction quality. Finally, the SR frames of all the upper sampling units are concatenated together, and a convolution layer conv (1 x 1) is used to reconstruct the central frame.”; Wherein the concatenation of SR frames containing SR features constitutes the conversion of high-resolution features into pixel values of points in target image).
Regarding claim 9, Ren in view of Tao and Satou discloses: The method according to claim 1, further comprising: determining, from a microscope image sequence of the first resolution, the to-be- processed image and one or more candidate auxiliary images satisfying a correlation condition with the to-be-processed image; and determining, from the one or more candidate auxiliary images, the auxiliary image that has the ratio greater than the reference value (Satou: 0035-0037: “As threshold values, there exist the following categories. A threshold value for correlation values for performing a judgment on the correlation values, a threshold value for area for performing a judgment on the area of the common region 104…As the threshold value for the area, there exist the following categories. The threshold value for the area of the common region 104, a threshold value for the ratio of the area of the common region 104 relative to the observation image 102, threshold values for a longitudinal width and a transverse width of the common region 104…The threshold value for the area is used for making a judgment as to whether or not to calculate the correlation values. If the area is larger than the threshold value, the correlation values will be calculated.”; Wherein the ratio is calculated based on the overlap area and an area of a video frame).
Regarding claim 10, Ren in view of Tao and Satou discloses: The method according to claim 1, further comprising: invoking a target super-resolution model that is configured to: register the to-be-processed image and the auxiliary image to obtain the registered image (Tao: Figure 2: Motion Estimation; Figure 3(a); Section 3: “Our method takes a sequence of NF LR images as input and produces one HR image IH0 . It is an end-to-end fully trainable framework that comprises of three modules: motion estimation, motion compensation and detail fusion. They are respectively responsible for motion field estimation between frames; aligning frames by compensating motion”); extract the one or more high-resolution features from the registered image (Ren: Figure 1: Patch Extraction and Nonlinear Mapping; Section III.C: “the input three frames I’LRt-1, ILRt, and I’LRt+1 are combined using a suitable fusion strategy to extract the initial features Finitial…asymmetric iterative up-and-down sampling units are used to realize nonlinear mapping from LR to SR, as shown in formula 8”); and reconstruct, based on the one or more high-resolution features, the target image of the second resolution corresponding to the to-be-processed image of the first resolution.” (Ren: Figure 1: Reconstruction; Section III.C: “Finally, the SR frames of all the upper sampling units are concatenated together, and a convolution layer conv (1 x 1) is used to reconstruct the central frame.”).
As per claim(s) 11, arguments made in rejecting claim(s) 1 are analogous. In addition, Section V of Ren discloses the implementation and training of models for video enhancement using a CPU, Memory, and GPUs, which constitutes “processing circuitry.”
As per claim(s) 13, arguments made in rejecting claim(s) 3 are analogous.
As per claim(s) 14, arguments made in rejecting claim(s) 4 are analogous.
As per claim(s) 15, arguments made in rejecting claim(s) 5 are analogous.
As per claim(s) 16, arguments made in rejecting claim(s) 6 are analogous.
As per claim(s) 17, arguments made in rejecting claim(s) 7 are analogous.
As per claim(s) 18, arguments made in rejecting claim(s) 8 are analogous.
As per claim(s) 19, arguments made in rejecting claim(s) 9 are analogous.
As per claim(s) 20, arguments made in rejecting claim(s) 10 are analogous.
As per claim(s) 21, arguments made in rejecting claim(s) 1 are analogous. In addition, Section V of Ren discloses the implementation and training of models for video enhancement using a CPU, Memory, and GPUs, which constitutes a “non-transitory computer-readable storage medium storing instructions which when executed by a processor cause the processor to perform a method for microscope-based super-resolution.”
As per claim(s) 22, arguments made in rejecting claim(s) 4 are analogous.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J RODRIGUEZ whose telephone number is (703)756-5821. The examiner can normally be reached Monday-Friday 10am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J RODRIGUEZ/
Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights