Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 7, 13 are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al (US9865036) in view of Lee et al (US20180225807).
Regarding Claim 1. Liu teaches A processor implemented method, comprising:
obtaining, via one or more hardware processors, a plurality of training images further comprising a set of low-resolution images 'X' of a target modality, a set of high-resolution images 'Y' of a guidance modality, and a set of high-resolution images 'Z' of the target modality (Liu, abstract, the invention describes A method of developing an image training library includes receiving, at a processor, a set of high resolution image samples at a first resolution, generating a set of low resolution image samples having a second resolution from the set of high resolution images, wherein the second resolution is lower than the first resolution, clustering the low resolution image samples using features in the low resolution images, generating a low resolution dictionary for each cluster, generating sparse coefficients for each sample, and using the sparse coefficients to generate a high resolution dictionary for each cluster.
Col 3, line 47-65, In FIG. 1, a display system 10 receives image data from one of many possible sources, including broadcast television, DVD, downloaded content, computer generated images, etc. The display system could be any type of device that includes a display 18, including a television, a computer monitor, a smart phone, a gaming console, etc. A receiver 12, which may be a television receiver, or merely an input buffer to receive the input image data, passes the image data to a video processor 14 that performs image processing on the data. A post processor 16 may perform further operations on the data including sharpening, upscaling, etc.
It is considered inherited that a processor has couple memory such as cache.);
initializing, via the one or more hardware processors, a plurality of
dictionaries and associated plurality of sparse coefficients, wherein the plurality of dictionaries comprise i) a first convolutional dictionary 'S' associated with a first set of sparse coefficients 'A' among the plurality of sparse coefficients, ii) a second convolutional dictionary 'G' associated with a second set of sparse coefficients 'B' among the plurality of sparse coefficients, iii) a first coupling convolutional dictionary 'W', and iv) a second coupling convolutional dictionary 'V' (Liu, col 2, line 24-44, The embodiments here involve sparse representation based image super resolution for upscaling from low resolution to high resolution images. This discussion will use several terms having particular definitions. "Low resolution dictionary" means a dictionary of features for low resolution image data. Similarly, "high resolution dictionary" means a dictionary of features for high resolution image data. The term "features" or "feature vector" generally means the high frequency components of both the low resolution and the high-resolution images.
Col 2, line 45-54, This process creates a dictionary of matched high and low resolution features in which a minimal linear combination of the features will accurately reconstruct the high resolution image patches from the low resolution image patches for the training set.
Col 2, line 55-63, At the reconstruction stage, in order to get an unknown high resolution patch, the process calculates the required sparse coefficients by using the patch's corresponding low resolution features and the low resolution dictionary features that matches those of the low resolution patch. The unknown high resolution sample feature vector is then sparsely represented by elements of the high resolution dictionary using the same coefficients that reconstructed the low resolution features from the low resolution dictionary.
Col 3, line 1-6, The embodiments here proposed to cluster input samples and train dictionaries for each cluster. This will result in smaller dictionaries allowing
for faster and more accurate reproduction of the high-resolution image.
Col 3, line 30-41, In the embodiments here, at the training stage, feature
vectors are generated for both the low and high resolution patches. The low-resolution features are used to generate clusters. Then for each cluster, sequential and joint dictionaries are learned. At the reconstruction stage, for each low-resolution feature vector, first determine which cluster it belongs to, then its high-resolution feature vector is initially reconstructed using sequential dictionaries. After all high-resolution features are reconstructed, refinements to the image data correct the errors and produce more details that are also more natural. One example of a refinement process is back projection constrained by joint dictionaries.
Col 4, line 1-11, FIG. 2 shows an overview of an embodiment of a process
to perform super resolution upscaling. The process has a training stage 30 and a reconstruction 40. During the training stage 30, a training sample library 32 is generated by collecting feature vectors of high and low resolution patches in all training images. The low resolution feature vectors are used for clustering at 34. The sequential high and low resolution dictionaries are learned at 36 and a joint dictionary is learned at 36. It should be noted that the training stage typically occurs only once during development of the video or post processor and is not part of the actual display system shown in FIG. 1. The display system includes the learned dictionaries and the reconstruction stage process.); and
Liu fails to explicitly teach, however, Lee teaches jointly training, via the one or more hardware processors, the initialized plurality of dictionaries and the associated plurality of sparse coefficients using the plurality of training images by performing a plurality of steps iteratively until convergence of an objective function is achieved, wherein the trained plurality of dictionaries and the associated plurality of sparse coefficients obtained upon achieving the convergence of the objective function are used for performing a multimodal image super resolution, and wherein the plurality of steps comprise:
training the plurality of sparse coefficients by keeping the plurality of dictionaries fixed; and
training the plurality of dictionaries by keeping the plurality of sparse coefficients fixed (Lee, abstract, the invention describes method and a device for single frame super resolution reconstruction based on sparse domain reconstruction, The disclosure mainly solves the technical problem in the prior art that the reconstructed image with high quality cannot be obtained by selecting the appropriate interpolation function according to the prior knowledge of the image. The disclosure adopts the first paradigm of the example mapping learning to train the mapping M of the low-resolution feature on the sparse domain Bl to the high-resolution feature on the sparse domain Bh, and the mapping of the high-resolution feature on the sparse domain Bh, to the high-resolution feature Ys, equalizing the mapping error and the reconstruction error to the mapping operator M, the reconstructed high-resolution dictionary ɸh, and the reconstructed high-resolution sparse coefficient Bh, the better solution to the problem, can be used for graphics processing.
[0079] Then, an iterative algorithm is proposed to establish the optimal target formula for the sparse domain reconstruction. Firstly, the initial optimization objective formula is established for the sparse representation term and the sparse domain mapping model of the high-resolution feature.
[0084] wherein, ɥh,i, represents the i atom of the high-resolution dictionary ɸh. According to the objective formula of the sparse domain reconstruction and the initial value of the high resolution dictionary ɸho, iteratively solving the high-resolution dictionary ɸh, the high-resolution characteristic coding coefficient Bh, the mapping matrix of the low-resolution characteristic coding coefficient to the high-resolution characteristic coding coefficient M, specifically, the obtained ɸho is used as the iterative initial value of the high-resolution dictionary, setting the iterative initial value of the high-resolution feature coding coefficient is set to Bh0=Bl, setting the iterative initial value of the mapping matrix to M0=E, where E represents the identity matrix; fixed the high-resolution feature coding coefficients Bh and mapping matrix M, so that it remains unchanged, the use of quadratic constrained quadratic programming method for high-resolution dictionary ɸh.
[0087] wherein, α is the coefficient of sparse domain mapping error term, which is 0.1, β is the coefficient of L1 norm optimization regular term, which is 0.01; Fixed high-resolution dictionary ɸh and high-resolution feature encoding coefficients Bh, keep it constant, using the ridge regression optimization method to solve the t iteration of the mapping matrix M(t).
Liu and Lee are analogous art because they both teach method of super-resolution reconstruction based on sparse domain reconstruction. Lee further teaches an iterative algorithm which establishes the optimal target formula for the sparse domain reconstruction with constant coefficient and dictionary. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the super-resolution reconstruction method (taught in Liu) to further use the iterative algorithm (taught in Lee), so as to provide a single-frame image super-resolution reconstruction algorithm based on sparse domain reconstruction, which can obtain high-quality reconstructed images based on prior knowledge of image selection and appropriate interpolation function (Lee, [0003]).
Claim 7 is similar in scope as Claim 1, and thus is rejected under same rationale.
Claim 13 is similar in scope as Claim 1, and thus is rejected under same rationale.
Allowable Subject Matter
Claims 2-6, 8-12, 14-18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding Claim 2, it recites “The method of claim 1, wherein the objective function is represented as:
PNG
media_image1.png
47
693
media_image1.png
Greyscale
PNG
media_image2.png
30
72
media_image2.png
Greyscale
” in the context of Claim 2.
The prior arts of record either alone or in combination fails to teach or suggest the above quoted limitation of Claim 2. Therefore, Claim 2 is allowable over prior art.
Regarding Claim 3, it recites “The method of claim 1, wherein training the plurality of sparse coefficients by keeping the plurality of dictionaries fixed comprises:
updating the first set of sparse coefficients based on the set of low-resolution images of the target modality, the set of high-resolution images of the target modality, the first convolutional dictionary, the first coupling convolutional dictionary, the second coupling convolutional dictionary, and the second set of sparse coefficients by solving
PNG
media_image3.png
42
187
media_image3.png
Greyscale
PNG
media_image4.png
41
235
media_image4.png
Greyscale
, wherein X' = Z - VB; and
updating the second set of sparse coefficients based on the set of high-resolution images of guidance modality, the set of high-resolution images of the target modality, the second convolutional dictionary, the first coupling convolutional dictionary, the second coupling convolutional
PNG
media_image5.png
42
105
media_image5.png
Greyscale
PNG
media_image6.png
38
310
media_image6.png
Greyscale
, wherein
PNG
media_image7.png
27
142
media_image7.png
Greyscale
” in the context of Claim 3.
The prior arts of record either alone or in combination fails to teach or suggest the above quoted limitation of Claim 3. Therefore, Claim 3 is allowable over prior art.
Regarding Claim 4, it recites “wherein training the plurality of dictionaries by
keeping the plurality of sparse coefficients fixed comprises:
updating the first convolutional dictionary based on the set of low-resolution images of the target modality;
updating the second convolutional dictionary based on the set of high-resolution images of the guidance modality;
updating the first coupling convolutional dictionary based on the set of high-resolution images of the target modality, the first set of sparse coefficients, the second set of sparse coefficients and the second coupling convolutional dictionary by converting into a standard Convolutional Dictionary Learning (CDL) problem; and
updating the second coupling convolutional dictionary based on the set of high-resolution images of the target modality, the first set of sparse coefficients, the second set of sparse coefficients and the first coupling convolutional dictionary by converting into a standard CDL problem.” in the context of Claim 4.
The prior arts of record either alone or in combination fails to teach or suggest the above quoted limitation of Claim 4. Therefore, Claim 4 is allowable over prior art.
Claim 5 depends from Claim 4 with respective additional limitations. Therefore, Claim 5 is allowable over prior art.
Regarding Claim 6, it recites “comprising performing multimodal image super
resolution on a new low-resolution image of the target modality by:
obtaining the new low-resolution image xtest of the target modality and a high-resolution image ytest of the guidance modality;
computing a first set of test coefficients Atest based on the trained first convolutional dictionary S and the low-resolution image of the target modality xtest by using a standard convolutional sparse coding update;
computing a second set of test coefficients Btest based on the trained second convolutional dictionary G and the high-resolution image of the guidance modality ytest by using the standard convolutional sparse coding update; and
generating a high-resolution image ztest of the target modality using the trained first coupling convolutional dictionary, the trained second coupling convolutional dictionary, the first set of test coefficients, and the second set of test coefficients as ztest = WAtest + VBtest.” in the context of Claim 6.
The prior arts of record either alone or in combination fails to teach or suggest the above quoted limitation of Claim 6. Therefore, Claim 6 is allowable over prior art.
Claim 8 recites similar limitations as discussed above with regard to claim 2. Therefore, claim 8 is allowable over prior art.
Claim 9 recites similar limitations as discussed above with regard to claim 3. Therefore, claim 9 is allowable over prior art.
Claim 10 recites similar limitations as discussed above with regard to claim 4. Therefore, claim 10 is allowable over prior art.
Claim 11 depends from Claim 10 with respective additional limitations. Therefore, Claim 11 is allowable over prior art.
Claim 12 recites similar limitations as discussed above with regard to claim 6. Therefore, claim 12 is allowable over prior art.
Claim 14 recites similar limitations as discussed above with regard to claim 2. Therefore, claim 14 is allowable over prior art.
Claim 15 recites similar limitations as discussed above with regard to claim 3. Therefore, claim 15 is allowable over prior art.
Claim 16 recites similar limitations as discussed above with regard to claim 4. Therefore, claim 16 is allowable over prior art.
Claim 17 depends from Claim 16 with respective additional limitations. Therefore, Claim 17 is allowable over prior art.
Claim 18 recites similar limitations as discussed above with regard to claim 6. Therefore, claim 18 is allowable over prior art.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Liu et al (US20190287216), abstract, the invention describes systems and methods for image processing for increasing resolution of a multi-spectral image. Accept a multi-spectral image including a set of images of a scene. A memory to store a set of dictionaries trained for different channels, and a set of filters trained for the different channels. A hardware processor is to process the set of images of the different channels with the set of filters, and to fuse, for each channel, the set of structures, to produce a set of fused structures. Wherein a fused structure of the channel is fused as a weighted combination of the set of structures using weights corresponding to the channel, such that the fused structures of different channels are combined with different weights. To process the set of fused structures with corresponding dictionaries from the set of dictionaries, to produce a super-resolution multi-spectral image. An output interface to render the super-resolution multi-spectral image.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIN SHENG whose telephone number is (571)272-5734. The examiner can normally be reached M-F 9:30AM-3:30PM 6:00PM-8:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at 5712723022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Xin Sheng/Primary Examiner, Art Unit 2619
Prosecution Insights