DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgement is made of Applicant’s claim of priority from JP2023-008181, filed January 23, 2023.
Information Disclosure Statement
The information disclosure statements (“IDS”) filed on January 22, 2024 was reviewed and the listed references were noted.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The following title is suggested: IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND NON-TRANSITORY COMPUTER-READABLE STORAGE MEDIUM FOR DEHAZING AN INPUT IMAGE.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-10 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claims recite a system, method, and non-transitory computer-readable medium for calculating atmospheric transmittance and sharpening the image based on calculated data. Consider method claim 9:
Step 1:
With regard to Step 1, the instant claim is directed to a method or a process; and therefore, the claim is directed to one of the statutory categories of invention.
Step 2A, Prong One:
With regard to 2A, Prong One, the limitations “calculating an atmospheric transmittance distribution based on an input image” and “sharpening the input image based on an illumination distribution that has been calculated based on the transmittance distribution” as drafted, recite an abstract idea, such as a process that, under its broadest reasonable interpretation, covers performance of the limitations manually and in the mind of a person. That is, a user or person skilled in the art may use mathematical concepts to manually calculate atmospheric transmittance distribution and apply calculated data to sharpen an image. This is the concept that falls under the grouping of abstract ideas mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations.
Step 2A, Prong Two:
The 2019 PEG defines the phrase “integration into a practical application” to require an additional step or a combination of additional steps in the claim to apply, rely on, or use the judicial exception. In addition, with respect to the system and computer-readable medium claims of claims 1 and 10, the mere recitation of a generic processor, memory, or storage medium to perform/store programming instructions of the recited/identified abstract idea does not integrate the identified abstract idea into a practical application. Accordingly, the above-mentioned additional elements/limitations do not integrate the abstract idea into a practical application; and therefore, the independent claims recite an abstract idea.
Step 2B:
Because the claims fail under Step 2A, the claims are further evaluated under Step 2B. The claims herein do not include additional elements that are sufficient to amount to significantly more than the judicial exception, because as discussed above with respect to integration of the abstract idea into practical application, the additional elements/limitations to perform the recited steps, amount to no more than insignificant extra-solution activity. Mere instructions to apply an exception using a generic component cannot provide an inventive concept. Therefore, independent claims 1, 9 and 10 are not patent eligible. In addition, claims 2-8 of the instant application provide limitations that both individually or in combination do not integrate the identified abstract idea into a practical application or provide significantly more than the identified abstract idea.
In order to overcome the abstract idea rejection, the claims need to describe particular transformation of the image (i.e., generating a corrected data) beyond a simple data manipulation or display of data.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1 and 7-10 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Lukac et al. (US 11,017,511 B2, filed February 13, 2019).
Regarding claim 1, Lukac teaches an image processing apparatus comprising:
one or more memories (Lukac, Col. 2, lines 37-57, the material disclosed herein also may be implemented as instructions stored on a machine-readable medium (i.e., memory), which may be read and executed by one or more processors); and
one or more processors (Lukac, Col. 2, lines 37-57, the material disclosed herein also may be implemented as instructions stored on a machine-readable medium (i.e., memory), which may be read and executed by one or more processors), wherein the one or more memories and the one or more processors are configured to:
calculate an atmospheric transmittance distribution based on an input image (Lukac, Col. Col. 8, lines 23-57, the process may include “estimate transmission distribution”. This process may include “determine minimum ratio between neighbor intensities and atmospheric light”, and particularly, by finding the minimum ratio between pixel intensities inside the neighborhood centered at a pixel location and the atmospheric light (i.e., calculate an atmospheric transmittance distribution based on an input image); and
sharpen the input image based on an illumination distribution that has been calculated based on the transmittance distribution (Lukac, Col. 10, lines 8-23, the process may include “normalize transmission values by dividing by luminance values” (i.e., calculate illumination distribution based on transmittance distribution). Col. 10, lines 47-62, the process may include generate correction parameters (i.e., from the normalized transmission values) and apply the correction parameters to the hazed image for correction (i.e., sharpen the input image)).
Regarding claim 7, Lukac teaches the image processing apparatus according to claim 1, further comprising:
an image capturing unit (Lukac, Col. 11, lines 48-48, the system has an image capture unit that may include camera hardware to capture images),
wherein the one or more memories and the one or more processors are further configured to generate the input image by performing development processing on a RAW image that is obtained by image capturing by the image capturing unit (Lukac, Col. 11, lines 48-48, the system has an image capture unit that may include camera hardware to capture images, or may include a camera module that receives raw image data from camera sensors for processing).
Regarding claim 8, Lukac teaches the image processing apparatus according to claim 1, wherein the one or more memories and the one or more processors are further configured to:
obtain, as the input image, an image captured by an image capturing apparatus (Lukac, Col. 11, lines 48-48, the system has an image capture unit that may include camera hardware to capture images).
Claim 9 recites a method with steps corresponding to the elements of the system recited in Claims 1. Therefore, the recited steps of this claim are mapped to the proposed reference in the same manner as the corresponding elements in its corresponding system claim.
Claim 10 recites a computer-readable storage medium storing a program with instructions corresponding to the steps recited in Claim 1. Therefore, the recited programming instructions of this claim are mapped to the proposed reference in the same manner as the corresponding steps in its corresponding method claim. Additionally, the Lukac reference discloses a computer readable storage medium (Lukac, Col. 2, lines 37-57, non-transitory article, such as a non-transitory computer readable medium).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Lukac et al. (US 11,017,511 B2, filed February 13, 2019) in view of Angamuthu Ganesan et al. (US 12,482,079 B2).
Regarding claim 2, Lukac teaches the image processing apparatus according to claim 1, wherein the one or more memories and the one or more processors are further configured to:
calculate the transmittance distribution based on the input image and the ambient light (Lukac, Col. Col. 8, lines 23-57, the process may include “estimate transmission distribution”. This process may include “determine minimum ratio between neighbor intensities and atmospheric light”, and particularly, by finding the minimum ratio between pixel intensities inside the neighborhood centered at a pixel location and the atmospheric light (i.e., ambient light)).
Although Lukac teaches obtaining the image airlight A by averaging several darkest pixels in the input image (Lukac, Col. 8, lines 7-22), Lukac does not explicitly teach to “generate, based on the input image, a dark channel image whose pixel values are minimum color channel values of respective local regions in the input image” and “calculate ambient light based on the dark channel image”. However, in an analogous field of endeavor, Angamuthu Ganesan teaches obtaining the dark channel image corresponding to the low-resolution image by determining the color channel (out of Red, Green, and Blue color channels) of minimum intensity (or near zero intensity) for each local patch (or a pixel) of the low-resolution image (i.e., pixel values are minimum color channel values of respective local regions in the input image) (Angamuthu Ganesan, Col. 7, lines 24-40). The atmospheric light estimation module identifies the top 10% brightest pixels in the dark channel image. Further, the atmospheric light estimation module compares each of the identified pixels (e.g. top 10% brightest pixels) with corresponding pixels in the actual received input digital image to identify the pixel(s) with highest intensity in the input digital image. The identified pixel(s) are then selected as atmospheric light, and accordingly, the atmospheric light value A (i.e., ambient light) is identified as the intensity of the identified pixel(s) (Angamuthu Ganesan, Col. 7 line 56 – Col. 8 line 4).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the apparatus of Lukac with the teachings of Angamuthu Ganesan by including determining a dark channel image whose pixel values are minimum color channel values and using the dark channel image to calculate ambient light (i.e., atmospheric light value). One having ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to combine these references because doing so would allow for improving visibility in hazy videos/images, as recognized by Angamuthu Ganesan. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Regarding claim 6, Lukac teaches the image processing apparatus according to claim 1, as described above.
Although Lukac teaches correcting a hazed image (Lukac, Col. 10, lines 8-23), Lukac does not explicitly teach to “perform gamma correction on the sharpened input image”. However, in an analogous field of endeavor, Angamuthu Ganesan teaches adjusting the brightness of the de-hazed output image (i.e., sharpened input image) by using gamma correction technique (Angamuthu Ganesan, Col. 9, lines 20-39).
The proposed combination as well as the motivation for combining the Lukac and Angamuthu Ganesan references presented in the rejection of Claim 2, apply to Claim 6 and are incorporated herein by reference. Thus, the system recited in Claim 6 is met by Lukac in view of Angamuthu Ganesan.
Claims 3-5 are rejected under 35 U.S.C. 103 as being unpatentable over Lukac et al. (US 11,017,511 B2, filed February 13, 2019) in view of Angamuthu Ganesan et al. (US 12,482,079 B2), as applied to claims 2 and 6 above, and further in view of Chen et al. (“A High-Efficiency and High-Speed Gain Intervention Refinement Filter for Haze Removal”), Han et al. (US 2008/0037868 A1) and Shinnosuke Osawa (US 2021/0233218 A1).
Regarding claim 3, Lukac in view of Angamuthu Ganesan teaches the image processing apparatus according to claim 2, as described above.
Although Lukac in view of Angamuthu Ganesan teaches sharpening an input image based on an illumination distribution calculated from a transmittance distribution (Lukac, Col. 10, lines 8-23), they do not explicitly teach to “calculate a first gain based on the input image, the ambient light, an application amount of dehazing processing, and the transmittance distribution”. However, in an analogous field of endeavor, Chen teaches obtaining a refined transmission map through gain intervention (i.e., based on transmittance distribution), wherein the gain coefficient is calculated based on the total number of pixels, the difference of position between the minimum intensity of trichromatic components within each pixel in the image (i.e., based on the input image) and the dark channel intensity of the image (i.e., based on the ambient light and amount of dehazing processing) (Chen, pg. 755).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the apparatus of Lukac in view of Angamuthu Ganesan with the teachings of Chen by including a gain calculation based on a transmission map (i.e., transmittance distribution), the input image, the ambient light, and an application amount of dehazing processing. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for dehazing images, as recognized by Chen.
Although Lukac in view of Angamuthu Ganesan further in view of Chen teaches refining a transmission map through gain intervention (Chen, pg. 755), they do not explicitly teach to “obtain a second gain based on the illumination distribution”. However, in an analogous field of endeavor, Han teaches determining an illumination level (i.e., illumination distribution) and providing a tone gain corresponding to the illumination level (Han, Para. [0093]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the apparatus of Lukac in view of Angamuthu further in view of Chen with the teachings of Han by including a second gain based on an illumination distribution (i.e., illumination level). One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for improving the visibility of an image according to the luminance of the environment, as recognized by Han.
Although Lukac in view of Angamuthu further in view of Chen and Han teaches refining a transmission map through gain intervention (Chen, pg. 755), they do not explicitly teach to “sharpen the input image based on a third gain which has been obtained based on the first gain, the second gain, and the illumination distribution; on the input image; on the transmittance distribution; on the application amount; and on the ambient light”. However, in an analogous field of endeavor, Osawa teaches a third gain map generation unit applies based on a mixed ratio image, a value of the second gain map to a gradation area and a value of the first gain map to an area that is not the gradation area to generate a third gain map (i.e., third gain based on first and second gain). The gain processing unit performs gain processing on the input image supplied via the image input unit based on the third gain map generated by the third gain map generation unit. The image on which tone correction (tone compression) by the gain processing based on the third gain map has been performed by the gain processing unit is output as an output image (i.e., sharpen input image based on third gain) (Osawa, Paras. [0021]-[0022]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the apparatus of Lukac in view of Angamuthu Ganesan further in view of Chen and Han with the teachings of Osawa by including a sharpening an input image using third gain based on the first and second gain (and based on illumination distribution, input image, transmittance distribution, application amount and on the ambient light because first and second gain are based on these values). One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for performing gain processing on an optically blurred portion in an image, as recognized by Osawa. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Regarding claim 4, Lukac in view of Angamuthu Ganesan further in view of Chen, Han and Osawa teaches the image processing apparatus according to claim 3, wherein the one or more memories and the one or more processors are further configured to:
use, as the application amount, a haze density that has been obtained based on a luminance value of the input image (Lukac, Col. 7, lines 39-50, The process then may include “obtain image statistics”, and particularly obtain the statistics, or the image data values, to be used to estimate the haze corruption. In the example used herein, pixel luminance data will be used (i.e., haze density based on luminance value)).
Regarding claim 5, Lukac in view of Angamuthu Ganesan further in view of Chen, Han and Osawa teaches the image processing apparatus according to claim 3, wherein the one or more memories and the one or more processors are further configured to:
use, as the application amount, a value that has been set according to a user operation (Lukac, Col. 7, lines 23-37, process may include “detect haze” and this may be performed by known statistics and atmospheric scattering model analysis techniques or determined by the user such as by selecting an area by bounding box on a screen as one example (i.e., a value set according to a user)).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Emma Rose Goebel whose telephone number is (703)756-5582. The examiner can normally be reached Monday - Friday 7:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at (571) 272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Emma Rose Goebel/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662