Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 7, 9, 20 is/are rejected under 35 U.S.C. 103 over Ng (US 2023/0245289 A1) in view of Shopovska, I., Jovanov, L., & Philips, W. (2019). Deep Visible and Thermal Image Fusion for Enhanced Pedestrian Visibility. Sensors, 19(17), 3727. https://doi.org/10.3390/s19173727 (Hereinafter “Shopovska”).
Regarding claim 1, Ng teaches a system for performing asymmetric multi-modal image fusion, the system comprising:
one or more processors (paragraph [0013]); and
one or more computer-readable recording media that store instructions that are executable by the one or more processors (paragraph [0014]) to configure the system to:
access a first image associated with a first imaging modality (Figure 20 #2002, Figure 16 #1602, paragraph [0011] - RGB image as first imaging modality);
decompose the first image into a first base layer and a first detail layer (Figure 20 #2008, Figure 16 #1610, paragraph [0011] – RGB base portion & detail portion);
determine a weight map based on pixel signals of the first image (paragraph [0098], [0094], [0132]); and
generate an output image by performing image fusion using the first base layer, the first detail layer, and a second detail layer of a second image associated with a second imaging modality (Figure 20 #2010, Figure 16 #1604, paragraphs [0011], [0090], [0095] – NIR & NIR detail layer) that is different from the first imaging modality (Figure 20 #2002, paragraph [0011]), wherein the weight map modifies the first detail layer and the second detail layer in the image fusion (Figure 20 #2010, paragraphs [0053], [0090]). Ng describes an image
fusion system which teaches the limitations of the claimed invention. Ng describes decomposing images from two imaging modalities into a base layer and a detail layer. Ng further describes a weighted combination of the various base and detail portions to produce a fused image. Additionally, Ng suggests that the weights in the set of weights correspond to a weight map.
Ng fails to teach wherein the image fusion refrains from using a second base layer of the second image as an input.
However, Shopovska teaches wherein the image fusion refrains from using a second base layer of the second image as an input (pgs. 4, 8, 9, figs 4-6, eq. 1). Shopovska describes using saliency maps as “guiding information representing the foreground of the scene” (paragraph 3 of pg. 8) to help bring out details in the image while maintaining the RGB image’s appearance. In this manner, the saliency maps of Shopovska can be considered detail layers of their image fusion model. Shopovska goes on to describe the fusion process using the saliency maps, the RGB image and portions of the thermal images. In this process, Shopovska uses a pre-processing step of identifying pedestrians present in the RGB image. This pedestrian detection is used in determining contribution of the thermal image. For example at night, when pedestrians are harder to detect, the contribution of the thermal image information “will need to be added to the background scene” (paragraph 4 of pg. 8). Shopovska does this using a loss function described on page 9. Shopovska mentions that this loss function can completely ignore the thermal image by setting parameters to zero, and while this loss function creates a new problem of a vanishing gradient, Shopovska remedies this deficiency with its object detection pre-processing. This step forces the output to retain original RGB values in regions of good visibility. This retainment of original RGB values in subsection of an image (which can be considered its own image), inherently ignores the values of the thermal image and is analogous to refraining from using a second base layer of the second image.
Shopovska is considered analogous to the claimed invention as it is in the same field of image fusion. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the teachings of Shopovska with Ng in order to maintain the appearance of the first imaging modality while enhancing details of said image.
Regarding claim 2, Ng in view of Shopovska teaches the system of claim 1. Ng further teaches wherein the first imaging modality comprises a visible light imaging modality (Figure 20 #2002, paragraph [0011] – RGB image). Ng describes one of the imaging modalities as an RGB image, which is analogous to a visible light imaging modality.
Regarding claim 3, Ng in view of Shopovska teaches the system of claim 1. Ng further teaches wherein the second imaging modality comprises a thermal imaging modality (Figure 20 #2002, paragraph [0011] – infrared image). Ng describes the use of an infrared imaging modality. While infrared and thermal imaging are not exactly the same types of imaging, a thermal imaging modality utilizes infrared energy to passively detect heat. Therefore, the thermal imaging modality is an infrared imaging modality and Ng teaches the thermal imaging of the claimed invention.
Regarding claim 4, Ng in view of Shopovska teaches the system of claim 1. Ng further teaches wherein the first base layer comprises a first low-frequency component of the first image (paragraph [0138], & [0144] – “brgb corresponding to low frequency information”), and wherein the first detail layer comprises a first high- frequency component of the first image (paragraph [0138] & [0144] – “drgb corresponding to high frequency information”). Ng describes the base and detail layers of the first image (RGB image) as including low-frequency and high-frequency information,
Regarding claim 7, Ng in view of Shopovska teaches the system of claim 4. Ng further teaches wherein the second detail layer comprises a second high- frequency component of a second image associated with the second imaging modality (paragraph [0186] – second detail luminance portion dnir). Ng describes illuminance components in relation to gathering image information through the infrared imaging modality. This luminance portion is decomposed into a base and detail portion similarly to the RGB image. Ng further clarifies that “each base or detail portion corresponds to low-frequency or high-frequency information of the corresponding luminance component”.
Regarding claim 9, Ng in view of Shopovska teaches the system of claim 7. Ng further teaches wherein the second base layer comprises the second low-frequency component (Paragraphs [0138], [0144], [0186] – “second detail luminance portion dnir”).
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ng in view of Shopovska and in further view of Öztireli (US 2017/0024852 A1).
Regarding claim 5, Ng in view of Shopovska teaches the system of claim 4, and generating the first low-frequency component by upscaling the filtered first image (Ng, Figure 5, paragraphs [0090],[0160]). Ng describes generating a low-frequency component, decomposing the images using a guided image filter, and upscaling a filtered image while incorporating details from the other image.
Ng in view of Shopovska fails to teach wherein the first low-frequency component is determined by: generating a downscaled first image by downscaling the first image; generating a filtered downscaled first image by applying a blurring or smoothing filter to the downscaled first image;
However, Öztireli teaches wherein the first low-frequency component is determined by: generating a downscaled first image by downscaling the first image (paragraphs [0053], [0109]); generating a filtered downscaled first image by applying a blurring or smoothing filter to the downscaled first image (paragraphs [0069], [0088], [0127], [0135], Figure 14); Öztireli describes downscaling a first image as well as applying a smoothing filter “to capture lower frequency bands” while not exactly the same as the determination of a specified low-frequency component, the function of filtering to capture low-frequency is the same.
Öztireli is considered analogous to the claimed invention as it is in the same field of image processing. Therefore it would have been obvious, before the effective filing date, to incorporate the teachings of Öztireli with Ng in view of Shopovska to implement a downscaling method which can reduce the amount of computing effort needed (paragraph [0009]).
Claim(s) 6, 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ng in view of Shopovska and in further view of Price1 .
Regarding claim 6, Ng in view of Shopovska teaches the system of claim 4. Ng in view of Shopovska fails to teach wherein the first high-frequency component is determined by subtracting the first low-frequency component from the first image.
However, Price1 teaches wherein the first high-frequency component is determined by subtracting the first low-frequency component from the first image (paragraph [0085], [0086]). Price1 describes the calculating the high frequency of the second modality by subtracting the low frequency component from the original image. Price1 then goes on to describe the first image (referred to as “J”) as being decomposed in the same manner. Price1 is considered analogous to the claimed invention as it is in the same field of image fusion. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date to combine the teachings of Price1 with Ng in view of Shopovska to improve image fusion quality.
Regarding claim 11, Ng in view of Shopovska teaches the system of claim 1.
Ng in view of Shopovska fails to teach wherein the weight map comprises an alpha map.
However, Price1 teaches wherein the weight map comprises an alpha map (Abstract, Figure 16B, Paragraphs [0006], [0028], [0078]).
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ng in view of Shopvoska and in further view of Price1 (US 2022/0351345 A1).
Regarding claim 20, Ng teaches a system for performing asymmetric multi-modal image fusion, the system comprising:
one or more processors (paragraph [0013]); and
one or more computer-readable recording media that store instructions that are executable by the one or more processors (paragraph [0014]) to configure the system to:
access a first image associated with a first imaging modality (Figure 20 #2002, Figure 16 #1602, paragraph [0011] - RGB image as first imaging modality);
generate an output image at least by combining a component of the first image with a component of a second image associated with a second imaging modality (Figure 20 #2010, paragraphs [0011], [0090], [0095]) that is different from the first imaging modality (Figure 20 #2002, paragraph [0011]).
Ng fails to teach determine a per-pixel light level of the first image associated with the first imaging modality; and
wherein a per-pixel contribution of the component of the second image is negatively related to the per-pixel light level of the first image of the first imaging modality.
However, Shopovska teaches determine a per-pixel light level of the first image associated with the first imaging modality (“saliency map” pg. 8 paragraphs 1-4, first imaging modality is RGB); and
wherein a per-pixel contribution of the component of the second image is negatively related to the per-pixel light level of the first image of the first imaging modality (pg. 8 paragraphs 1-4, 8, pg. 9, figs. 1, 5-6). Shopovska describes using a combination of pedestrian detection and saliency maps of the images in order to determine which regions of the images have good visibility and how each image should contribute to the fused image. The saliency maps can be considered analogous to the per-pixel signal levels of both images. Additionally the determination of visibility based on object detection can be considered analogous to a per-pixel signal level. Furthermore, the contribution of the thermal image (the second image) is negatively related to the contribution of the first image based on this visibility determination. In other words, when a region of the first image has poor visibility the contribution of the second image is greater.
Shopovska is considered analogous to the claimed invention as it is in the same field of image fusion. Therefore it would have been obvious to one of ordinary skill in the art to combine the teachings of Shopovska with Ng for the same reasons described in claim 1.
Ng in view of Shopovska fails to teach a light level. However, Price1 teaches determine a light level (paragraph [0133]). Price is considered analogous to the claimed invention as it is in the same field of image fusion. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to substitute the determine per-pixel signal level of Shopvoska with the determined light level of Price1 in order to improve feature expression when the environment has a relatively lower amount of ambient light.
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Price1 (US 2022/0351345 A1) in view of Ng and Schürf (US 2018/0376048 A1).
Regarding claim 16, Price1 teaches
A system for performing asymmetric multi-modal image fusion, the system comprising:
one or more processors (paragraph [0142]); and
one or more computer-readable recording media that store instructions that are executable by the one or more processors (paragraph [0147) to configure the system to:
access a first image associated with a first imaging modality (paragraph [0125]);
decompose the first image into a first low-frequency component and a first high-frequency component (paragraph [0084], [0086]);
access a second image associated with a second imaging modality that is different from the first imaging modality (paragraph [0125]);
decompose the second image into a second low-frequency component and a second high-frequency component (paragraphs [0084], paragraph [0085]);
generate a fused detail layer by fusing the first high-frequency component associated with the first imaging modality with the second high-frequency component associated with the second imaging modality using a weight map based on pixel signals of the first image (paragraphs [0086]-[0088]); and
generate an output image by combining with the first low-frequency component associated with the first imaging modality (paragraphs [0084]-[0087]).
Price1 fails to teach generate an output image by combining the fused detail layer with the first low-frequency component
However Ng teaches generate an output image by combining the fused detail layer (paragraph [0095]). Ng is considered analogous to the claimed invention as it is in the same field of image fusion. Therefore, it would have been obvious to one of ordinary skill in the art to combine the teachings of Ng with Price1 to improve the quality of image fusion (Ng, paragraph [0053]).
Price1 in view of Ng fails to teach wherein generating the output image refrains from using the second low-frequency component
However, Schürf teaches wherein generating the output image refrains from using the second low-frequency component (paragraph [0033]-[0035]). Schürf describes specifically omitting low frequency components when generating a Laplacian Gaussian pyramid, which is used in the process of generating a combined output image. Schürf is considered analogous to the claimed invention as it is in the same field of invention of image processing and combing multiple images. Therefore it would have obvious to one of ordinary skill in the art to combine the teachings of Schürf to omit low-frequency information with Price1 and Ng to provide an efficient way of including higher frequencies of other images in the output image (paragraph [0036]).
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ng in view of Shopovska and in further view of Öztireli and Price1 (US 2022/0351345 A1).
Regarding claim 8, Ng in view of Shopovska teaches the system of claim 7 and generating a second low-frequency component by upscaling the filtered second image (Ng, Paragraphs [0090], [0160], Figure 5);
Ng in view of Shopovska fails to teach wherein the second detail layer is obtained by: generating a downscaled second image by downscaling the second image; generating a filtered downscaled second image by applying a blurring or smoothing filter to the downscaled second image; and subtracting the second low-frequency component from the second image.
However, Öztireli teaches wherein the second detail layer is obtained by: generating a downscaled second image by downscaling the second image (paragraph [0053]); generating a filtered downscaled second image by applying a blurring or smoothing filter to the downscaled second image (Paragraphs [0069], [0088], [0127], [0135], Figure 14); Öztireli describes downscaling a first image as well as applying a smoothing filter “to capture lower frequency bands” while not exactly the same as the determination of a specified low-frequency component, the function of filtering to capture low-frequency is the same. Therefore it would have been obvious, before the effective filing date, to incorporate the teachings of Öztireli with Ng in view of Shopovska to implement a downscaling method which can reduce the amount of computing effort needed (paragraph [0009]).
Öztireli fails to teach subtracting the second low-frequency component from the second image.
However, Price1 teaches subtracting the second low-frequency component from the second image. (paragraph [0085]). Price1 describes calculating the high frequency of the image captured by second modality by subtracting the low frequency component from the original image. Price1 is considered analogous to the claimed invention as it is in the same field of image fusion. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date to combine the teachings of Price1 with Ng in view of Shopovska and Öztireli to improve image fusion quality.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ng in view of Shopovska and in further view of Smirnov (US 10,853,928 B2).
Regarding claim 10, Ng in view of Shopovska teaches the system of claim 7. Ng teaches wherein the system comprise geometry-corrected images (Figure 5 #536). Ng describes geometry manipulation of the images before image fusion and while this manipulation can be considered correction, it is done during fusion and not comprised within the first and second images. Therefore Ng fails to teach wherein the first image and the second image comprise geometry-corrected images.
However, Smirnov teaches wherein the first image and the second image comprise geometry-corrected images (Col. 12 lines 4-18). Smirnov describes geometric as a part of the on the fly processing of processing of images which is directly analogous to the first and second images comprising geometry-corrected images. Therefore, it would have been obvious to substitute the method of Ng’s geometry manipulation with Smirnov’s geometry correction so that "enhancements to the captured image data can be performed in an expedient way without consuming other system resources.” (Col. 1 lines 22-25).
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ng in view of Shopovska and Price1 and in further view of Price2 (US 2023/0145672 A1).
Regarding claim 12, Ng in view of Shopovska and Price1 teaches the system of claim 11.
Ng in view of Shopovska and Price1 fails to teach wherein values of the alpha map are determined by evaluating a negative exponential function at each of the pixel signals of the first image.
However, Price2 teaches wherein values of the alpha map are determined by evaluating a negative exponential function (Figure 6, paragraphs [0071], [0083]) at each of the pixel signals of the first image (paragraph [0071]). Price2 describes output of the negative exponential function as being “for that pixel” which is implicitly describing the use of the negative exponential function at each of the pixels of the image.
Price2 is considered analogous to the claimed invention as it is in the same field of image processing. Therefore it would have been obvious to combine the teaching of Price2 with Ng in view of Shopovska and Price1 to utilize a negative exponential function in determining the alpha map in order to improve computer performance and image quality (paragraphs [0031] & [0087]).
Claim(s) 13, 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ng in view of Shopovska and in further view of Yonezawa (JP 2006031165 A).
Regarding claim 13, Ng in view of Shopovska teaches the system of claim 1. Ng in view of Shopovska fails to teach wherein the image fusion further uses a pedestal component to shift pixel values into a positive domain.
However Yonezawa teaches a pedestal component to shift pixel vales into a positive domain (Page 13, paragraph 5 – “By the way, the expansion process described above moves the pixels from the negative area image(for example, A− area image 122) to the positive area image (for example, A + area image 121) after generating each area image (121 to 124)”). Yonezawa is considered analogous to the claimed invention as it is in the same field of image processing and image combination. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the teachings of Yonezawa with Ng in view of Shopovska to implement a positive domaining shifting component in order to improve accuracy of the image combination (page 14 paragraph 4).
Regarding claim 15, Ng in view of Shopovska and Yonezawa teaches the system of claim 13. Yonezawa further teaches wherein the pedestal component is modified by the weight map (Pages 13, paragraph 4-6). Yonezawa describes a normalized correlation value image which can be considered analogous to a weight map. This is input into the expansion process to produce an expanded normalized correlation value image.
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ng in view of Shopovska and Yonezawa and in further view of Suthar (US 2024/0202874 A1).
Regarding claim 14, Ng in view of Shopovska and Yonezawa teaches the system of claim 13. Ng in view of Shopovska and Yonezawa fail to teach wherein the pedestal component is based on a differences between a maximum pedestal value and pixel values of the first base layer.
However, Suthar teaches wherein the pedestal component is based on a differences between a maximum pedestal value and pixel values of the first base layer (paragraphs [0062] & [0063]). Suthar describes utilizing the difference between maximum and minimum pixel values.
Suthar is considered analogous to the claimed invention as it is in the same field of computer graphics. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to incorporate the teachings of Suthar with Ng in view of Shopovska and Yonezawa, substituting the minimum pixel values of Suthar with the base layer pixel values of Ng, to improve the quality of blended images.
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Price1 in view of Ng and Schürf and in further view of Price2 (US 2023/0145672 A1).
Regarding claim 18, Price1 in view of Ng and Schürf teaches the system of claim 16.
Price1 in view of Ng and Schürf fails to teach wherein values of the weight map are determined by evaluating a negative exponential function at each of the pixel signals of the first image.
However, Price2 teaches wherein values of the weight map are determined by evaluating a negative exponential function (Figure 6, paragraphs [0071], [0083]) at each of the pixel signals of the first image (paragraph [0071]). Price2 describes output of the negative exponential function as being “for that pixel” which is implicitly describing the use of the negative exponential function at each of the pixels of the image.
Price2 is considered analogous to the claimed invention as it is in the same field of image processing. Therefore it would have been obvious to combine the teaching of Price2 with Price1 in view Ng and Schürf of to utilize a negative exponential function in determining the alpha map in order to improve computer performance and image quality (paragraphs [0031] & [0087]).
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Price1 in view of Ng and Schürf and in further view of Yonezawa (JP 2006031165 A).
Regarding claim 19, Price1 in view of Ng and Schürf teaches the system of claim 16. Price1 in view of Ng and Schürf fails to teach wherein the image fusion further uses a pedestal component to shift pixel values into a positive domain.
However Yonezawa teaches a pedestal component to shift pixel vales into a positive domain (Pages 13, paragraph 5). Yonezawa is considered analogous to the claimed invention as it is in the same field of image processing and image combination. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the teachings of Yonezawa with Ng implement a positive domaining shifting component in order to improve accuracy of the image combination (page 14 paragraph 4).
Response to Arguments
Applicant’s arguments with respect to claim(s) 20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant's arguments filed 03/10/2026 have been fully considered but they are not persuasive. Applicant argues on page 9, with regard to claim 1, that “Shopvoska fails to teach or suggest an embodiment in which the fusion process ‘refrains from using a second base layer of the second image as an input’ as recited in claim 1”. Applicant additionally argues Shopvoska “teaches a system that combines different losses to avoid the degenerative outcome in which thermal is ‘completely ignor[ed]’, rather than teaching or suggesting a fusion process that omits thermal-related input components from the fusion inputs. Examiner does not find this argument to be persuasive. Shopvoska does aim to incorporate features of the thermal input by using multiple loss functions to counteract a vanishing gradient effect. However, in order to “enforce that the fused image resembles the original RGB input” the minimization of the appearance loss function leads to a minimal contribution of the thermal image, which may be zero. While the complete omission of the thermal components in Shopvoska may lead to issues such as that of the vanishing gradient problem, the original claim language does not state omission of the thermal image components, but rather a “refrain” of using the second base layer of the second image. Under broadest reasonable interpretation, refraining from using is not the same as omitting or abstaining from use and Shopvoska.
Applicant argues on pages 9 and 10 that amended claim 16’s specification that the recited low and high frequency components are modality associated, and that the references of record fail to teach or suggest such features. Examiner respectfully disagrees. Applicant argues on page 10 that Schürf fails to teach the “base image” and “further image” being obtained from different imaging modalities. Examiner does not find this argument persuasive. Examiner disagrees with this interpretation of Schürf, and believes differing exposure times of a camera can be considered different imaging modalities under broadest reasonable interpretation. However, if Schürf fails to teach difference in imaging modalities, it would still be obvious to one of ordinary skill in the art to utilize the omission of a low-frequency component present in Schürf, with the specific modalities described in Price and Ng, with the same motivation to do so as stated in the claim.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. FUKUHARA (CN 103491377 A).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Aidan W McCoy whose telephone number is (571)272-5935. The examiner can normally be reached 8:00 AM-5:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571)272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AIDAN W MCCOY/Examiner, Art Unit 2611
/TAMMY PAIGE GODDARD/Supervisory Patent Examiner, Art Unit 2611