DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgement is made of applicant’s claim for priority under 35 U.S.C. 119(e) to US provisional application, 63/610,664, filed 12/15/2023.
Specification
The disclosure is objected to because of the following informalities:
[¶0085] of the specification state that “the right side of FIGURE 7 is treated as input, and the left side of FIGURE 7 is treated as output for the converter(s) 403”. When looking at FIGURE 7, an arrow from left to right is shown, indicating the left is treated as an input rather than the right.
Appropriate correction is required.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 6, 13 & 19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
In claim 6, the limitation “and a corresponding color channel for the ground truth image” is recited. A second plurality of color channels for the noisy ground truth is first recited in claim 1, which claim 6 depends on. It is unclear if applicant is introducing a new plurality of color channels distinct from the second plurality of color channels for the ground truth, or if these corresponding color channels for the ground truth are a distinct set of color channels for the ground truth image that has not had noise added. The examiner interprets this limitation to be the same as the previously recited second plurality of color channels.
Claims 13 & 19 recite the same limitation and is subsequently rejected under the same grounds as claim 6.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4, 7-8, 11, 14, 17 & 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pan et al (US 2021/0241429 A1), hereinafter referred to as “Pan”, in view of Egorov et al (US 2021/0390658 A1), hereinafter referred to as “Egorov”.
Regarding claim 1, Pan disclose a deep residual network for color filter array images that trained with noise augmented data to make the model robust to various levels of noise. More specifically, Pan teach A method (the method used to train the network 500 [¶0036,41; Figs. 5 & 6]) comprising:
obtaining a noisy training image (a noisy multispectral filter arrays (MSFA) raw image is obtained in step 605 [¶0041; Fig. 6]) and a ground truth image (noise-free images [¶0028 and 0038]);
converting the noisy training image into a first plurality of color channels (channel splitting of the noisy MSFA raw image in step 605 [0041; Fig. 5 & 6]);
generating, using an artificial intelligence/machine learning (AI/ML)-based denoiser, denoised images from the first plurality of color channels, wherein each denoised image corresponds to a respective color channel of the first plurality of color channels (a denoised image per channel C is produced via a deep learning model [0039-41; 44-48; Figs. 5 & 6]);
generating a noisy ground truth image from the ground truth image (a noise-free (ground-truth) raw image can be augmented to have noise added [0038]);
converting the noisy ground truth image into a second plurality of color channels (a noise-free image can be sampled for each respective spectral channel i corresponding to C channels of the MSFA raw image [¶0028 & 41]);
determining a loss based on a comparison between the denoised images and the second plurality of color channels; and (a loss is calculated between z (noise-free, ground truth image) and
z
^
(the denoised MSFA image) [0039; eq. 6], this is loss is carried on for split-channel denoising experiments [0045-47]), but fails to teach adapting the weights of the deep residual network based on a loss.
Egorov, however, is analogous art pertinent to the field of endeavor and disclose an image processing algorithm leveraging an neural network with adaptive weighting based on a loss function. More particularly, Egorov teach adapting weights of the AI/ML-based denoiser based on the loss determined based on the comparison between the denoised images and the second plurality of color channels (Egorov: adjusting filter weights of the neural network 103 to minimize a loss function across an enhanced (denoised) image 0002 and a high quality bayer image 0003 [0027-28; 62-63; Fig. 2]).
Egorov further teach that their neural network is directed to improve overall visual quality of CFM images prior to any downstream lossy image processing [¶0052]. Therefore, it would have been obvious to one of ordinary skill to combine the adaptive loss function weighting of Egorov to improve the loss optimization of Pan for enhanced image quality to arrive at the invention of the instant application.
With respect to claim 4, Pan in view of Egorov teach The method of Claim 1 (as described above), wherein generating the noisy ground truth image comprises adding zero-mean Gaussian noise to the ground truth image (Egorov: performing image degradation by adding gaussian noise to regular (ground-truth) images to generate noisy images [0060]).
Egorov further teach that their neural network is directed to improve overall visual quality of CFM images prior to any downstream lossy image processing [¶0052]. Therefore, it would have been obvious to one of ordinary skill to combine the adaptive loss function weighting of Egorov to improve the loss optimization of Pan for enhanced image quality to arrive at the invention of the instant application.
As for claim 7, Pan in view of Egorov teach The method of Claim 1 (as described previously), wherein the noisy training image comprises one of a plurality of noisy training images, each noisy training image having a different proportion of noise (Pan: the average PSNR (peak signal-to-noise ratio) for each set of experiments are calculated, with the PSNR from the noisy raw images included for reference, indicating a non-uniform degree of noise across the raw noisy images [0048; Table 1]).
Regarding claim 8, Pan teach An electronic device (computing system 700 [¶0056; Fig. 7]) comprising:
at least one processing device configured to train an artificial intelligence/machine learning (AI/ML)-based denoiser (CPU(s) 701 of the computing system that execute the training of the deep residual network [¶0056; Fig. 7]);
wherein, to train the AI/ML-based denoiser, the at least one processing device is configured to:
obtain a noisy training image (a noisy multispectral filter arrays (MSFA) raw image is obtained in step 605 [¶0041; Fig. 6]) and a ground truth image (noise-free images [¶0038]);
convert the noisy training image into a first plurality of color channels (channel splitting of the noisy MSFA raw image in step 605 [0041; Fig. 5 & 6]);
generate, using the AI/ML-based denoiser, denoised images from the first plurality of color channels, wherein each denoised image corresponds to a respective color channel of the first plurality of color channels (a denoised image per channel C is produced via a deep learning model [0039-41; 44-48; Figs. 5 & 6]);
generate a noisy ground truth image from the ground truth image (a noise-free (ground-truth) raw image can be augmented to have noise added [0038]);
convert the noisy ground truth image into a second plurality of color channels (a noise-free image can be sampled for each respective spectral channel i corresponding to C channels of the MSFA raw image [¶0028 & 41]);
determine a loss based on a comparison between the denoised images and the second plurality of color channels; and (a loss is calculated between z (noise-free, ground truth image) and
z
^
(the denoised MSFA image) [0039; eq. 6], this is loss is carried on for split-channel denoising experiments [0045-47]), but fails to teach adapting the weights of the deep residual network based on a loss.
Egorov, on the other hand, is analogous art pertinent to the field of endeavor and disclose an image processing algorithm leveraging an neural network with adaptive weighting based on a loss function. Egorov teach adapt weights of the AI/ML-based denoiser based on the loss determined based on the comparison between the denoised images and the second plurality of color channels (Egorov: adjusting filter weights of the neural network 103 to minimize a loss function across an enhanced (denoised) image 0002 and a high quality bayer image 0003 [0027-28; 62-63; Fig. 2]).
Egorov further teach that their neural network is directed to improve overall visual quality of CFM images prior to any downstream lossy image processing [¶0052]. Therefore, it would have been obvious to one of ordinary skill to combine the adaptive loss function weighting of Egorov to improve the loss optimization of Pan for enhanced image quality to arrive at the invention of the instant application.
Considering claim 11, Pan in view of Egorov teach The electronic device of Claim 8 (as described above), wherein, to generate the noisy ground truth image, the at least one processing device is configured to add zero-mean Gaussian noise to the ground truth image (Egorov: performing image degradation by adding gaussian noise to regular (ground-truth) images to generate noisy images [0060]).
With respect to claim 14, Pan teach A method (process 600 [¶0041; Fig. 6]) comprising:
obtaining a noisy captured image (a noisy multispectral filter arrays (MSFA) raw image is obtained in step 605 [¶0041; Fig. 6]); and
denoising the noisy captured image using an artificial intelligence/machine learning (AI/ML)-based denoiser (a denoised image per channel C is produced via a deep learning model [0039-41; 44-48; Figs. 5 & 6]), wherein the AI/ML-based denoiser is trained by:
obtaining a noisy training image (a noisy multispectral filter arrays (MSFA) raw image is obtained in step 605 [¶0041; Fig. 6]) and a ground truth image (noise-free images [¶0038]);
converting the noisy training image into a first plurality of color channels (channel splitting of the noisy MSFA raw image in step 605 [0041; Fig. 5 & 6]);
generating, using the AI/ML-based denoiser, denoised images from the first plurality of color channels, wherein each denoised image corresponds to a respective color channel of the first plurality of color channels (a denoised image per channel C is produced via a deep learning model [0039-41; 44-48; Figs. 5 & 6]);
generating a noisy ground truth image from the ground truth image (a noise-free (ground-truth) raw image can be augmented to have noise added [0038]);
converting the noisy ground truth image into a second plurality of color channels (a noise-free image can be sampled for each respective spectral channel i corresponding to C channels of the MSFA raw image [¶0028 & 41]);
determining a loss based on a comparison between the denoised images and the second plurality of color channels; and (a loss is calculated between z (noise-free, ground truth image) and
z
^
(the denoised MSFA image) [0039; eq. 6], this is loss is carried on for split-channel denoising experiments [0045-47]), but fails to teach adapting the weights of the deep residual network based on a loss.
Egorov, however, is analogous art pertinent to the field of endeavor and disclose an image processing algorithm leveraging an neural network with adaptive weighting based on a loss function. More particularly, Egorov teach adapting weights of the AI/ML-based denoiser based on the loss determined based on the comparison between the denoised images and the second plurality of color channels (Egorov: adjusting filter weights of the neural network 103 to minimize a loss function across an enhanced (denoised) image 0002 and a high quality bayer image 0003 [0027-28; 62-63; Fig. 2]).
Egorov further teach that their neural network is directed to improve overall visual quality of CFM images prior to any downstream lossy image processing [¶0052]. Therefore, it would have been obvious to one of ordinary skill to combine the adaptive loss function weighting of Egorov to improve the loss optimization of Pan for enhanced image quality to arrive at the invention of the instant application.
Regarding claim 17, Pan in view of Egorov teach The method of Claim 14 (as described above), wherein generating the noisy ground truth image comprises adding zero-mean Gaussian noise to the ground truth image (Egorov: performing image degradation by adding gaussian noise to regular (ground-truth) images to generate noisy images [0060]).
Egorov further teach that their neural network is directed to improve overall visual quality of CFM images prior to any downstream lossy image processing [¶0052]. Therefore, it would have been obvious to one of ordinary skill to combine the adaptive loss function weighting of Egorov to improve the loss optimization of Pan for enhanced image quality to arrive at the invention of the instant application.
As for claim 20, Pan in view of Egorov teach The method of Claim 14 (as described above), wherein the noisy training image comprises one of a plurality of noisy training images, each noisy training image having a different proportion of noise (Pan: the average PSNR (peak signal-to-noise ratio) for each set of experiments are calculated, with the PSNR from the noisy raw images included for reference, indicating a non-uniform degree of noise across the raw noisy images [0048; Table 1]).
Claim(s) 2, 9, 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pan et al (US 2021/0241429 A1), hereinafter referred to as “Pan”, in view of Egorov et al (US 2021/0390658 A1), hereinafter referred to as “Egorov”, further in view of Zeng et al (“Inheriting Bayer’s Legacy: Joint Remosaicing and Denoising for Quad Bayer Image Sensor, arXiv, 2023), hereinafter referred to as “Zeng”.
As for claim 2, Pan in view of Egorov The method of Claim 1 (as described previously), however they fail to teach the noisy and training image having a non-Bayer CFA pattern.
Zeng, however, is analogous art pertinent to the field of endeavor and disclose a method for denoising Quad Bayer images. More particularly, Zeng teach wherein:
the noisy training image and the ground truth image have an identical image color filter array (CFA) pattern; and
the identical image CFA pattern is one of: a Tetra CFA pattern (the input noisy image and corresponding ground truth are from a Quad Bayer mosaic – the examiner notes that Quad Bayer is identical to Tetra CFA patterns [Sec 4.3 Dual-head Joint Remosaic and Denoise - ¶03-05 & Sec 4.4 Bottleneck Data Mining - ¶01; Fig. 5]), a Hexa-Deca CFA pattern, or a Nona CFA pattern (Zeng: the examiner notes that given the use of the disjunctive “or”, only one of the listed limitations needs to be mapped to).
Zeng provides that their dual-head joint remosaicing and denoising network enables the conversion of Quad Bayer patterns without any loss is image resolution [Sec – Abstract]. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to incorporate aspects of Zeng’s network specific to Quad Bayer to the denoising system disclosed by Pan in view of Egorov to arrive at the invention of the instant application.
Considering claim 9, Pan in view of Egorov The electronic device of Claim 8 (as described previously), however they fail to teach the noisy and training image having a non-Bayer CFA pattern.
Zeng, on the other hand, is analogous art pertinent to the field of endeavor and disclose a method for denoising Quad Bayer images. More particularly, Zeng teach wherein:
the noisy training image and the ground truth image have an identical image color filter array (CFA) pattern; and
the identical image CFA pattern is one of: a Tetra CFA pattern (the input noisy image and corresponding ground truth are from a Quad Bayer mosaic – the examiner notes that Quad Bayer is identical to Tetra CFA patterns [Sec 4.3 Dual-head Joint Remosaic and Denoise - ¶03-05 & Sec 4.4 Bottleneck Data Mining - ¶01; Fig. 5]), a Hexa-Deca CFA pattern, or a Nona CFA pattern (Zeng: the examiner notes that given the use of the disjunctive “or”, only one of the listed limitations needs to be mapped to).
Zeng provides that their dual-head joint remosaicing and denoising network enables the conversion of Quad Bayer patterns without any loss is image resolution [Sec – Abstract]. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to incorporate aspects of Zeng’s network specific to Quad Bayer to the denoising system disclosed by Pan in view of Egorov to arrive at the invention of the instant application.
With respect to claim 15, Pan in view of Egorov The method of Claim 14 (as described previously), however they fail to teach the noisy and training image having a non-Bayer CFA pattern.
Zeng, however, is analogous art pertinent to the field of endeavor and disclose a method for denoising Quad Bayer images. More particularly, Zeng teach wherein:
the noisy training image and the ground truth image have an identical image color filter array (CFA) pattern; and
the identical image CFA pattern is one of: a Tetra CFA pattern (the input noisy image and corresponding ground truth are from a Quad Bayer mosaic – the examiner notes that Quad Bayer is identical to Tetra CFA patterns [Sec 4.3 Dual-head Joint Remosaic and Denoise - ¶03-05 & Sec 4.4 Bottleneck Data Mining - ¶01; Fig. 5]), a Hexa-Deca CFA pattern, or a Nona CFA pattern (Zeng: the examiner notes that given the use of the disjunctive “or”, only one of the listed limitations needs to be mapped to).
Zeng provides that their dual-head joint remosaicing and denoising network enables the conversion of Quad Bayer patterns without any loss is image resolution [Sec – Abstract]. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to incorporate aspects of Zeng’s network specific to Quad Bayer to the denoising system disclosed by Pan in view of Egorov to arrive at the invention of the instant application.
Claim(s) 3, 10 & 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pan et al (US 2021/0241429 A1), hereinafter referred to as “Pan”, in view of Egorov et al (US 2021/0390658 A1), hereinafter referred to as “Egorov”, further in view of Pan et al (“DeSmoke-LAP: improved unpaired image-to-image translation for desmoking in laproscopic surgery”, 2022, International Journal of Computer Assisted Radiology and Surgery), hereinafter referred to as “DeSmoke”.
Considering claim 3, Pan in view of Egorov teach The method of Claim 1 (as described previously), wherein determining the loss comprises using a linear combination of a mean absolute error (L1) loss, a multi-scale structural similarity loss, (Egorov: the loss function optimizer 204 can train CNN 103 using a combination of different loss functions like an L1 loss function (mean absolute error), an MS-SSIM (multi-scale structural similarity loss) [¶0063; Fig. 2]).
Egorov further teach that their neural network is directed to improve overall visual quality of CFM images prior to any downstream lossy image processing [¶0052]. Therefore, it would have been obvious to one of ordinary skill to combine the adaptive loss function weighting of Egorov to improve the loss optimization of Pan for enhanced image quality to arrive at the invention of the instant application. Egorov, however, fails to teach utilizing an inter-channel loss.
DeSmoke, however, is analogous art pertinent to the field of endeavor of image correction and disclose a method using an inter-channel loss function for removing smoke from laparoscopic images while maintaining original scene illumination and semantics. DeSmoke teach and an inter-channel loss (DeSmoke: an inter-channel (IC) loss is utilized to describe the differences between channels of a pixel in an image [Sec – Proposed Method, Subsec – Inter-channel (IC) Loss; Eq. 5-8).
DeSmoke further teach that the difference in pixel values across an inter-channel loss accounts for the level of blur for a given pixel region [Sec – Proposed Method, Subsec – Inter-channel (IC) Loss]. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to formulate a linear combination of the inter-channel loss taught by DeSmoke, in addition to the previously taught L1 and MS-SSIM loss of Pan in view of Egorov to control for blurring caused during the denoising process.
As for claim 10, Pan in view of Egorov teach The electronic device of Claim 8 (as described previously), wherein, to determine the loss, the at least one processing device is configured to use a linear combination of a mean absolute error (L1) loss, a multi-scale structural similarity loss (Egorov: the loss function optimizer 204 can train CNN 103 using a combination of different loss functions like an L1 loss function (mean absolute error), an MS-SSIM (multi-scale structural similarity loss) [¶0063; Fig. 2]).
Egorov further teach that their neural network is directed to improve overall visual quality of CFM images prior to any downstream lossy image processing [¶0052]. Therefore, it would have been obvious to one of ordinary skill to combine the adaptive loss function weighting of Egorov to improve the loss optimization of Pan for enhanced image quality to arrive at the invention of the instant application. Egorov, however, fails to teach utilizing an inter-channel loss.
DeSmoke, however, is analogous art pertinent to the field of endeavor of image correction and disclose a method using an inter-channel loss function for removing smoke from laparoscopic images while maintaining original scene illumination and semantics. DeSmoke teach and an inter-channel loss (DeSmoke: an inter-channel (IC) loss is utilized to describe the differences between channels of a pixel in an image [Sec – Proposed Method, Subsec – Inter-channel (IC) Loss; Eq. 5-8).
DeSmoke further teach that the difference in pixel values across an inter-channel loss accounts for the level of blur for a given pixel region [Sec – Proposed Method, Subsec – Inter-channel (IC) Loss]. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to formulate a linear combination of the inter-channel loss taught by DeSmoke, in addition to the previously taught L1 and MS-SSIM loss of Pan in view of Egorov to control for blurring caused during the denoising process.
Regarding claim 16, Pan in view of Egorov teach The method of Claim 14 (as described previously), wherein determining the loss comprises using a linear combination of a mean absolute error (L1) loss (Egorov: the loss function optimizer 204 can train CNN 103 using a combination of different loss functions like an L1 loss function (mean absolute error), an MS-SSIM (multi-scale structural similarity loss) [¶0063; Fig. 2]).
Egorov further teach that their neural network is directed to improve overall visual quality of CFM images prior to any downstream lossy image processing [¶0052]. Therefore, it would have been obvious to one of ordinary skill to combine the adaptive loss function weighting of Egorov to improve the loss optimization of Pan for enhanced image quality to arrive at the invention of the instant application. Egorov, however, fails to teach utilizing an inter-channel loss.
DeSmoke, however, is analogous art pertinent to the field of endeavor of image correction and disclose a method using an inter-channel loss function for removing smoke from laparoscopic images while maintaining original scene illumination and semantics. DeSmoke teach and an inter-channel loss (DeSmoke: an inter-channel (IC) loss is utilized to describe the differences between channels of a pixel in an image [Sec – Proposed Method, Subsec – Inter-channel (IC) Loss; Eq. 5-8).
DeSmoke further teach that the difference in pixel values across an inter-channel loss accounts for the level of blur for a given pixel region [Sec – Proposed Method, Subsec – Inter-channel (IC) Loss]. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to formulate a linear combination of the inter-channel loss taught by DeSmoke, in addition to the previously taught L1 and MS-SSIM loss of Pan in view of Egorov to control for blurring caused during the denoising process.
Claim(s) 5, 12 & 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pan et al (US 2021/0241429 A1), hereinafter referred to as “Pan”, in view of Egorov et al (US 2021/0390658 A1), hereinafter referred to as “Egorov”, further in view of Smolic et al (EP 3913572 A1), hereinafter referred to as “Smolic”.
Considering claim 5, Pan in view of Egorov teach The method of Claim 1 (as described previously), however they fail to explicitly teach using a a noisy training image with a low exposure value.
Smolic, per contra, is analogous art pertinent to the field of endeavor of the present application and disclose neural network training method utilizing one or more loss functions for image correction. Smolic teach wherein the noisy training image comprises an exposure value zero (EV0) image or a lower exposure value image (Smolic: a dark/noisy image is input into the encoder-decoder architecture [¶0041,45; Fig. 1], such as the low exposure image 2100 [¶0045-47; Fig. 2]).
Smolic further teach that their approach provides a simple yet effective method for correcting noise introduced in images obtained from over or underexposure lighting conditions [¶0028], and that the frequency loss function they implement reduced overall image noise for smoother edges and accurate color capture [¶0045-47]. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to utilize the low exposure images of Smolic to as training data for the denoising apparatus taught by Pan in view of Egorov to better enable to neural network to more effectively denoise images obtained under non-ideal lighting conditions.
With respect to claim 12, Pan in view of Egorov teach The electronic device of Claim 8 (as described previously), however they fail to explicitly teach using a a noisy training image with a low exposure value.
Smolic, on the other hand, is analogous art pertinent to the field of endeavor of the present application and disclose neural network training method utilizing one or more loss functions for image correction. Smolic teach wherein the noisy training image comprises an exposure value zero (EV0) image or a lower exposure value image (Smolic: a dark/noisy image is input into the encoder-decoder architecture [¶0041,45; Fig. 1], such as the low exposure image 2100 [¶0045-47; Fig. 2]).
Smolic further teach that their approach provides a simple yet effective method for correcting noise introduced in images obtained from over or underexposure lighting conditions [¶0028], and that the frequency loss function they implement reduced overall image noise for smoother edges and accurate color capture [¶0045-47]. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to utilize the low exposure images of Smolic to as training data for the denoising apparatus taught by Pan in view of Egorov to better enable to neural network to more effectively denoise images obtained under non-ideal lighting conditions.
Regarding claim 18, Pan in view of Egorov teach The method of Claim 14 (as described previously), however they fail to explicitly teach using a a noisy training image with a low exposure value.
Smolic, however, is analogous art pertinent to the field of endeavor of the present application and disclose neural network training method utilizing one or more loss functions for image correction. Smolic teach wherein the noisy training image comprises an exposure value zero (EV0) image or a lower exposure value image (Smolic: a dark/noisy image is input into the encoder-decoder architecture [¶0041,45; Fig. 1], such as the low exposure image 2100 [¶0045-47; Fig. 2]).
Smolic further teach that their approach provides a simple yet effective method for correcting noise introduced in images obtained from over or underexposure lighting conditions [¶0028], and that the frequency loss function they implement reduced overall image noise for smoother edges and accurate color capture [¶0045-47]. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to utilize the low exposure images of Smolic to as training data for the denoising apparatus taught by Pan in view of Egorov to better enable to neural network to more effectively denoise images obtained under non-ideal lighting conditions.
Claim(s) 6, 13 & 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pan et al (US 2021/0241429 A1), hereinafter referred to as “Pan”, in view of Egorov et al (US 2021/0390658 A1), hereinafter referred to as “Egorov”, further in view of Liu et al (“Joint Demosaicing and Denoising with Self Guidance”, 2020, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)), hereinafter referred to as “Liu”.
Regarding claim 6, Pan in view of Egorov teach The method of Claim 1 (as described previously), but they fail to teach a total variation loss between denoised images and channels of the ground truth image.
Liu, contrastingly, disclose a joint demosaicing and denoising neural network that leverages a total variation loss to enhance edge-aware smoothness. In this regard, Liu teach wherein determining the loss comprises using a total variation loss between the denoised images and a corresponding color channel for the ground truth image (Liu: a modified total variation loss (L) is calculated utilizing between a denoised output image and the green channel of the ground truth [Sec 3.4.2 – Edge-aware smoothness loss; Eq. 10]).
Liu additionally describe that total variation loss is implemented to smooth out noise and unexpected image artifacts, and their modified total variation loss prevents the over-smoothing of edge regions to maintain texture [Sec 3.4.2 – Edge-aware smoothness loss]. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the modified total variation loss utilized by Liu to provide a dynamic edge-ware smoothing function to the denoising neural network of Pan in view Egorov to effectively denoise images without a loss of texture and edges.
With respect to claim 13, Pan in view of Egorov teach The electronic device of Claim 8 (as described previously), but they fail to teach a total variation loss between denoised images and channels of the ground truth image.
Liu, contrastingly, disclose a joint demosaicing and denoising neural network that leverages a total variation loss to enhance edge-aware smoothness. In this regard, Liu teach wherein, to determine the loss, the at least one processing device is configured to use a total variation loss between the denoised images and a corresponding color channel for the ground truth image (Liu: a modified total variation loss (L) is calculated utilizing between a denoised output image and the green channel of the ground truth [Sec 3.4.2 – Edge-aware smoothness loss; Eq. 10]).
Liu additionally describe that total variation loss is implemented to smooth out noise and unexpected image artifacts, and their modified total variation loss prevents the over-smoothing of edge regions to maintain texture [Sec 3.4.2 – Edge-aware smoothness loss]. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the modified total variation loss utilized by Liu to provide a dynamic edge-ware smoothing function to the denoising neural network of Pan in view Egorov to effectively denoise images without a loss of texture and edges.
Considering claim 19, Pan in view of Egorov teach The method of Claim 14 (as described previously), but they fail to teach a total variation loss between denoised images and channels of the ground truth image.
Liu, contrastingly, disclose a joint demosaicing and denoising neural network that leverages a total variation loss to enhance edge-aware smoothness. In this regard, Liu teach wherein determining the loss comprises using a total variation loss between the denoised images and a corresponding color channel for the ground truth image (Liu: a modified total variation loss (L) is calculated utilizing between a denoised output image and the green channel of the ground truth [Sec 3.4.2 – Edge-aware smoothness loss; Eq. 10]).
Liu additionally describe that total variation loss is implemented to smooth out noise and unexpected image artifacts, and their modified total variation loss prevents the over-smoothing of edge regions to maintain texture [Sec 3.4.2 – Edge-aware smoothness loss]. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the modified total variation loss utilized by Liu to provide a dynamic edge-ware smoothing function to the denoising neural network of Pan in view Egorov to effectively denoise images without a loss of texture and edges.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Lee; Joohyun (US 2022/0309712 A1) teach a neural processing unit utilizing a loss function with adaptive weighting for image signal processing.
Akiyama et al (“Pseudo four-channel image denoising for noisy CFA raw Data”, 2015, IEEE) disclose an algorithm for denoising CRFA raw data based on principal component analysis.
Han et al (“Denoising and Motion Artifact Removal Using Deformable Kernel Prediction Neural Network for Color-Intensified CMOS”, 2021, MDPI-Sensors) describe using a deformable kernel prediction neural network to simultaneously denoise and remove motion artifacts.
Condat et al (“Joint Demosaicking and Denoising by Total Variation Minimization”, 2012, IEEE) teach minimizing a total variation across channels of a Bayer CFA image.
Song et al (“CC-Loss: Channel Correlation Loss for Image Classification”, 2020, IEEE) disclose an algorithm for accounting for differences across color channels using a channel attention module for more accurate image feature classification.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael M. Sofroniou whose telephone number is (571)272-0287. The examiner can normally be reached M-F: 8:30 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John M. Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL M SOFRONIOU/Examiner, Art Unit 2661
/JOHN VILLECCO/Supervisory Patent Examiner, Art Unit 2661
Prosecution Insights