DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 11-14 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over US 12,488,436 B2 to Marras et al (hereinafter ‘Marras’) in view of US 2021/0390375 A1 to Luo et al (hereinafter ‘Luo’).
Regarding claim 1, Marras discloses a method performed by at least one processor (column 5, lines 6-7, an apparatus for denoising an image), the method comprising: collecting a first set of images of a scene with a sensor in accordance with a first condition (column 12, lines 62-66, wherein at step 801, the method comprises receiving a plurality of input images, each captured by an image sensor, receiving a plurality of noise signatures; receiving at least one noise statistic for the image sensor(s) that captured the plurality of input images); collecting a second set of images of the scene with the sensor in accordance with a second condition (column 5, lines 6-7, and column 10, lines 64-67, wherein more than one E can be learned targeting different camera sensors, as well as different image ISO group values. The reg pathway implicitly estimates the latent clean image with the operations in the hidden layers); collecting one or more noise sample sets based on the first set of images and the second set of images (Column 12, lines 64-65 and Fig. 8, step 802, wherein receiving a plurality of noise signatures, as collecting noise sample from each image); generating a calibrated noise model based on the one or more noise sample sets (column 9, line 43-47 and Fig. 3b, wherein the network therefore receives a plurality of training input images 304, each captured by an image sensor, a plurality of noise signatures 306, as the noise model, and the noise statistic for the image sensor(s) that captured the plurality of input images, e.g. 310. For each training image, a noise signature is applied to the image to form the noisy input image 305); and generating a noisy image by applying the calibrated noise model to a noise free image (column 9, lines 29-32, wherein the noise signatures v, 306, as the noise model, may be generated by a noise model and applied to the respective ground truth input image y, 304, as the noise free image, to give the noisy input image s, 305.). Marras does not specifically disclose calibrated noise model however Marras discloses in column 5, lines 31-35, wherein the at least one noise statistic may comprise the noise variance of the image sensor. The noise variance projection layer of the network may explicitly exploit the noise standard deviation information to correct the statistics of the predicted noise initially estimated by a generator, as the initial calibration. Additionally, Luo discloses (Para [0052] and Fig. 2, wherein the sensors 204 of the mobile device 202 are calibrated using a calibration process 210. The calibration process 210 is performed to obtain key calibration parameters for each sensor 204, including the sensor's intrinsic and extrinsic parameters 215, lens model parameters 216 (such as lens distortion coefficients), noise model parameters 217 (such as sensor noise distribution parameters)). Therefore, before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the initial calibration of the generator in order to correct the statistics of the predicted noise (column 5, lines 34-35).
Regarding claim 2, in the combination of Marras and Luo, Marras discloses the method further comprising: inputting the noisy image into a machine learning model to generate an estimated noise free image (column 19, lines 33-35, and Fig. 3(b), wherein the noisy RGB input images, 305, is input into the network, and the network regresses a denoised output image, 307.); and updating the machine learning model based on comparing the estimated noise free image and the noise free image using a loss function (column 10, lines 5-9, and Fig. 3b, wherein (the difference between the regressed output image 307 and the GT image 304 forms an error, which is then back-propagated through the network from the output to the input though gradients. The weights of the network are then updated to reduce the error).
Regarding claim 3, in the combination of Marras and Luo, Marras discloses wherein the machine learning model is a neural image signal processor (ISP) network (column 8, lines 37-41, wherein both the reg pathway 301 and AE pathway 302 are based on deep learning and may, for example, apply a CNN to process the input image. A CNN learns a collection of filters, which are applied to the image through convolution.).
Regarding claim 4, in the combination of Marras and Luo, Marras discloses wherein the collecting the first set of images in accordance with the first condition further comprises: collecting a first burst of raw images of the scene with the sensor at a first International Organization Standardization (ISO) value, and wherein the collecting the second set of images in accordance with the second condition further comprises: collecting a second burst of raw images of the scene with the sensor at a second ISO value higher than the first ISO value (column 15, lines 7-12, wherein the network cannot explicitly understand that in images with small ISO values the estimated noise standard deviation should be significant less than in the case of high ISO values. Because of that, in many cases the denoiser destroys the image high frequency details especially for low ISO images, wherein the system inherently capable of collecting images of different ISO).
Regarding claims 9, in the combination of Marras and Luo, Luo discloses wherein the scene is a calibration chart comprising a plurality of exposure values (Para [0054], wherein FIG. 3 illustrates additional details of an example photometric calibration 211 according to this disclosure. As shown in FIG. 3, a color chart 302 represents a real image of a color checkerboard generated by the mobile device 202 under predetermined lighting conditions and lens settings (such as ISO-50, f/2.4, 1/2000s, 9000 lux). Another color chart 304 represents a synthetically-generated image of the same color checkerboard for the same lighting conditions and lens settings).
Regarding claim 11, in the combination of Marras and Luo, Marras discloses an apparatus (column 5, lines 6-7, an apparatus for denoising an image) comprising: a memory storing one or more instructions; at least one processor operatively coupled to the memory and configured to execute one or more instructions stored in the memory, wherein the one or more instructions, when executed by the at least one processor (column 13, lines 30-34, wherein a camera 901 is connected to a communications network. Camera 901 comprises an image sensor 902. The camera also comprises a memory 903, a processor 904 and a transceiver 905. The memory stores in non-transient form code that can be run by the processor 904), cause the at least one processor to: Please refer to the corresponding method claim 1 above for further teachings.
Regarding claims 12-14 and 19, please refer to the corresponding method claims 2-4 and 9, respectively above.
Regarding claim 20, Marras discloses a non-transitory computer readable medium having instructions stored therein, which when executed by a processor cause the processor to execute a method (column 13, lines 33-34, wherein the memory stores in non-transient form code that can be run by the processor 904) comprising: Please refer to the corresponding method claim 1 above for further teachings.
Claims 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Marras in view of US 11,995,800 B2 to Zhu et al (hereinafter ‘Zhu’).
Regarding claims 5 and 15, Marras does not specifically disclose wherein the scene comprises a plurality of intensity values, and wherein the collecting the first set of images in accordance with the first condition further comprises: generating a ground-truth image by averaging each image in the first burst of raw images at each intensity value. Zhu discloses averaging each image in the first burst of raw images at each intensity value (column 10, lines 6-12, wherein an output image corresponding to a respective input image in the training set may be obtained by producing multiple captures of the input image (e.g., at the same and/or a similar ISO setting used to capture the input set of images) and subsequently processing the set of input images, such as by averaging the intensities for each pixel across the multiple captures). Marras and Zhu are combinable because they both disclose sensor image noise elimination. Therefore, before the effective filing data of the claimed invention, it would have been obvious to combine the generating a ground-truth image by averaging each image in the first burst of raw images at each intensity value, of Zhu’s method/apparatus with Marras’ in order to keep the thermal noise properties of the output consistent with that of the input frame (column 10, lines 19-25).
Allowable Subject Matter
Claims 6-8, 10 and 16-18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The following is a statement of reasons for the indication of allowable subject matter: The prior art or the prior art of record specifically, Marras, Zhu and Luo, does not disclose:
. . . . wherein the collecting the one or more noise sample sets based on the first set of images and the second set of images further comprises: for each intensity level in the ground-truth image, collecting corresponding pixels from the second burst of images to form a noise sample set per intensity level, of claims 6 and 16 combined with other features and elements of the claims;
Claims 7-8, and 17-18 depend from an allowable base claim and are thus allowable themselves;
. . . . wherein the generating the calibrated noise model based on the one or more noise sample sets further comprises: measuring a variance of each noise sample set; fitting a normal distribution to each noise sample set based on the variance to generate the calibrated noise model, of claim 10 combined with other features and elements of the claim.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHERVIN K NAKHJAVAN whose telephone number is (571)272-5731. The examiner can normally be reached Monday-Friday 9:00-05:00 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sue Lefkowitz can be reached at (571)272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHERVIN K NAKHJAVAN/ Primary Examiner, Art Unit 2672
Prosecution Insights