Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
Applicant requests consideration of references #14-20 of the information disclosure statement filed 08/03/2023, stating that copies of these references are of record in parent Application No. 17/147,067. To the extent the present application is entitled to rely on the earlier application under 37 CFR 1.98(d), copies of those references need not be resubmitted; references #14-20 have accordingly been considered and an initialed copy of the PTO-1449 is enclosed.
Response to Arguments
The 112(a) rejection is overcome.
The IDS submitted 8/3/23 has been considered.
Applicant's arguments (Remarks, pp. 8-9) directed to the amended independent claims have been fully considered but are not persuasive. Applicant argues that Ratnasingam does not disclose the use of a neural network for cluster-level bad pixel correction, but only pixel-level correction, relying on the 0.01% defective pixel figure of section 4.3 of Ratnasingam.
This argument is not persuasive because it attacks Ratnasingam individually. The rejection of the amended independent claims is based on the combination of Jerdev, Wang and Ratnasingam. Ratnasingam is not relied upon to teach cluster-level bad pixel correction. Jerdev teaches performing a cluster-level bad pixel correction operation based on first coordinate information about a plurality of clusters, each cluster including a plurality of first bad pixels (Jerdev, pars. 25, 30 and 32, where at least one neighboring pixel of the same color must also be defective in order for the pixels to be considered part of a defect cluster). Ratnasingam is relied upon only to teach that the bad pixel correction operation is performed using a neural network (Ratnasingam, Abstract and section 4.3: We trained our CNN to learn to identify and correct the response of the defective pixels).
Furthermore, the 0.01% value cited by applicant defines the density of defective pixels used in Ratnasingam's simulation and does not expressly require the defective pixels to be isolated or non-adjacent.
In the proposed combination, Jerdev's first coordinate information identifying the defective-pixel clusters is retained for identifying the cluster pixels to be corrected, while Ratnasingam's neural-network-based defect pixel correction is substituted for Jerdev's conventional correction technique to perform the correction of the identified defective pixel data. The modification thus performs the cluster-level bad pixel correction operation using a neural network based on Jerdev's first coordinate information, without eliminating Jerdev's use of the defect cluster coordinate information to identify the defective pixels subject to correction.
Please see the rejection below.
Claim Rejections - 35 USC 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 21-23, 27, 29, 30, 33, 34, 36 and 38 are rejected under 35 U.S.C. 103 as being unpatentable over Jerdev (US 2007/0030365) in view of Wang et al. (US 2020/0336684) and further in view of Ratnasingam (Deep Camera: A Fully Convolutional Neural Network for Image Signal Processing).
Regarding claim 21, Jerdev teaches an image processing apparatus (apparatus that allows for the correction of defective pixel clusters in an imaging device, par. 0008) comprising: an image sensor comprising a plurality of pixels (FIGS. 2A-2B, pixel arrays 100, 110, par. 0025) and a color filter array stacked on the plurality of pixels, the color filter array having a first pattern (a Bayer pattern color filter array 50, FIG. 1, par. 0026), the image sensor being configured to generate first image data by using the plurality of pixels (the sample-and-hold circuit 261 and analog-to-digital converter 275 digitize the pixel signals supplied to the image processor 280, pars. 0036 and 0032),
wherein the image processing system is configured to generate the second image data by performing a cluster-level bad pixel correction operation on the first image data, based on first coordinate information about a plurality of clusters, each cluster including a plurality of first bad pixels (at an initial step 201 the defective pixels 32a, 32b are located; at step 202 the selection kernel 101a for a cluster-labeled defective pixel 32 is selected and the symmetrically located pixels are evaluated; at steps 204-205 the average value A is substituted for the defective pixel P32a; and in order for pixels 32a, 32b to be considered part of a defect cluster, at least one neighboring pixel of the same color must also be identified as defective, pars. 0025, 0030 and 0032);
Jerdev fails to explicitly disclose a first processor configured to perform a post-processing operation on second image data.
However, Wang teaches a first processor configured to perform a post-processing operation on second image data (image data is processed by a remosaicing system to produce a second set of image data, which is mapped to a Bayer CFA pattern 240-a to produce a third set of image data that is processed by an ISP to produce a final RGB image data set, par. 0050).
It would have been obvious prior to the effective filing date of the claimed invention to one of ordinary skill in the art to include in Jerdev the post-processing operation as taught by Wang in order to obtain higher pixel correction quality, as recognized by Wang.
Jerdev in view of Wang fails to explicitly disclose performing the cluster-level bad pixel correction operation using a neural network.
Ratnasingam teaches performing a bad pixel correction operation using a neural network (a fully convolutional neural network (CNN) is presented to perform defect pixel correction, Abstract; We trained our CNN to learn to identify and correct the response of the defective pixels ... our CNN-based ISP pipeline can effectively perform defect pixel correction, section 4.3).
It would have been obvious prior to the effective filing date of the claimed invention to one of ordinary skill in the art to perform the cluster-level bad pixel correction operation of Jerdev using a neural network as taught by Ratnasingam in order to improve image reconstruction quality by reducing the residual error that accumulates in a conventional sequential image signal processing pipeline, as recognized by Ratnasingam (Abstract).
Regarding claim 22, in the proposed combination Jerdev's image processor 280 is retained as the processor performing the cluster-level bad pixel correction (Jerdev, pars. 0032 and 0036), modified to perform the correction using Ratnasingam's neural-network technique (Ratnasingam, section 4.3), while Wang's downstream ISP is provided as a separate processor performing the post-processing operation (Wang, par. 0050). The correction processor is therefore a second processor distinct from the first processor performing the post-processing operation.
Regarding claim 23, Jerdev teaches selecting a plurality of pieces of region of interest (ROI) data based on the first coordinate information, detecting pieces of pixel data corresponding to the plurality of clusters, and performing the cluster-level bad pixel correction operation based on the detected pieces of pixel data (pars. 0030 and 0032). Under a broadest reasonable interpretation, Jerdev's selected kernel regions centered at the identified cluster-defective pixel locations constitute pieces of region-of-interest data, because each is a selected portion of the first image data associated with a defective-pixel location and selected for cluster correction (selection kernel 101a, par. 0030).
Regarding claim 27, see Wang pars. 0050 and 0042: the first processor performs a remosaic operation on the second image data to generate image data corresponding to a second pattern, and performs a demosaic operation on the generated image data to generate third image data that is full-color image data.
Regarding claim 29, Jerdev teaches an image processing apparatus comprising: an image sensor comprising a plurality of pixels and a color filter array stacked on the plurality of pixels, the color filter array having a predetermined pattern, the image sensor being configured to generate first image data by using the plurality of pixels (FIGS. 2A-2B, pars. 0025, 0026 and 0036);
a first processor configured to perform a cluster-level bad pixel correction operation on the first image data, based on the first coordinate information, wherein the first coordinate information comprises coordinate information about a plurality of clusters formed by the plurality of first bad pixels (steps 201-205; selection kernel 101a for a cluster-labeled defective pixel 32; at least one neighboring pixel of the same color must also be identified as defective, pars. 0025, 0030 and 0032);
Jerdev fails to explicitly disclose a non-volatile memory configured to store first coordinate information about a plurality of first bad pixels. However, Wang teaches a non-volatile memory configured to store the first coordinate information (database 130 stores configuration information for pattern-configurable pixel correction, and device 115 refers to one or more configurable tables defining pixel locations, pars. 0045, 0057 and 0036), the first coordinate information being read from the non-volatile memory (device 115 retrieves the stored data from database 130 to perform the pixel correction, pars. 0058 and 0045).
It would have been obvious prior to the effective filing date of the claimed invention to one of ordinary skill in the art to include in Jerdev the non-volatile memory as taught by Wang in order to obtain higher pixel correction quality, as recognized by Wang.
Jerdev in view of Wang fails to explicitly disclose performing the cluster-level bad pixel correction operation using a neural network. However, Ratnasingam teaches performing a bad pixel correction operation using a neural network (Abstract and section 4.3).
It would have been obvious prior to the effective filing date of the claimed invention to one of ordinary skill in the art to perform the cluster-level bad pixel correction operation of Jerdev using a neural network as taught by Ratnasingam in order to improve image reconstruction quality by reducing the residual error that accumulates in a conventional sequential image signal processing pipeline, as recognized by Ratnasingam (Abstract). In the proposed combination, Jerdev's first coordinate information is retained to identify the defective-pixel clusters to be corrected, while Ratnasingam's neural network performs the correction of the identified defective pixel data.
Regarding claim 30, see Jerdev par. 0038: the imaging device may be combined with a processor on a single integrated circuit that includes the image sensor.
Regarding claim 33, see Ratnasingam Abstract and section 3.1: the neural network comprises a fully convolutional network.
Regarding claim 34, in the proposed combination Wang's ISP is provided as a second processor distinct from the first processor and is configured to generate third image data by performing a post-processing operation on second image data that comprises the first image data corrected by the first processor (Wang, pars. 0045 and 0050).
Regarding claim 36, Jerdev teaches a method of operating an image processing apparatus, wherein the image processing apparatus comprises an image sensor, a first processor and a second processor, and wherein the image sensor comprises a plurality of pixels and a color filter array stacked on the plurality of pixels, the color filter array having a predetermined pattern (methods of correcting pixel defects in a solid state imager device, par. 0001; FIGS. 2A-2B, pars. 0025 and 0026), the method comprising: generating first image data using the plurality of pixels (par. 0036);
performing a cluster-level bad pixel correction operation on the first image data, based on first coordinate information about a plurality of first bad pixels from among the plurality of pixels, wherein the cluster-level bad pixel correction operation is performed by the first processor (image processor 280; steps 201-205; at least one neighboring pixel of the same color must also be defective, pars. 0025, 0030, 0032 and 0036);
Jerdev fails to explicitly disclose performing a post-processing operation on second image data output by the first processor as a result of the cluster-level bad pixel correction operation, wherein the post-processing operation is performed by the second processor. However, Wang teaches performing a post-processing operation on second image data output by the first processor, wherein the post-processing operation is performed by a second processor (device 115 applies a configuration file to an ISP; the remosaiced second set of image data is processed to produce a final RGB image data set, pars. 0045 and 0050).
It would have been obvious prior to the effective filing date of the claimed invention to one of ordinary skill in the art to include in Jerdev the post-processing operation performed by a second processor as taught by Wang in order to obtain higher pixel correction quality, as recognized by Wang.
Jerdev in view of Wang fails to explicitly disclose that the cluster-level bad pixel correction operation is performed using a neural network. However, Ratnasingam teaches performing a bad pixel correction operation using a neural network (Abstract and section 4.3).
It would have been obvious prior to the effective filing date of the claimed invention to one of ordinary skill in the art to perform the cluster-level bad pixel correction operation of Jerdev using a neural network as taught by Ratnasingam in order to improve image reconstruction quality by reducing the residual error that accumulates in a conventional sequential image signal processing pipeline, as recognized by Ratnasingam (Abstract). In the proposed combination, Jerdev's first coordinate information is retained to identify the defective-pixel clusters to be corrected, while Ratnasingam's neural network performs the correction of the identified defective pixel data.
Regarding claim 38, Jerdev teaches selecting a plurality of pieces of ROI data based on the first coordinate information, detecting pieces of pixel data corresponding to the plurality of clusters from the plurality of pieces of ROI data, and correcting the detected pieces of pixel data (pars. 0030 and 0032). As set forth for claim 23 above, under a broadest reasonable interpretation Jerdev's selected kernel regions centered at the identified cluster-defective pixel locations constitute pieces of region-of-interest data.
Claim(s) 24, 28, 32, 39 and 41 are rejected under 35 U.S.C. 103 as being unpatentable over Jerdev in view of Wang and Ratnasingam, as applied above, and further in view of Meisenzahl et al. (US 2003/0179418).
Regarding claim 24, Meisenzahl teaches the image processing system configured to further perform a pixel-level bad pixel correction operation on the first image data to generate the second image data, based on second coordinate information about a plurality of second bad pixels which are isolated from each other (a defective pixel with corrupted data 11 is classified as a single defective pixel, as distinguished from a defective cluster in which more than one defective pixel touches an adjacent defective pixel horizontally, vertically or diagonally, FIG. 1; the defective pixel address is recorded in memory as an x,y address and the defective pixel value is replaced with the average of two correction pixels identified by offsets, FIGS. 5-6).
It would have been obvious prior to the effective filing date of the claimed invention to one of ordinary skill in the art to include in Jerdev, Wang and Ratnasingam the pixel-level correction of isolated single defective pixels based on their stored coordinate information as taught by Meisenzahl in order to accurately correct isolated defective pixels in addition to defective clusters, thereby improving image quality, as recognized by Meisenzahl.
Regarding claim 28, Meisenzahl teaches performing the cluster-level bad pixel correction operation on clusters of an arbitrary shape. Meisenzahl discloses that defective cluster pixels appear in different shapes and sizes, and defines a defective cluster as more than one defective pixel touching another adjacent defective pixel horizontally, vertically, or diagonally, such that under a broadest reasonable interpretation the clusters may take any shape formed by adjacent defective pixels (FIG. 1).
Regarding claim 32, Meisenzahl teaches the non-volatile memory configured to further store second coordinate information about a plurality of second bad pixels which are isolated from each other, and the first processor configured to further perform a pixel-level bad pixel correction operation on the first image data based on the second coordinate information (the defect map storing the x,y address of each single defective pixel, and correction by replacing the value with the average of stored correction pixels, FIGS. 5-6); see also claim 24 above.
Regarding claim 39, Meisenzahl teaches performing a pixel-level bad pixel correction operation on the first image data based on second coordinate information characterizing a plurality of second bad pixels which are isolated from each other (single defective pixels stored by x,y address and corrected, FIGS. 1 and 5-6); see also claim 24 above.
Regarding claim 41, in the proposed combination the second processor performing the cluster-level bad pixel correction operation (Jerdev image processor 280 modified per Ratnasingam, section 4.3) is distinct from the first processor performing the pixel-level bad pixel correction operation on the first image data (Meisenzahl, correction of isolated single defective pixels based on stored coordinate information, FIGS. 5-6); see also claim 24 above.
Claim 25 is rejected under 35 U.S.C. 103 as being unpatentable over Jerdev in view of Wang, Ratnasingam and Meisenzahl, as applied above, and further in view of Smirnov et al. (US 2020/0302582 A1, hereinafter Smirnov).
Regarding claim 25, Meisenzahl teaches generating third image data by performing a pixel-level bad pixel correction operation on the first image data based on second coordinate information about a plurality of second bad pixels which are isolated from each other (correction of isolated single defective pixels, FIGS. 1 and 5-6). Neither Jerdev, Wang nor Meisenzahl explicitly discloses the first processor merging the third image data with the second image data and performing the post-processing operation on merged image data. However, Smirnov teaches a first processor configured to merge third image data with second image data (image fusion processing circuit 444 performs per-pixel blending to generate an unscaled single-color version of a fused image 446, par. 0069) and to perform the post-processing operation on merged image data (post-processing circuit 450 performs post-processing to obtain a post-processed fused image 472, par. 0070).
It would have been obvious prior to the effective filing date of the claimed invention to one of ordinary skill in the art to merge the separately corrected outputs as taught by Smirnov so that corrected pixel information from the pixel-level correction path and corrected pixel information from the cluster-level correction path are incorporated into a common image before downstream processing, thereby preserving the corrections directed to the respective defect types in a single output image.
Claim(s) 35 and 40 are rejected under 35 U.S.C. 103 as being unpatentable over Jerdev in view of Wang and Ratnasingam, as applied above, and further in view of Darshan (US 2016/0284314, ).
Regarding claim 35, see Darshan par. 0022: the first and second image data corresponds to tetra data (raw image data 111 in an up to 4x4 lattice with RGBC, where C may be any other color); and Wang par. 0050: the third image data corresponds to RGB data.
It would have been obvious prior to the effective filing date of the claimed invention to one of ordinary skill in the art to include in Jerdev, Wang and Ratnasingam the tetra data as taught by Darshan in order to improve image quality such as fewer artifacts, as recognized by Darshan.
Regarding claim 40, see Darshan par. 0034: the performing of the post-processing operation comprises performing at least one of a denoising operation and a sharpening operation on the second image data (the luma-chroma separated image processing module 109 may perform denoising and sharpening); see also Wang par. 0050 (remosaic) and par. 0042 (demosaic).
It would have been obvious prior to the effective filing date of the claimed invention to one of ordinary skill in the art to include in Jerdev, Wang and Ratnasingam the denoising and sharpening as taught by Darshan in order to improve image quality, as recognized by Darshan.
Claim 26 is rejected under 35 U.S.C. 103 as being unpatentable over Jerdev in view of Wang and Ratnasingam, as applied above, and further in view of Campbell (10,666,884).
Regarding claim 26, neither Jerdev, Wang nor Ratnasingam explicitly discloses that the second processor is configured to generate the first coordinate information based on masking information indicating positions of the plurality of first bad pixels.
Campbell teaches generating coordinate information based on masking information indicating positions of a plurality of bad pixels. Campbell generates defective-pixel maps into which defective pixel sensors are entered according to their positions, and combines them into a combined defective-pixel map (the masking information indicating positions of the bad pixels); a processor then applies a clustering algorithm to the combined defective-pixel map to generate a defect list containing boundary coordinates for each identified defect (FIGS. 5-6, items 142 and 144 and col. 9 lines 54 to col. 10 line 20).
It would have been obvious prior to the effective filing date of the claimed invention to one of ordinary skill in the art to generate the first coordinate information from masking information as taught by Campbell in order to automatically produce accurate defective-pixel coordinate information from the defective-pixel map, as recognized by Campbell.
Claim 42 is rejected under 35 U.S.C. 103 as being unpatentable over Jerdev in view of Wang and Ratnasingam, as applied above, and further in view of Aleksic (2012/0019693).
Regarding claim 42, neither Jerdev, Wang nor Ratnasingam explicitly discloses that the image processing system is configured to periodically or dynamically update the first coordinate information. However, Aleksic teaches an image processing system configured to dynamically update stored coordinate information of bad pixels. Aleksic's dynamic bad pixel detection component 804 identifies bad pixels during operation and updates the stored bad pixel information 826 in a re-writable, non-volatile storage component, where each stored entry comprises a coordinate data field 902 providing x-y coordinates identifying the location of a corresponding bad pixel (FIGS. 8-9).
It would have been obvious prior to the effective filing date of the claimed invention to one of ordinary skill in the art to include in Jerdev, Wang and Ratnasingam the dynamic updating of the stored bad-pixel coordinate information as taught by Aleksic in order to account for bad pixels that arise during operation of the image sensor, as recognized by Aleksic.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: Tajbakhsh (Efficient defect pixel cluster detection and correction for Bayer CFA image sequences) teaches table-based detection and correction of defect pixel clusters using stored defect-pixel coordinate information.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HADI AKHAVANNIK whose telephone number is (571)272-8622. The examiner can normally be reached 9 AM - 5 PM Monday to Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HADI AKHAVANNIK/Primary Examiner, Art Unit 2676
Prosecution Insights