DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) document submitted on January 27, 2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
1. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1,6,12 are rejected under 35 U.S.C. 103 as being unpatentable over Venkataraman (US Pub. No.: US 2011/0069189 A1), in view of Taketomi (US Pub. No.: US 2018/0270431 A1).
Regarding claim 1, Venkataraman discloses a system Para 34; camera array 100) comprising:
a memory (Para 129; computer readable storage medium) operable to store a plurality of threshold functions ( Para 41, 69; much of the pixel control logic is a single collection of functions common to all or most of the imagers with a smaller set of functions applicable each imager ) ;
a defective pixel detector ( Para 69; an image pixel correlation module 514; the image pixel correlation module 514 aligns portions of images captured by different imagers to compensate for the parallax. In one embodiment, the image pixel correlation module 514 compares the difference between the average values of neighboring pixels with a threshold and flags the potential presence of parallax when the difference exceeds the threshold.) coupled to the memory and operable to:
receive image pixels;
apply, for each image pixel received, a select threshold function of the
plurality of threshold functions to values of nearest-neighbor image pixels to obtain a threshold value ( Para 69; the image pixel correlation module 514 aligns portions of images captured by different imagers to compensate for the parallax. In one embodiment, the image pixel correlation module 514 compares the difference between the average values of neighboring pixels with a threshold and flags the potential presence of parallax when the difference exceeds the threshold. The threshold may change dynamically as a function of the operating conditions of the camera array. Further, the neighborhood calculations may also be adaptive and reflect the particular operating conditions of the selected imagers.); and
determine, for each image pixel received, whether the image pixel is defective
based on a comparison of a value of the image pixel to the threshold value (Para 69; the image pixel correlation module 514 aligns portions of images captured by different imagers to compensate for the parallax. In one embodiment, the image pixel correlation module 514 compares the difference between the average values of neighboring pixels with a threshold and flags the potential presence of parallax when the difference exceeds the threshold.);
and a statistics generator coupled to the defective pixel detector and configured to:
receive each image pixel that is determined to be defective (Para 70; The image is then processed by the parallax confirmation and measurement module 518 to detect and meter the parallax. In one embodiment, parallax detection is accomplished by a running pixel correlation monitor.).
However, Venkataraman does not disclose “determine a number of defective image pixels in a specified unit of image pixels and a location of each defective image pixel in the specified unit”.
Taketomi discloses a number of defective image pixels in a specified unit of image pixels and a location of each defective image pixel in the specified unit (Para 18-19; outputs a defective pixel flag as 1 for a defective pixel, and as 0 for a non-defective pixel. The number of the threshold comparison units 201 is the same as the number of simultaneously input pixels of the input image data, and each of the threshold comparison units 201 executes processing in parallel. The write control unit 204 obtains OR of bits of defective pixel flags output from the respective threshold comparison units 201, and executes AND operation of the result thereof and the detection enable signal, thereby generating a write enable signal for the defective pixel information storage unit 205. Fig.3B; para 19; A data structure of the defective pixel information stored in the defective pixel information storage unit 205 includes a Bayer Y coordinate, a Bayer X coordinate, a defective pixel flag as illustrated in FIG. 3B.).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Venkataraman with the teaching of Taketomi in order to flag the defective pixels and store location of them in order to perform defect correction and provide images with improved quality.
Regarding claim 6, the combination of Venkataraman and Taketomi teaches
the system of claim 1, wherein each image pixel received is associated with one of a
red color channel, a blue color channel, a green color channel, and an infrared color channel (Venkataraman; Para 76; Fig. 6A; In FIGS. 6A through 4E, "R" represents an imager having a red filter, "G" represents a imager having a green filter, "B" represents an imager having a blue filter).
Regarding claim 12, the combination of Venkataraman and Taketomi teaches The system of claim 1, further comprising a defective pixel correction circuit coupled
to the defective pixel detector ( Venkataraman; Para 74; The image processing pipeline module 420 may include a correction module for correcting abnormalities in images caused by a single pixel defect or a cluster of pixel defects. The correction module may be embodied on the same chip as the camera array, as a component separate from the camera array or as a part of the super-resolution module 526.) .
Claims 2,4 are rejected under 35 U.S.C. 103 as being unpatentable over Venkataraman (US Pub. No.: US 2011/0069189 A1), in view of Taketomi (US Pub. No.: US 2018/0270431 A1), and further in view of Nagata (US Pub. No.: US 2012/0212654 A1).
Regarding claim 2, the combination of Venkataraman and Taketomi does not teach wherein the defective pixel detector is operable to select the nearest-neighbor image pixels based on a color channel of the image pixel.
Nagata discloses the defective pixel detector is operable to select the nearest-neighbor image pixels based on a color channel of the image pixel ( Para 44; The signal of the defective pixel can be compensated by using a signal of a pixel that is disposed near the defective pixel and that has the same color in a direction different from the pupil division direction. The pixel that is disposed near the defective pixel (the peripheral pixel) means for example a pixel that is in an area A between two pixels (b-pixel of SB2 and a-pixel of SA3) that have the same color as the defective pixel in the microlens nearest to the microlens including the defective pixel in the pupil division direction of FIG. 7.).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Venkataraman with the teaching of Nagata to choose and utilize the neighbor image pixels that have the same color as the defective pixels for compensation so that image quality can be improved.
Regarding claim 4, the combination of Venkataraman and Taketomi does not teach the system of claim 2, wherein the defective pixel detector is operable to identify the nearest-neighbor image pixels using an offset pattern corresponding to the color channel of the image pixel and the nearest-neighbor image pixels.
Nagata discloses the defective pixel detector is operable to identify the nearest-neighbor image pixels using an offset pattern corresponding to the color channel of the image pixel and the nearest-neighbor image pixels.
( Para 44; the signal of the defective pixel can be compensated by using a signal of a pixel that is disposed near the defective pixel and that has the same color in a direction different from the pupil division direction.).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Venkataraman and Taketomi with the teaching of Nagata to choose neighbor image pixels that has the same color as the defective pixels in order to perform pixel correction and provide images with better quality.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Schaumburg et al. (US Pub. No.: US 2021/0056287 A1), in view of Meitav et al. (US Patent. No.: US 7,612,803 B2).
Regarding claim 13, Schaumburg et al. discloses a method (Para 2; identifying regions of interest in images ) comprising:
receiving image pixels of a frame ( Para 10; The method can include generating, from at least the first area of the biomedical image, a plurality of patches. Each patch of the plurality of patches can have a plurality of pixels) ; for each image pixel received:
identifying a color channel of the image pixel (Para 11; The one or more pixel types defined by the extraction policy specify that at least one of the plurality of pixels in the corresponding patch is to be within a range of color values to qualify for selection.);
identifying nearest-neighbor image pixels of the image pixel using a pattern, in
which the nearest-neighbor image pixels occupy an area defined by a number of rows
and a number of columns and the nearest-neighbor image pixels are of the color
channel of the image pixel ( Para 487-488; the image analyzer 1945 can take the column represented by the x-coordinate of the leftmost pixel of the nuclear region as the left boundary for the region, the column of the x-coordinate of the rightmost pixel of the nuclear region as the right boundary for the region, the row of the y-coordinate of the uppermost pixel of the nuclear region as the upper boundary for the region, and the row of the y-coordinate of the lowermost pixel of the nuclear region as the lower boundary for the region. Then using the bounding region, the image analyzer 1945 can identify a set of non-nuclear pixels that are along the boundary of the bounding box but are not classified as nuclear pixels. The set of non-nuclear pixels can then be grown (e.g., neighbors identified as candidate pixels, the candidate pixels are grown over if a predetermined number of pixels adjacent to the candidate pixel are non-nuclear pixels, etc.) to expand over any pixel in the bounding box that does not correspond to a nuclear pixel. The pixels within the bounding box that are not grown over by the non-nuclear pixels, but are also not nuclear pixels, can be considered gap pixels. The gap pixels can then be grown (e.g., neighbors identified as candidate pixels, the candidate pixels are grown over if a predetermined number of pixels adjacent to the candidate pixel are gap pixels, etc.) over non-nuclear pixels within the bounding box to form a gap region. A gap region can be a region within the nuclear region that does not contain nuclear pixels, but may be surrounded by nuclear pixels. The pixels of the gap region can be set to a predetermined color region, such that they may be easily identified as gap regions. Region withering can include identifying the classification of the eight neighboring pixels (e.g., up, left, down, right, upper right, upper left, lower left, and lower right), and counting the number of neighboring pixels that have a nuclear type classification.) ;
identifying at least one of a row or a column in the area that does not contain
the image pixel nor any of the nearest-neighbor image pixels (Para 487-488; Then using the bounding region, the image analyzer 1945 can identify a set of non-nuclear pixels that are along the boundary of the bounding box but are not classified as nuclear pixels.) .
However, Schaumburg et al does not disclose “storing, in memory, values of the image pixels of the area except image pixels of the identified row or column in the area that does not contain any of the nearest-neighbor image pixels”.
Meitav et al. discloses storing, in memory, values of the image pixels of the area except image pixels of the identified row or column in the area that does not contain any of the nearest-neighbor image pixels ( Col 10, lines 31-64; the compression operations of the present invention are performed in real-time, in a single pass, and are deterministic and predictable with regard to compression time, compressed output data rate and compressed data size, as image data are read from the sensor, these predefined time intervals are repeatable and can be quite short. Therefore, these additional images correlate well with the scene of interest. The support data are preferably compressed in the same manner as acquired image data in order to reduce the amount of the memory 45 required to store these data. Alternatively, the support data may be sub-sampled or averaged by the camera processor 17 before storage in the memory 45, in order to reduce support data storage requirements. To further reduce memory use, only predefined image regions, or image regions responsive to predefined criteria, can be stored as support data in the memory 45).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Schaumburg et al. with the teaching of Meitav et al. to only store the image region of pixels within certain range and disregard the image region outside of certain region (not including neighbor pixels) to reduce memory use and increase system efficiency.
Claims 14, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Schaumburg et al. (US Pub. No.: US 2021/0056287 A1), in view of Meitav et al. (US Patent. No.: US 7,612,803 B2) and in view of Venkataraman (US Pub. No.: US 2011/0069189 A1).
Regarding claim 14, the combination of Schaumburg and Meitav does not teach The method of claim 13, further comprising:
for each image pixel received: applying a threshold function to the values of the nearest-neighbor image pixels to obtain a threshold value, and comparing a value of the image pixel to the threshold value to determine whether the image pixel is defective.
Venkataraman discloses a threshold function to the values of the nearest-neighbor image pixels to obtain a threshold value (Para 69; the image pixel correlation module 514 aligns portions of images captured by different imagers to compensate for the parallax. In one embodiment, the image pixel correlation module 514 compares the difference between the average values of neighboring pixels with a threshold and flags the potential presence of parallax when the difference exceeds the threshold. The threshold may change dynamically as a function of the operating conditions of the camera array. Further, the neighborhood calculations may also be adaptive and reflect the particular operating conditions of the selected imagers), and comparing a value of the image pixel to the threshold value to determine whether the image pixel is defective (Para 69; the image pixel correlation module 514 aligns portions of images captured by different imagers to compensate for the parallax. In one embodiment, the image pixel correlation module 514 compares the difference between the average values of neighboring pixels with a threshold and flags the potential presence of parallax when the difference exceeds the threshold) .
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Schaumburg et al. with the teaching of Venkataraman to detect defective pixels efficiently in order to perform correction and provide images with better quality.
Regarding claim 20, the combination of Schaumburg, Meitav and Venkataraman teaches wherein the color channel is one among a red color channel, a blue color channel, a green color channel, and an infrared color channel (Venkataraman; Para 76; Fig. 6A; In FIGS. 6A through 4E, "R" represents an imager having a red filter, "G" represents a imager having a green filter, "B" represents an imager having a blue filter; It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Schaumburg et al. with the teaching of Venkataraman to include RGB filters for the pixels to provide images with colors.).
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Schaumburg et al. (US Pub. No.: US 2021/0056287 A1), in view of Meitav et al. (US Patent. No.: US 7,612,803 B2) and in view of Venkataraman (US Pub. No.: US 2011/0069189 A1) and Nagata (US Pub. No.: US 2012/0212654 A1).
Regarding claim 15, the combination of Schaumburg et al. , Meitav et al. and Venkataraman does not teach wherein the defective pixel detector is operable to select the nearest-neighbor image pixels based on a color channel of the image pixel.
Nagata discloses the defective pixel detector is operable to select the nearest-neighbor image pixels based on a color channel of the image pixel ( Para 44; The signal of the defective pixel can be compensated by using a signal of a pixel that is disposed near the defective pixel and that has the same color in a direction different from the pupil division direction. The pixel that is disposed near the defective pixel (the peripheral pixel) means for example a pixel that is in an area A between two pixels (b-pixel of SB2 and a-pixel of SA3) that have the same color as the defective pixel in the microlens nearest to the microlens including the defective pixel in the pupil division direction of FIG. 7.).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Schaumburg et al. , Meitav et al with the teaching of Nagata to choose and utilize the neighbor image pixels that have the same color as the defective pixels for compensation so that image quality can be improved.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Schaumburg et al. (US Pub. No.: US 2021/0056287 A1), in view of Meitav et al. (US Patent. No.: US 7,612,803 B2) and in view of Nagata (US Pub. No.: US 2012/0212654 A1).
Regarding claim 16, the combination of Schaumburg et al. , Meitav et al. does not teach wherein the pattern for identifying the nearest-neighbor image pixels of the image pixel is selected based on the color channel of the image pixel and the nearest-neighbor image pixels.
Nagata discloses wherein the pattern for identifying the nearest-neighbor
image pixels of the image pixel is selected based on the color channel of the image pixel and the nearest-neighbor image pixels ( Para 44; the signal of the defective pixel can be compensated by using a signal of a pixel that is disposed near the defective pixel and that has the same color in a direction different from the pupil division direction.).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Schaumburg et al. , Meitav et al with the teaching of Nagata to choose neighbor image pixels that has the same color as the defective pixels in order to perform pixel correction and provide images with better quality.
Allowable Subject Matter
Claims 3,5,7,8,9,10,11, 17,18,19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 3, prior art on record Venkataraman discloses the defective pixel detector ( Para 69; an image pixel correlation module 514; the image pixel correlation module 514 aligns portions of images captured by different imagers to compensate for the parallax. In one embodiment, the image pixel correlation module 514 compares the difference between the average values of neighboring pixels with a threshold and flags the potential presence of parallax when the difference exceeds the threshold).
However, Venkataraman does not disclose wherein the defective pixel detector is operable to select the threshold function based on the color channel of the image pixel.
Regarding claim 5, prior art on record Venkataraman discloses the defective pixel detector ( Para 69; an image pixel correlation module 514; the image pixel correlation module 514 aligns portions of images captured by different imagers to compensate for the parallax. In one embodiment, the image pixel correlation module 514 compares the difference between the average values of neighboring pixels with a threshold and flags the potential presence of parallax when the difference exceeds the threshold). However, the prior art does not disclose “wherein the defective pixel detector is operable to compute an average of the values of the nearest-neighbor image pixels, in which the threshold value is based on the average”.
Regarding claim 7, prior art on record Venkataraman discloses a threshold function (Para 69; the image pixel correlation module 514 aligns portions of images captured by different imagers to compensate for the parallax. In one embodiment, the image pixel correlation module 514 compares the difference between the average values of neighboring pixels with a threshold and flags the potential presence of parallax when the difference exceeds the threshold. The threshold may change dynamically as a function of the operating conditions of the camera array. Further, the neighborhood calculations may also be adaptive and reflect the particular operating conditions of the selected imagers).
However, none of the prior art discloses the plurality of threshold functions includes a first set of one or more threshold functions for image pixels of the red and the blue color channels, and a second set of one or more threshold functions for image pixels of the green and the infrared color channels.
Claim 8 is objected to as being dependent from claim 7.
Regarding claim 9, prior art on record Venkataraman discloses a memory (Para 129; computer readable storage medium).
However, prior art does not disclose the statistics generator includes buffer circuitry and frame counting circuitry.
Claims 10 and 11 are objected to as being dependent from claim 9.
Regarding claim 17, prior art on record Venkataraman discloses a threshold function (Para 69; the image pixel correlation module 514 aligns portions of images captured by different imagers to compensate for the parallax. In one embodiment, the image pixel correlation module 514 compares the difference between the average values of neighboring pixels with a threshold and flags the potential presence of parallax when the difference exceeds the threshold. The threshold may change dynamically as a function of the operating conditions of the camera array. Further, the neighborhood calculations may also be adaptive and reflect the particular operating conditions of the selected imagers).
However, the prior art does not disclose “wherein the threshold function applied to the values of the nearest-neighbor image pixels is selected from a plurality of threshold functions” in combination of other limitation in the claim.
Claim 18 is objected to as being dependent from claim 17.
Regarding claim 19, prior art on record Schaumburg et al .discloses the at least one of a row or a column in the area that does not contain the image pixel nor any of the neareast0neigobor image pixels ( Para 487-488; Then using the bounding region, the image analyzer 1945 can identify a set of non-nuclear pixels that are along the boundary of the bounding box but are not classified as nuclear pixels).
However, the prior art does not disclose “ at least one of a row or a column in the area that does not contain the image pixel nor any of the nearest-neighbor image pixels is based on the pattern and the color channel of the image pixel”.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XI WANG whose telephone number is 469-295-9155. The examiner can normally be reached on 9:00 am-5:00 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, SINH TRAN can be reached on 571-272-7564. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XI WANG/Primary Examiner, Art Unit 2637
Prosecution Insights