DETAILED ACTION
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-20 are rejected under 35 U.S.C. 102 (a)(1) as being unpatentable as being anticipated by Cote (US 20150296193 A1).
Re Claim 1, COTE discloses an image sensor (see COTE: e.g., --a digital camera, to acquire image data to be processed in the image processing circuitry 32. The input structures 14 may enable user input to the electronic device, and may include hardware keys, a touch-sensitive element of the display 28, and/or a microphone. [0224] The processor(s) 16 may control the general operation of the device 10. For instance, the processor(s) 16 may execute an operating system, programs, user and application interfaces, and other functions of the electronic device 10.--, in [0223]-[0224], and [0227]-[0229]) comprising:
a pixel array including a plurality of pixels in a plurality of rows and a plurality of columns, the pixel array configured to generate a raw image (see COTE: e.g., --[0227] The imaging device(s) 30 of the electronic device 10 may represent a digital camera that may acquire both still images and video. Each imaging device 30 may include a lens and an image sensor capture and convert light into electrical signals. By way of example, the image sensor may include a CMOS image sensor (e.g., a CMOS active-pixel sensor (APS)) or a CCD (charge-coupled device) sensor. Generally, the image sensor of the imaging device 30 includes an integrated circuit with an array of photodetectors. The array of photodetectors may detect the intensity of light captured at specific locations on the sensor. Photodetectors are generally only able to capture intensity, however, and may not detect the particular wavelength of the captured light.
[0228] Accordingly, the image sensor may include a color filter array (CFA) that may overlay the pixel array of the image sensor to capture color information. The color filter array may include an array of small color filters, each of which may overlap a respective location—namely, a picture element, or pixel—of the image sensor and filter the captured light by wavelength. Thus, together, the color filter array and the photodetectors may detect both the wavelength and intensity of light through the lens. The resulting image information may represent a frame of raw image data.--, in [0227]-[0229]);
an image signal processor (ISP) configured to process the raw image and generate an output image (see COTE: e.g., --[0229] The color filter array may be a Bayer color filter array, an example of which appears in FIG. 2. A Bayer color filter array provides a filter pattern that captures 50% green elements, 25% red elements, and 25% blue elements of light reaching the sensor. In the example of FIG. 2, 2 green elements (Gr and Gb), 1 red element (R), and 1 blue element (B) will repeat in the pattern shown across the full pixel array of the sensor(s) of the imaging device(s) 30. Thus, an image sensor with a Bayer color filter array may provide information regarding the intensity of the light received by the imaging device 30 at the green, red, and blue wavelengths, whereby each image pixel records only one of the three colors (RGB). This information, which may be referred to as “raw image data” or data in the “raw domain,” may be processed using one or more demosaicing techniques to convert the raw image data into a full color image, generally by interpolating a set of red, green, and blue values for each pixel. As will be discussed further below, such demosaicing techniques may be performed by the image processing circuitry 32.--, in [0229]-[0230]; also see: --[0019] FIG. 8 is a block diagram of an example of the image processing circuitry of FIG. 1, including statistics logic, a raw-format processing block, an RGB-format processing block, and a YCC-format processing block, in accordance with an embodiment;
[0020] FIG. 9 is flowchart depicting a method for processing image data in the ISP pipe processing logic 80 logic of FIG. 10, in accordance with an embodiment;
[0021] FIG. 10 is block diagram illustrating a configuration of double buffered registers and control registers that may be used for processing image data in the ISP pipe processing logic 80 logic, in accordance with an embodiment;--, in [0019]-[0021], and, --[0734] As also depicted in FIG. 122, one effect of the binning process is that the spatial sampling of the binned pixels may not be equally spaced. This spatial distortion may, in some systems, result in aliasing (e.g., jagged edges), which is generally not desirable. Further, because certain image processing steps in the ISP pipe logic 82 may depend upon on the linear placement of the color samples in order to operate correctly, the raw scaler logic 1040 may be applied to perform re-sampling and re-positioning of the binned pixels such that the binned pixels are spatially evenly distributed.--, in [0733]-[0734]);
wherein the ISP is configured to:
obtain the raw image;
generate RGB image data based on the raw image (see COTE: e.g., --[0253] The ISP pipe processing logic 80 may also include several image processing blocks, some of which may operate in parallel with the statistics logic 140a and 140b. For example, a raw block 150 (also referred to as RAWProc or DMA destination D4) also may receive one of several possible raw image data signals via selection logic 152 and may process the raw image data using raw image processing logic 154. The raw image processing logic 154 may perform several raw image data processing operations, including sensor linearization (SLIN), black level compensation (BLC), fixed pattern noise reduction (FPNR), temporal filtering (TF), defective pixel correction (DPC), collection of additional noise statistics (NS), spatial noise filtering (SNF), lens shading correction (LSC), white balance gain (WBG), highlight recovery (HR), and/or raw scaling (RSCL).
[0254] The output of the raw block 150 may be stored in the memory 100 or continue to an RGB-format processing block 160 (also referred to as RgbProc or DMA destination D5). The RGB block 160 may receive one of two image data signals via selection logic 162, which may be processed by RGB image processing logic 164. The RGB image processing logic 164 may perform several image data processing operations, including demosaicing (DEM) to obtain RGB-format image data from raw image data. Having obtained RGB-format image data, the RGB image processing logic 164 may perform local tone mapping (LTM); color correction using a color correction matrix (CCM); color correction using a three-dimensional color lookup table (CLUT); gamma/degamma (GAM); gain, offset, and clipping (GOC); and/or color space conversion (CSC), producing image data in a YCC format (e.g., YCbCr or YUV).--, in [0253]-[0254]; also see: --[0229] The color filter array may be a Bayer color filter array, an example of which appears in FIG. 2. A Bayer color filter array provides a filter pattern that captures 50% green elements, 25% red elements, and 25% blue elements of light reaching the sensor. In the example of FIG. 2, 2 green elements (Gr and Gb), 1 red element (R), and 1 blue element (B) will repeat in the pattern shown across the full pixel array of the sensor(s) of the imaging device(s) 30. Thus, an image sensor with a Bayer color filter array may provide information regarding the intensity of the light received by the imaging device 30 at the green, red, and blue wavelengths, whereby each image pixel records only one of the three colors (RGB). This information, which may be referred to as “raw image data” or data in the “raw domain,” may be processed using one or more demosaicing techniques to convert the raw image data into a full color image, generally by interpolating a set of red, green, and blue values for each pixel. As will be discussed further below, such demosaicing techniques may be performed by the image processing circuitry 32.--, in [0229]-[0230]; also see: --[0019] FIG. 8 is a block diagram of an example of the image processing circuitry of FIG. 1, including statistics logic, a raw-format processing block, an RGB-format processing block, and a YCC-format processing block, in accordance with an embodiment;
[0020] FIG. 9 is flowchart depicting a method for processing image data in the ISP pipe processing logic 80 logic of FIG. 10, in accordance with an embodiment;
[0021] FIG. 10 is block diagram illustrating a configuration of double buffered registers and control registers that may be used for processing image data in the ISP pipe processing logic 80 logic, in accordance with an embodiment;--, in [0019]-[0021], and, --[0034] FIGS. 29-34 show examples of memory formats for full-color RGB image data that may be supported by the image processing circuitry of FIG. 7 or FIG. 8,--, in [0034]);
generate corrected RGB image data by applying a selected image data processing method to the RGB image data (see COTE: e.g., --[0213] FIG. 227 is graph of noise statistics as represented by a plot of standard deviations for portions of image data versus pixel intensity values, in accordance with an embodiment; [0214] FIG. 228 is an example image that has been corrected for geometric distortion--, in [0213]-[0214]; also see: --[0363] As may be appreciated, the image sensor(s) 90 may not always perfectly capture every pixel of light. Some of the pixels of the sensor(s) 90 may be “defective pixels,” a term that refers to imaging pixels within the image sensor(s) 90 that fail to sense light levels accurately. …[0364] The defective pixel replacement (DPR) logic 474 may correct defective pixels by replacing them with other values before the pixels are considered in statistics collection in the statistics core 146a. With reference again to FIG. 48, it may be seen that the DPR logic 474 appears after the BLC logic 472. By performing defective pixel replacement after, rather than before, black level compensation, the black levels may be more accurately represented…0365] In one embodiment, defective pixel correction is performed independently for each color component (e.g., R, B, Gr, and Gb for a Bayer pattern). Generally, the DPR logic 474 may provide for dynamic defect correction, wherein the locations of defective pixels are determined automatically based upon directional gradients computed using neighboring pixels of the same color. ….if the stuck pixel is in a region of the current image that is dominated by black or darker colors, then the stuck pixel may be identified as a defective pixel during processing by the DPR logic 474 and corrected accordingly.--, in [0362]-[0365], and, --Thereafter, at step 566, a count C of the number of gradients that are less than or equal to a particular threshold dprTh is determined. As shown at decision logic 568, if C is less than or equal to dprMaxC, then the process 560 continues to step 570, and the current pixel is identified as being defective. The defective pixel is then corrected at step 572 using a replacement value.--, in [0370]);
identify an image distortion level corresponding to each pixel of the RGB image data by comparing the RGB image data with the corrected RGB image data (see COTE: e.g., --[0123] FIG. 129 is an illustration of typical distortion curves for red, green, and blue color channels; [0124] FIG. 130 is an illustration of a 1920×1080 resolution RAW frame that simulates the lens distortion of FIG. 129 [0125] FIG. 131 is an image, illustrating the results of applying demosaic logic to a frame with chromatic aberrations;
[0126] FIG. 132 is a graph illustrating the relative distortion for chromatic aberration correction; [0127] FIG. 133 is a simulated image where chromatic aberrations are removed prior to demosaicing the image;--, in [0123]-[0127], and, --[0187] FIG. 195 is a block diagram of a YCC scaler with geometric distortion correction and scaling-formatting functions, in accordance with an embodiment; [0188] FIG. 196 is a flowchart describing a method for geometric distortion correction, in accordance with an embodiment;--, in [1087]-[0188]; --[0195] FIG. 206 is a block diagram of vertical luminance coordinate generation logic to determine displacement caused by geometric distortion--, in [0195]; and, --[0218] Acquired image data may undergo significant processing before appearing as a finished image. Accordingly, the disclosure below will describe image processing circuitry that can efficiently process image data. Statistics logic of the image processing circuitry may obtain statistics associated with an image in raw format in parallel with other image data processing. A raw-format processing block may also process the raw image data, using the statistics to correct fixed pattern noise, defective pixels, recover highlights lost by the sensor, and/or perform other operations. An RGB-format processing block may employ a more efficient organization, better demosaicing, improved local tone mapping, and/or color correction to correct colors from image data from more than one sensor vendor. A YCC-format processing block may similarly offer a more efficient organization, as well as improved sharpening, geometric distortion correction, and chroma noise reduction. Moreover, many operations may be performed using signed, rather than unsigned, pixel data. Using signed pixel data may preserve image data when operations produce interim negative pixel results, as well when a sensor produces black level noise in the negative direction.--, in [0218]; and, --[0378] The lens shading correction (LSC) gains may be represented in the same order as a Bayer image, with 16-bit gain per color component. The color of the first pixel in the LSC grid gain may be programmed by software…by utilizing the address of lens shading gain base 600 and the grid offsets, the same gain memory can be used while the sensor cropping region is changing. For example, instead of the ISP circuitry having to update grid gain values in internal memory, the ISP circuitry, by merely updating a few parameters (e.g., the grid point intervals 602 and 604), may align the proper grid points for the changed cropping region. By way of example only, this may be useful when cropping is used during digital zooming operations. Further, while the gain grid 600 shown in the embodiment of FIG. 56 is depicted as having generally equally spaced grid points, it should be understood that in other embodiments, the grid points may not necessarily be equally spaced. For instance, in some embodiments, the grid points may be distributed unevenly (e.g., logarithmically), such that the grid points are less concentrated in the center of the LSC region 588, but more concentrated towards the corners of the LSC region 588, typically where lens shading distortion is more noticeable.--, in [0378]-[0380], and [0711]-[0713]; and, -- For instance, FIG. 60 provides an example of how the photograph 632 from FIG. 55 may appear after lens shading correction is applied. As shown, compared to the original image from FIG. 55, the overall light intensity is generally more uniform across the image. Particularly, the light intensity at the approximate center of the image may be substantially equal to the light intensity values at the corners and/or edges of the image. Additionally, as mentioned above, the interpolated gain calculation (Equations 6a and 6b) may, in some embodiments, be replaced with an additive “delta” between grid points by taking advantage of the sequential column and row incrementing structure. As will be appreciated, this reduces computational complexity…Thus, even though the estimation technique may be somewhat less accurate than utilizing the calculation technique in determining R (Equation 8), the margin of error is low enough that the estimated values or R are suitable for determining radial gain components for the present lens shading correction techniques.--, in [0385]-[0390]; and, --[0581] In one embodiment, the temporal filter block 1028 may apply filter coefficients to pixel data from the received image data to generate the filtered pixel output (Yout). The filter coefficients may be adjusted adaptively on a per pixel basis based at least partially upon motion data between an input pixel x(t) and a reference pixel r(t−1). For instance, the input pixel x(t), with the variable “t” denoting a temporal value, may be compared to the reference pixel r(t−1) in a previously filtered frame or a previous original frame to determine the motion data associated with the input pixel. In one embodiment, the motion data may be used to generate a motion table index value (m) that corresponds to a motion table (M).--, in [0581]; and, --[0584] At block 1112, the temporal filter 1028 may receive image data. At block 1114, the temporal filter block 1028 may determine a motion delta value for each respective pixel in the image data. The motion delta value may represent the amount of motion occurring in a respective pixel between frames. The motion delta value may be determined by calculating the difference between a pixel value for the respective pixel in a respective frame and a pixel value for the respective pixel in its previous frame. By comparing these two time dependent pixel values, the temporal filter block 1028 may represent the amount of motion occurring in the respective pixel in the motion delta value.--, in [581]-[0584]; and, -- The total absolute difference value is then normalized by the number of pixels in a window, which may be done here by dividing the total absolute difference value by 9. Similarly, when determining the absolute difference value between P24 and P11, the 3×3 window 1390 and the 3×3 window 1396 (centered about P11) are compared, and the absolute difference between each of P32 and P19, P31 and P18, P30 and P17, P25 and P12, P24 and P11, P23 and P10, P18 and P5, P17 and P6, and P16 and P7 are summed to determine a total absolute difference between the windows 1390 and 1396, and then divided by 9 to obtain a normalized absolute difference value between P24 and P11. As can be appreciated, this process may then be repeated for each neighbor pixel within the 7×7 block 1328 by comparing the 3×3 window 1390 with 3×3 windows centered about every other neighbor pixel within the 7×7 block 1328, with edge pixels being replicated for neighbor pixels located at the edges of the 7×7 block.--, in [0696]-[0698]; and, --[0734] As also depicted in FIG. 122, one effect of the binning process is that the spatial sampling of the binned pixels may not be equally spaced. This spatial distortion may, in some systems, result in aliasing (e.g., jagged edges), which is generally not desirable. Further, because certain image processing steps in the ISP pipe logic 82 may depend upon on the linear placement of the color samples in order to operate correctly, the raw scaler logic 1040 may be applied to perform re-sampling and re-positioning of the binned pixels such that the binned pixels are spatially evenly distributed.--, in [0733]-[0734]; and, --[0759] As discussed above, the raw scaler circuitry 1652 may also provide chromatic aberration correction logic 1737. Chromatic aberration refers generally to the spatial shift of blue and red components with respect to green components. These shifts may be caused by the chromatic aberration of the lens used to capture the image data. …This dependency results in differing geometric distortion for red, green, and blue color components. Longitudinal chromatic aberration causes different colors of light to focus on different planes. Lateral chromatic aberration results in a radial shift between the red, green, and blue wavelengths. [0760] Geometric distortion manifests as a radial variation in the magnification of the lens, resulting in barrel distortion if the magnification decreases radially or pincushion distortion if the magnification increases radially. Under certain circumstances, it may be possible for a lens to exhibit both barrel and pincushion distortion at the same time. For example, the magnification may first decrease radially and then increase near the edge of the lens. Such distortion may be referred to a moustache distortion. Both the geometric distortion and the chromatic aberrations may degrade the quality of the resultant image provided by the ISP. Thus, by either fully or partially correcting the geometric distortion, the chromatic aberration, or both, smaller, thinner, and cheaper lenses may be used while maintain sufficient visual quality in the video and still frames produced by the camera.
[0761] FIG. 129 illustrates typical distortion curves for red, green, and blue color channels 1738, 1739, and 1740, respectively. As illustrated, the graph plots the distortion, sometimes referred to as displacement, versus an ideal undistorted radius. The distortion, as a percentage of the maximum radius, may be represented by the following equation:--, in [0759]-[0762]; and, --[0776] As illustrated in FIG. 139, the red vertical offset for output line 0 decreases to (2) at output sample 140 and the vertical offset for line 2 decreases to (2) at output sample 140. In general the offsets will decrease as radius decreases. This will tend to reduce line storage requirements towards the vertical center of the frame. FIG. 140 illustrates the offset between the vertical position of the center tap on the red (and blue) component and the corresponding green component. Note that this example is for a 1920×1080 frame with approximately 1% chromatic distortion.
[0777] FIG. 140 illustrates vertical offsets from the green channel. As illustrated, a decrease in the magnitude of a positive offset or an increase in the magnitude of a negative offset may indicate that more than one poutput line is generated for each input line, thus indicating that the same input lines are used when generating a pair of vertically adjacent output samples of the same color component (e.g., up-scaling).--, in [0776]-[0777]; and, --[0989] The most significant bits (e.g., the upper 8 bits) of the r signal may index a lookup table (LUT) 4802, which may provide the two nearest displacement values to interpolation logic 4804. It should be appreciated that the LUT 4802 may be a lookup table that is programmed based on the lens used to generate the image data currently being processed. Thus, software may program the LUT 4802 with different values when the image data derives from different cameras. In some embodiments, geometric distortion from third-party cameras may be corrected by programming the LUT 4802 with values sufficient to correct geometric distortion from such third-party cameras and/or lenses (and/or camera and lens combinations). The exact values used in the LUT 4802 may be simulated and/or experimentally obtained by comparing uncorrected images from the imaging device(s) 30 and/or third-party cameras and lenses and determining an amount of horizontal and vertical shifting that may at least partially correct for the effect of geometric distortion. [0990] The interpolation logic 4804 may interpolate the values from the LUT 4802 linearly based on the least significant bits (e.g., the lower 4 bits) of the r signal to produce a radial displacement value.--, in [0989]; and, --[1015] FIG. 209 represents an example of the horizontal luminance resampling logic. As seen in FIG. 209, input buffers 5020 may receive input data dIn. Control logic 5022 may send an enable signal to the input buffers 5020 based on an indication that the data is ready (din_rdy), the location within the line, and current and next xpointer signals. A counter 5024 may count the location within the line and a comparator may compare the count to the (inWidth−1) value to determine when an end of line has been reached. The control logic 5022 may also control the gating of the next xpointer signal into a buffer 5028 and the next xphase signal into a buffer 5030. [1016] Coefficient RAM 5032 receives the xphase signal, which may be used to determine the sampling coefficients to sample the proper fractional amount of each pixel around the displaced coordinates, so as to correct for geometric distortion in the scaled version of the image after resampling.--, in [1015]-[1016]; and, --1025] The coordinate generation logic of the horizontal chrominance scaling logic 4582, 4584 may operate in substantially the same way as the coordinate generation logic of the horizontal luminance scalers 4566, 4568, with slight modifications to accommodate the differences discussed above. Namely, the corrected x-coordinate determined by comparing the displacement and the SourceX coordinate may be divided by 2 (since half as many chrominance samples may be present as luminance samples) to obtain the ultimate distortion-corrected x-coordinate.--, in [1025], and [1033]); and
correct each chrominance value corresponding to each pixel of the RGB image data based on the image distortion level corresponding to each pixel of the RGB image data (see COTE: e.g., --[1022] The horizontal chrominance scaling module is very similar to the horizontal luminance scalers 4566, 4568. The differences may be as follows: [1023] 1. The input to the horizontal chrominance scaler 4582, 4584 is interleaved Cb and Cr samples. Pairs of Cb/Cr samples are cosited and are cosited with the even luminance samples. [1024] 2. If one of the output channels is in 4:2:0 format, there will be half as many chrominance lines as luminance lines output by the channel.
[1025] The coordinate generation logic of the horizontal chrominance scaling logic 4582, 4584 may operate in substantially the same way as the coordinate generation logic of the horizontal luminance scalers 4566, 4568, with slight modifications to accommodate the differences discussed above. Namely, the corrected x-coordinate determined by comparing the displacement and the SourceX coordinate may be divided by 2 (since half as many chrominance samples may be present as luminance samples) to obtain the ultimate distortion-corrected x-coordinate.
[1026] Similarly, the horizontal chrominance resampling logic of the horizontal chrominance scaling logic 4582, 4584 may operate in substantially the same way as the horizontal luminance resampling logic of the horizontal luminance scalers 4566, 4568, with a few exceptions. FIG. 210 illustrates an example of the horizontal chrominance resampling logic of the horizontal chrominance scaling logic 4582, 4584. In the example of FIG. 210, elements 5122, 5124, 5126, 5128, 5130, 5132, 5134, 5136, 5138, 5140, 5142, 5144, and 5146 may respectively operate in the same general way as elements 5022, 5024, 5026, 5028, 5030, 5032, 5034, 5036, 5038, 5040, 5042, 5044, and 5046 of FIG. 209. The control logic 5122 may also differ in that it may control two pipelines of buffers rather than just one—buffers 5120 for Cb chrominance data and buffers 5121 for Cr chrominance data. Data from the two pipelines of buffers thus may be selected by a cr_select signal to multiplexers 5137.
[1027] Essentially, for each input line to the horizontal chrominance scaler 4582, 4584, consisting of “inWidth” samples (inWidth/2 Cb/Cr pairs), the horizontal chrominance coordinate generator component will generate “outWidth/2” X coordinates, one per Cb/Cr pair of output samples. These coordinates define the position of the (geometric-distortion-corrected) output sample relative to the (non-geometric-distortion-corrected, in the horizontal coordinate) input samples, where the position of the input sample is implicit in their numbering (0-inWidth/2−1). The horizontal chrominance coordinate generator produces two output values, “xpointer” and “xphase”.--, in [1022]-[1028], and, --[1033] The image data output by the YCC scaler 4012 thus may be scaled to one or two desired resolutions, while also correcting for geometric distortion. When the upper-left-hand portion of the input image data generally appears as in FIG. 131 (which has been corrected for chromatic aberration but not geometric distortion), the YCC scaler 4012 may produce an output image with the upper-left-hand image shown in FIG. 228.--, in [1033]); and
generate the output image based on each corrected chrominance value (see COTE: e.g., --[1022] The horizontal chrominance scaling module is very similar to the horizontal luminance scalers 4566, 4568. The differences may be as follows: [1023] 1. The input to the horizontal chrominance scaler 4582, 4584 is interleaved Cb and Cr samples. Pairs of Cb/Cr samples are cosited and are cosited with the even luminance samples. [1024] 2. If one of the output channels is in 4:2:0 format, there will be half as many chrominance lines as luminance lines output by the channel.
[1025] The coordinate generation logic of the horizontal chrominance scaling logic 4582, 4584 may operate in substantially the same way as the coordinate generation logic of the horizontal luminance scalers 4566, 4568, with slight modifications to accommodate the differences discussed above. Namely, the corrected x-coordinate determined by comparing the displacement and the SourceX coordinate may be divided by 2 (since half as many chrominance samples may be present as luminance samples) to obtain the ultimate distortion-corrected x-coordinate.
[1026] Similarly, the horizontal chrominance resampling logic of the horizontal chrominance scaling logic 4582, 4584 may operate in substantially the same way as the horizontal luminance resampling logic of the horizontal luminance scalers 4566, 4568, with a few exceptions. FIG. 210 illustrates an example of the horizontal chrominance resampling logic of the horizontal chrominance scaling logic 4582, 4584. In the example of FIG. 210, elements 5122, 5124, 5126, 5128, 5130, 5132, 5134, 5136, 5138, 5140, 5142, 5144, and 5146 may respectively operate in the same general way as elements 5022, 5024, 5026, 5028, 5030, 5032, 5034, 5036, 5038, 5040, 5042, 5044, and 5046 of FIG. 209. The control logic 5122 may also differ in that it may control two pipelines of buffers rather than just one—buffers 5120 for Cb chrominance data and buffers 5121 for Cr chrominance data. Data from the two pipelines of buffers thus may be selected by a cr_select signal to multiplexers 5137.
[1027] Essentially, for each input line to the horizontal chrominance scaler 4582, 4584, consisting of “inWidth” samples (inWidth/2 Cb/Cr pairs), the horizontal chrominance coordinate generator component will generate “outWidth/2” X coordinates, one per Cb/Cr pair of output samples. These coordinates define the position of the (geometric-distortion-corrected) output sample relative to the (non-geometric-distortion-corrected, in the horizontal coordinate) input samples, where the position of the input sample is implicit in their numbering (0-inWidth/2−1). The horizontal chrominance coordinate generator produces two output values, “xpointer” and “xphase”.--, in [1022]-[1028], and, --[1033] The image data output by the YCC scaler 4012 thus may be scaled to one or two desired resolutions, while also correcting for geometric distortion. When the upper-left-hand portion of the input image data generally appears as in FIG. 131 (which has been corrected for chromatic aberration but not geometric distortion), the YCC scaler 4012 may produce an output image with the upper-left-hand image shown in FIG. 228.--, in [1033]).
Re Claim 2, Cote further discloses wherein the ISP is configured to determine a pixel value of each pixel of the RGB image data by interpolation of pixel values of each pixel of the raw image (see COTE: e.g., --[0229] The color filter array may be a Bayer color filter array, an example of which appears in FIG. 2. A Bayer color filter array provides a filter pattern that captures 50% green elements, 25% red elements, and 25% blue elements of light reaching the sensor. In the example of FIG. 2, 2 green elements (Gr and Gb), 1 red element (R), and 1 blue element (B) will repeat in the pattern shown across the full pixel array of the sensor(s) of the imaging device(s) 30. Thus, an image sensor with a Bayer color filter array may provide information regarding the intensity of the light received by the imaging device 30 at the green, red, and blue wavelengths, whereby each image pixel records only one of the three colors (RGB). This information, which may be referred to as “raw image data” or data in the “raw domain,” may be processed using one or more demosaicing techniques to convert the raw image data into a full color image, generally by interpolating a set of red, green, and blue values for each pixel. As will be discussed further below, such demosaicing techniques may be performed by the image processing circuitry 32.--, in [0229]-[0230]).
Re Claim 3, Cote further discloses wherein the selected image data processing method comprises at least one image data processing method that causes the corrected RGB image data to have non-linearity, and the ISP is configured to generate the corrected RGB image data by applying at least one of gamma correction, color correction, or shading correction to the RGB image data (see COTE: e.g., --[0229] The color filter array may be a Bayer color filter array, an example of which appears in FIG. 2. A Bayer color filter array provides a filter pattern that captures 50% green elements, 25% red elements, and 25% blue elements of light reaching the sensor. In the example of FIG. 2, 2 green elements (Gr and Gb), 1 red element (R), and 1 blue element (B) will repeat in the pattern shown across the full pixel array of the sensor(s) of the imaging device(s) 30. Thus, an image sensor with a Bayer color filter array may provide information regarding the intensity of the light received by the imaging device 30 at the green, red, and blue wavelengths, whereby each image pixel records only one of the three colors (RGB). This information, which may be referred to as “raw image data” or data in the “raw domain,” may be processed using one or more demosaicing techniques to convert the raw image data into a full color image, generally by interpolating a set of red, green, and blue values for each pixel. As will be discussed further below, such demosaicing techniques may be performed by the image processing circuitry 32.--, in [0229]-[0230]; and, --[0254] The output of the raw block 150 may be stored in the memory 100 or continue to an RGB-format processing block 160 (also referred to as RgbProc or DMA destination D5). The RGB block 160 may receive one of two image data signals via selection logic 162, which may be processed by RGB image processing logic 164. The RGB image processing logic 164 may perform several image data processing operations, including demosaicing (DEM) to obtain RGB-format image data from raw image data. Having obtained RGB-format image data, the RGB image processing logic 164 may perform local tone mapping (LTM); color correction using a color correction matrix (CCM); color correction using a three-dimensional color lookup table (CLUT); gamma/degamma (GAM); gain, offset, and clipping (GOC); and/or color space conversion (CSC), producing image data in a YCC format (e.g., YCbCr or YUV).--, in [0253]-[0254]; and, --[0378] The lens shading correction (LSC) gains may be represented in the same order as a Bayer image, with 16-bit gain per color component. The color of the first pixel in the LSC grid gain may be programmed by software…by utilizing the address of lens shading gain base 600 and the grid offsets, the same gain memory can be used while the sensor cropping region is changing. For example, instead of the ISP circuitry having to update grid gain values in internal memory, the ISP circuitry, by merely updating a few parameters (e.g., the grid point intervals 602 and 604), may align the proper grid points for the changed cropping region. By way of example only, this may be useful when cropping is used during digital zooming operations. Further, while the gain grid 600 shown in the embodiment of FIG. 56 is depicted as having generally equally spaced grid points, it should be understood that in other embodiments, the grid points may not necessarily be equally spaced. For instance, in some embodiments, the grid points may be distributed unevenly (e.g., logarithmically), such that the grid points are less concentrated in the center of the LSC region 588, but more concentrated towards the corners of the LSC region 588, typically where lens shading distortion is more noticeable.--, in [0378]-[0380], and [0711]-[0713]; and, -- For instance, FIG. 60 provides an example of how the photograph 632 from FIG. 55 may appear after lens shading correction is applied. As shown, compared to the original image from FIG. 55, the overall light intensity is generally more uniform across the image. Particularly, the light intensity at the approximate center of the image may be substantially equal to the light intensity values at the corners and/or edges of the image. Additionally, as mentioned above, the interpolated gain calculation (Equations 6a and 6b) may, in some embodiments, be replaced with an additive “delta” between grid points by taking advantage of the sequential column and row incrementing structure. As will be appreciated, this reduces computational complexity…Thus, even though the estimation technique may be somewhat less accurate than utilizing the calculation technique in determining R (Equation 8), the margin of error is low enough that the estimated values or R are suitable for determining radial gain components for the present lens shading correction techniques.--, in [0385]-[0390], and, --[0411] In the present example, the non-linear CSC logic 807 may be configured to perform a 3×3 matrix multiply, followed by a non-linear mapping implemented as a lookup table, and further followed by another 3×3 matrix multiply with an added offset. This allows for the 3A statistics color space conversion logic 807 to replicate the color processing of the RGB processing logic 160 in the ISP pipe processing logic 80 (e.g., applying white balance gain, applying a color correction matrix, applying RGB gamma adjustments, and performing color space conversion) for a given color temperature. It may also provide for the conversion of the Bayer RGB values to a more color consistent color space such as CIELab, or any of the other color spaces discussed above (e.g., YCbCr, a red/blue normalized color space, etc.). Under some conditions, a Lab color space may be more suitable for white balance operations because the chromaticity is more linear with respect to brightness.--, in [0411]).
Re Claim 4, Cote further discloses wherein each parameter related to the gamma correction, the color correction, and the shading correction is determined based on parameters related to the gamma correction, the color correction, and the shading correction of an application processor (see COTE: e.g., --[0229] The color filter array may be a Bayer color filter array, an example of which appears in FIG. 2. A Bayer color filter array provides a filter pattern that captures 50% green elements, 25% red elements, and 25% blue elements of light reaching the sensor. In the example of FIG. 2, 2 green elements (Gr and Gb), 1 red element (R), and 1 blue element (B) will repeat in the pattern shown across the full pixel array of the sensor(s) of the imaging device(s) 30. Thus, an image sensor with a Bayer color filter array may provide information regarding the intensity of the light received by the imaging device 30 at the green, red, and blue wavelengths, whereby each image pixel records only one of the three colors (RGB). This information, which may be referred to as “raw image data” or data in the “raw domain,” may be processed using one or more demosaicing techniques to convert the raw image data into a full color image, generally by interpolating a set of red, green, and blue values for each pixel. As will be discussed further below, such demosaicing techniques may be performed by the image processing circuitry 32.--, in [0229]-[0230]; and, --[0254] The output of the raw block 150 may be stored in the memory 100 or continue to an RGB-format processing block 160 (also referred to as RgbProc or DMA destination D5). The RGB block 160 may receive one of two image data signals via selection logic 162, which may be processed by RGB image processing logic 164. The RGB image processing logic 164 may perform several image data processing operations, including demosaicing (DEM) to obtain RGB-format image data from raw image data. Having obtained RGB-format image data, the RGB image processing logic 164 may perform local tone mapping (LTM); color correction using a color correction matrix (CCM); color correction using a three-dimensional color lookup table (CLUT); gamma/degamma (GAM); gain, offset, and clipping (GOC); and/or color space conversion (CSC), producing image data in a YCC format (e.g., YCbCr or YUV).--, in [0253]-[0254]; and, --[0378] The lens shading correction (LSC) gains may be represented in the same order as a Bayer image, with 16-bit gain per color component. The color of the first pixel in the LSC grid gain may be programmed by software…by utilizing the address of lens shading gain base 600 and the grid offsets, the same gain memory can be used while the sensor cropping region is changing. For example, instead of the ISP circuitry having to update grid gain values in internal memory, the ISP circuitry, by merely updating a few parameters (e.g., the grid point intervals 602 and 604), may align the proper grid points for the changed cropping region. By way of example only, this may be useful when cropping is used during digital zooming operations. Further, while the gain grid 600 shown in the embodiment of FIG. 56 is depicted as having generally equally spaced grid points, it should be understood that in other embodiments, the grid points may not necessarily be equally spaced. For instance, in some embodiments, the grid points may be distributed unevenly (e.g., logarithmically), such that the grid points are less concentrated in the center of the LSC region 588, but more concentrated towards the corners of the LSC region 588, typically where lens shading distortion is more noticeable.--, in [0378]-[0380], and [0711]-[0713]; and, -- For instance, FIG. 60 provides an example of how the photograph 632 from FIG. 55 may appear after lens shading correction is applied. As shown, compared to the original image from FIG. 55, the overall light intensity is generally more uniform across the image. Particularly, the light intensity at the approximate center of the image may be substantially equal to the light intensity values at the corners and/or edges of the image. Additionally, as mentioned above, the interpolated gain calculation (Equations 6a and 6b) may, in some embodiments, be replaced with an additive “delta” between grid points by taking advantage of the sequential column and row incrementing structure. As will be appreciated, this reduces computational complexity…Thus, even though the estimation technique may be somewhat less accurate than utilizing the calculation technique in determining R (Equation 8), the margin of error is low enough that the estimated values or R are suitable for determining radial gain components for the present lens shading correction techniques.--, in [0385]-[0390], and, --[0411] In the present example, the non-linear CSC logic 807 may be configured to perform a 3×3 matrix multiply, followed by a non-linear mapping implemented as a lookup table, and further followed by another 3×3 matrix multiply with an added offset. This allows for the 3A statistics color space conversion logic 807 to replicate the color processing of the RGB processing logic 160 in the ISP pipe processing logic 80 (e.g., applying white balance gain, applying a color correction matrix, applying RGB gamma adjustments, and performing color space conversion) for a given color temperature. It may also provide for the conversion of the Bayer RGB values to a more color consistent color space such as CIELab, or any of the other color spaces discussed above (e.g., YCbCr, a red/blue normalized color space, etc.). Under some conditions, a Lab color space may be more suitable for white balance operations because the chromaticity is more linear with respect to brightness.--, in [0411]).
Re Claim 5, Cote further discloses calculate edge connectivity of each pixel of the RGB image data (see COTE: e.g., --[0584] At block 1112, the temporal filter 1028 may receive image data. At block 1114, the temporal filter block 1028 may determine a motion delta value for each respective pixel in the image data. The motion delta value may represent the amount of motion occurring in a respective pixel between frames. The motion delta value may be determined by calculating the difference between a pixel value for the respective pixel in a respective frame and a pixel value for the respective pixel in its previous frame. By comparing these two time dependent pixel values, the temporal filter block 1028 may represent the amount of motion occurring in the respective pixel in the motion delta value.--, in [581]-[0584]; and, -- The total absolute difference value is then normalized by the number of pixels in a window, which may be done here by dividing the total absolute difference value by 9. Similarly, when determining the absolute difference value between P24 and P11, the 3×3 window 1390 and the 3×3 window 1396 (centered about P11) are compared, and the absolute difference between each of P32 and P19, P31 and P18, P30 and P17, P25 and P12, P24 and P11, P23 and P10, P18 and P5, P17 and P6, and P16 and P7 are summed to determine a total absolute difference between the windows 1390 and 1396, and then divided by 9 to obtain a normalized absolute difference value between P24 and P11. As can be appreciated, this process may then be repeated for each neighbor pixel within the 7×7 block 1328 by comparing the 3×3 window 1390 with 3×3 windows centered about every other neighbor pixel within the 7×7 block 1328, with edge pixels being replicated for neighbor pixels located at the edges of the 7×7 block.--, in [0696]-[0698]; and, --[0734] As also depicted in FIG. 122, one effect of the binning process is that the spatial sampling of the binned pixels may not be equally spaced. This spatial distortion may, in some systems, result in aliasing (e.g., jagged edges), which is generally not desirable. Further, because certain image processing steps in the ISP pipe logic 82 may depend upon on the linear placement of the color samples in order to operate correctly, the raw scaler logic 1040 may be applied to perform re-sampling and re-positioning of the binned pixels such that the binned pixels are spatially evenly distributed.--, in [0733]-[0734]);
calculate edge connectivity of each pixel of the corrected RGB image data (see Cote: e.g., --[0584] At block 1112, the temporal filter 1028 may receive image data. At block 1114, the temporal filter block 1028 may determine a motion delta value for each respective pixel in the image data. The motion delta value may represent the amount of motion occurring in a respective pixel between frames. The motion delta value may be determined by calculating the difference between a pixel value for the respective pixel in a respective frame and a pixel value for the respective pixel in its previous frame. By comparing these two time dependent pixel values, the temporal filter block 1028 may represent the amount of motion occurring in the respective pixel in the motion delta value.--, in [581]-[0584]; and, -- The total absolute difference value is then normalized by the number of pixels in a window, which may be done here by dividing the total absolute difference value by 9. Similarly, when determining the absolute difference value between P24 and P11, the 3×3 window 1390 and the 3×3 window 1396 (centered about P11) are compared, and the absolute difference between each of P32 and P19, P31 and P18, P30 and P17, P25 and P12, P24 and P11, P23 and P10, P18 and P5, P17 and P6, and P16 and P7 are summed to determine a total absolute difference between the windows 1390 and 1396, and then divided by 9 to obtain a normalized absolute difference value between P24 and P11. As can be appreciated, this process may then be repeated for each neighbor pixel within the 7×7 block 1328 by comparing the 3×3 window 1390 with 3×3 windows centered about every other neighbor pixel within the 7×7 block 1328, with edge pixels being replicated for neighbor pixels located at the edges of the 7×7 block.--, in [0696]-[0698]; and, --[0734] As also depicted in FIG. 122, one effect of the binning process is that the spatial sampling of the binned pixels may not be equally spaced. This spatial distortion may, in some systems, result in aliasing (e.g., jagged edges), which is generally not desirable. Further, because certain image processing steps in the ISP pipe logic 82 may depend upon on the linear placement of the color samples in order to operate correctly, the raw scaler logic 1040 may be applied to perform re-sampling and re-positioning of the binned pixels such that the binned pixels are spatially evenly distributed.--, in [0733]-[0734]);
calculate distortion weight corresponding to each pixel of the RGB image data based on the edge connectivity of each pixel of the RGB image data, the edge connectivity of each pixel of the corrected RGB image data and a noise level of the image sensor (see Cote: e.g., --[0799] A demosaicing technique that may be implemented by the demosaicing logic 404 will now be described in accordance with one embodiment. On the green color channel, missing color samples may be interpolated using a low pass directional filter on known green samples and a high pass (or gradient) filter on the adjacent color channels (e.g., red and blue). For the red and blue color channels, the missing color samples may be interpolated in a similar manner, but by using low pass filtering on known red or blue values and high pass filtering on co-located interpolated green values. Further, in one embodiment, demosaicing on the green color channel may utilize a 5×5 pixel block edge-adaptive filter based on the original Bayer color data. As will be discussed further below, the use of an edge-adaptive filter may provide for the continuous weighting based on gradients of horizontal and vertical filtered values, which reduce the appearance of certain artifacts, such as aliasing, “checkerboard,” or “rainbow” artifacts, commonly seen in conventional demosaicing techniques.
[0800] During demosaicing on the green channel, the original values for the green pixels (Gr and Gb pixels) of the Bayer image pattern are used unless the GNU correction mode two is enabled. However, to obtain a full set of data for the green channel, green pixel values may be interpolated at the red and blue pixels of the Bayer image pattern.--, in [0799]-[0800]);
calculate an image distortion level corresponding to each pixel of the RGB image data based on the distortion weight (see COTE: e.g., --[0123] FIG. 129 is an illustration of typical distortion curves for red, green, and blue color channels; [0124] FIG. 130 is an illustration of a 1920×1080 resolution RAW frame that simulates the lens distortion of FIG. 129 [0125] FIG. 131 is an image, illustrating the results of applying demosaic logic to a frame with chromatic aberrations;
[0126] FIG. 132 is a graph illustrating the relative distortion for chromatic aberration correction; [0127] FIG. 133 is a simulated image where chromatic aberrations are removed prior to demosaicing the image;--, in [0123]-[0127], and, --[0187] FIG. 195 is a block diagram of a YCC scaler with geometric distortion correction and scaling-formatting functions, in accordance with an embodiment; [0188] FIG. 196 is a flowchart describing a method for geometric distortion correction, in accordance with an embodiment;--, in [1087]-[0188]; --[0195] FIG. 206 is a block diagram of vertical luminance coordinate generation logic to determine displacement caused by geometric distortion--, in [0195]; and, --[0218] Acquired image data may undergo significant processing before appearing as a finished image. Accordingly, the disclosure below will describe image processing circuitry that can efficiently process image data. Statistics logic of the image processing circuitry may obtain statistics associated with an image in raw format in parallel with other image data processing. A raw-format processing block may also process the raw image data, using the statistics to correct fixed pattern noise, defective pixels, recover highlights lost by the sensor, and/or perform other operations. An RGB-format processing block may employ a more efficient organization, better demosaicing, improved local tone mapping, and/or color correction to correct colors from image data from more than one sensor vendor. A YCC-format processing block may similarly offer a more efficient organization, as well as improved sharpening, geometric distortion correction, and chroma noise reduction. Moreover, many operations may be performed using signed, rather than unsigned, pixel data. Using signed pixel data may preserve image data when operations produce interim negative pixel results, as well when a sensor produces black level noise in the negative direction.--, in [0218]; and, --[0378] The lens shading correction (LSC) gains may be represented in the same order as a Bayer image, with 16-bit gain per color component. The color of the first pixel in the LSC grid gain may be programmed by software…by utilizing the address of lens shading gain base 600 and the grid offsets, the same gain memory can be used while the sensor cropping region is changing. For example, instead of the ISP circuitry having to update grid gain values in internal memory, the ISP circuitry, by merely updating a few parameters (e.g., the grid point intervals 602 and 604), may align the proper grid points for the changed cropping region. By way of example only, this may be useful when cropping is used during digital zooming operations. Further, while the gain grid 600 shown in the embodiment of FIG. 56 is depicted as having generally equally spaced grid points, it should be understood that in other embodiments, the grid points may not necessarily be equally spaced. For instance, in some embodiments, the grid points may be distributed unevenly (e.g., logarithmically), such that the grid points are less concentrated in the center of the LSC region 588, but more concentrated towards the corners of the LSC region 588, typically where lens shading distortion is more noticeable.--, in [0378]-[0380], and [0711]-[0713]; and, -- For instance, FIG. 60 provides an example of how the photograph 632 from FIG. 55 may appear after lens shading correction is applied. As shown, compared to the original image from FIG. 55, the overall light intensity is generally more uniform across the image. Particularly, the light intensity at the approximate center of the image may be substantially equal to the light intensity values at the corners and/or edges of the image. Additionally, as mentioned above, the interpolated gain calculation (Equations 6a and 6b) may, in some embodiments, be replaced with an additive “delta” between grid points by taking advantage of the sequential column and row incrementing structure. As will be appreciated, this reduces computational complexity…Thus, even though the estimation technique may be somewhat less accurate than utilizing the calculation technique in determining R (Equation 8), the margin of error is low enough that the estimated values or R are suitable for determining radial gain components for the present lens shading correction techniques.--, in [0385]-[0390]; and, --[0581] In one embodiment, the temporal filter block 1028 may apply filter coefficients to pixel data from the received image data to generate the filtered pixel output (Yout). The filter coefficients may be adjusted adaptively on a per pixel basis based at least partially upon motion data between an input pixel x(t) and a reference pixel r(t−1). For instance, the input pixel x(t), with the variable “t” denoting a temporal value, may be compared to the reference pixel r(t−1) in a previously filtered frame or a previous original frame to determine the motion data associated with the input pixel. In one embodiment, the motion data may be used to generate a motion table index value (m) that corresponds to a motion table (M).--, in [0581]; and, --[0584] At block 1112, the temporal filter 1028 may receive image data. At block 1114, the temporal filter block 1028 may determine a motion delta value for each respective pixel in the image data. The motion delta value may represent the amount of motion occurring in a respective pixel between frames. The motion delta value may be determined by calculating the difference between a pixel value for the respective pixel in a respective frame and a pixel value for the respective pixel in its previous frame. By comparing these two time dependent pixel values, the temporal filter block 1028 may represent the amount of motion occurring in the respective pixel in the motion delta value.--, in [581]-[0584]; and, -- The total absolute difference value is then normalized by the number of pixels in a window, which may be done here by dividing the total absolute difference value by 9. Similarly, when determining the absolute difference value between P24 and P11, the 3×3 window 1390 and the 3×3 window 1396 (centered about P11) are compared, and the absolute difference between each of P32 and P19, P31 and P18, P30 and P17, P25 and P12, P24 and P11, P23 and P10, P18 and P5, P17 and P6, and P16 and P7 are summed to determine a total absolute difference between the windows 1390 and 1396, and then divided by 9 to obtain a normalized absolute difference value between P24 and P11. As can be appreciated, this process may then be repeated for each neighbor pixel within the 7×7 block 1328 by comparing the 3×3 window 1390 with 3×3 windows centered about every other neighbor pixel within the 7×7 block 1328, with edge pixels being replicated for neighbor pixels located at the edges of the 7×7 block.--, in [0696]-[0698]; and, --[0734] As also depicted in FIG. 122, one effect of the binning process is that the spatial sampling of the binned pixels may not be equally spaced. This spatial distortion may, in some systems, result in aliasing (e.g., jagged edges), which is generally not desirable. Further, because certain image processing steps in the ISP pipe logic 82 may depend upon on the linear placement of the color samples in order to operate correctly, the raw scaler logic 1040 may be applied to perform re-sampling and re-positioning of the binned pixels such that the binned pixels are spatially evenly distributed.--, in [0733]-[0734]; and, --[0759] As discussed above, the raw scaler circuitry 1652 may also provide chromatic aberration correction logic 1737. Chromatic aberration refers generally to the spatial shift of blue and red components with respect to green components. These shifts may be caused by the chromatic aberration of the lens used to capture the image data. …This dependency results in differing geometric distortion for red, green, and blue color components. Longitudinal chromatic aberration causes different colors of light to focus on different planes. Lateral chromatic aberration results in a radial shift between the red, green, and blue wavelengths. [0760] Geometric distortion manifests as a radial variation in the magnification of the lens, resulting in barrel distortion if the magnification decreases radially or pincushion distortion if the magnification increases radially. Under certain circumstances, it may be possible for a lens to exhibit both barrel and pincushion distortion at the same time. For example, the magnification may first decrease radially and then increase near the edge of the lens. Such distortion may be referred to a moustache distortion. Both the geometric distortion and the chromatic aberrations may degrade the quality of the resultant image provided by the ISP. Thus, by either fully or partially correcting the geometric distortion, the chromatic aberration, or both, smaller, thinner, and cheaper lenses may be used while maintain sufficient visual quality in the video and still frames produced by the camera.
[0761] FIG. 129 illustrates typical distortion curves for red, green, and blue color channels 1738, 1739, and 1740, respectively. As illustrated, the graph plots the distortion, sometimes referred to as displacement, versus an ideal undistorted radius. The distortion, as a percentage of the maximum radius, may be represented by the following equation:--, in [0759]-[0762]; and, --[0776] As illustrated in FIG. 139, the red vertical offset for output line 0 decreases to (2) at output sample 140 and the vertical offset for line 2 decreases to (2) at output sample 140. In general the offsets will decrease as radius decreases. This will tend to reduce line storage requirements towards the vertical center of the frame. FIG. 140 illustrates the offset between the vertical position of the center tap on the red (and blue) component and the corresponding green component. Note that this example is for a 1920×1080 frame with approximately 1% chromatic distortion.
[0777] FIG. 140 illustrates vertical offsets from the green channel. As illustrated, a decrease in the magnitude of a positive offset or an increase in the magnitude of a negative offset may indicate that more than one poutput line is generated for each input line, thus indicating that the same input lines are used when generating a pair of vertically adjacent output samples of the same color component (e.g., up-scaling).--, in [0776]-[0777]; and, --where interp1 performs linear interpolation of weights the white pin LUT 3668 (e.g., LUT_WhitePin), which represent the weights used for determining whether to blend the target white value or not. A blending value from the white pin LUT 3668 of 1 may be considered equivalent to keeping the original values and bypassing the blending. When the white pin LUT 3668 is disabled, the BlendWeightWhite for blending white into the output pixel signal Rout, Gout, and Bout 3550 may be set to 0x8000. In conclusion, by processing the RGB image data through the local tone mapping (LTM) logic 3004, the image data may be gained up or down to preserve specular highlight information as well as image information contained in dark areas of the image scene. Moreover, local variations in color due to different illuminants in different areas of the scene may also ensure proper color reproduction. Even when applying certain gains could cause the pixel to appear gray when the pixel should appear white (e.g., for particularly bright areas of the image frame nearer to the optical center of the image frame), the pin-to-white logic may ensure that the output pixel is pinned to white to avoid such color distortions.--, in [0870]; and, --[0989] The most significant bits (e.g., the upper 8 bits) of the r signal may index a lookup table (LUT) 4802, which may provide the two nearest displacement values to interpolation logic 4804. It should be appreciated that the LUT 4802 may be a lookup table that is programmed based on the lens used to generate the image data currently being processed. Thus, software may program the LUT 4802 with different values when the image data derives from different cameras. In some embodiments, geometric distortion from third-party cameras may be corrected by programming the LUT 4802 with values sufficient to correct geometric distortion from such third-party cameras and/or lenses (and/or camera and lens combinations). The exact values used in the LUT 4802 may be simulated and/or experimentally obtained by comparing uncorrected images from the imaging device(s) 30 and/or third-party cameras and lenses and determining an amount of horizontal and vertical shifting that may at least partially correct for the effect of geometric distortion. [0990] The interpolation logic 4804 may interpolate the values from the LUT 4802 linearly based on the least significant bits (e.g., the lower 4 bits) of the r signal to produce a radial displacement value.--, in [0989]; and, --[1015] FIG. 209 represents an example of the horizontal luminance resampling logic. As seen in FIG. 209, input buffers 5020 may receive input data dIn. Control logic 5022 may send an enable signal to the input buffers 5020 based on an indication that the data is ready (din_rdy), the location within the line, and current and next xpointer signals. A counter 5024 may count the location within the line and a comparator may compare the count to the (inWidth−1) value to determine when an end of line has been reached. The control logic 5022 may also control the gating of the next xpointer signal into a buffer 5028 and the next xphase signal into a buffer 5030. [1016] Coefficient RAM 5032 receives the xphase signal, which may be used to determine the sampling coefficients to sample the proper fractional amount of each pixel around the displaced coordinates, so as to correct for geometric distortion in the scaled version of the image after resampling.--, in [1015]-[1016]; and, --1025] The coordinate generation logic of the horizontal chrominance scaling logic 4582, 4584 may operate in substantially the same way as the coordinate generation logic of the horizontal luminance scalers 4566, 4568, with slight modifications to accommodate the differences discussed above. Namely, the corrected x-coordinate determined by comparing the displacement and the SourceX coordinate may be divided by 2 (since half as many chrominance samples may be present as luminance samples) to obtain the ultimate distortion-corrected x-coordinate.--, in [1025], and [1033]).
Re Claim 6, Cote further discloses with respect to each pixel of image data, the ISP is configured to calculate the edge connectivity based on a pixel value of a pixel and a pixel value of each neighboring pixel (see Cote: e.g., --[0584] At block 1112, the temporal filter 1028 may receive image data. At block 1114, the temporal filter block 1028 may determine a motion delta value for each respective pixel in the image data. The motion delta value may represent the amount of motion occurring in a respective pixel between frames. The motion delta value may be determined by calculating the difference between a pixel value for the respective pixel in a respective frame and a pixel value for the respective pixel in its previous frame. By comparing these two time dependent pixel values, the temporal filter block 1028 may represent the amount of motion occurring in the respective pixel in the motion delta value.--, in [581]-[0584]; and, -- The total absolute difference value is then normalized by the number of pixels in a window, which may be done here by dividing the total absolute difference value by 9. Similarly, when determining the absolute difference value between P24 and P11, the 3×3 window 1390 and the 3×3 window 1396 (centered about P11) are compared, and the absolute difference between each of P32 and P19, P31 and P18, P30 and P17, P25 and P12, P24 and P11, P23 and P10, P18 and P5, P17 and P6, and P16 and P7 are summed to determine a total absolute difference between the windows 1390 and 1396, and then divided by 9 to obtain a normalized absolute difference value between P24 and P11. As can be appreciated, this process may then be repeated for each neighbor pixel within the 7×7 block 1328 by comparing the 3×3 window 1390 with 3×3 windows centered about every other neighbor pixel within the 7×7 block 1328, with edge pixels being replicated for neighbor pixels located at the edges of the 7×7 block.--, in [0696]-[0698]; and, --[0734] As also depicted in FIG. 122, one effect of the binning process is that the spatial sampling of the binned pixels may not be equally spaced. This spatial distortion may, in some systems, result in aliasing (e.g., jagged edges), which is generally not desirable. Further, because certain image processing steps in the ISP pipe logic 82 may depend upon on the linear placement of the color samples in order to operate correctly, the raw scaler logic 1040 may be applied to perform re-sampling and re-positioning of the binned pixels such that the binned pixels are spatially evenly distributed.--, in [0733]-[0734]).
Re Claim 7, Cote further discloses wherein the ISP is configured to determine the image distortion level corresponding to each pixel of the RGB image data based on a lower threshold value, an upper threshold value, and the distortion weight corresponding to each pixel of the RGB image data (see Cote: e.g., --Once the pixel-to-pixel gradients have been determined, defective pixel detection may be performed by the DPR logic 474 as follows. First, it is assumed that a pixel is defective if a certain number of its gradients G.sub.k are at or below a particular threshold, denoted by the variable dprTh. Thus, for each pixel, a count (C) of the number of gradients for neighboring pixels inside the picture boundaries that are at or below the threshold dprTh is accumulated. By way of example, for each neighbor pixel inside the raw frame 310, the accumulated count C of the gradients G.sub.k that are at or below the threshold dprTh may be computed--, in [0368]-[0370]).
Re Claim 8, Cote further discloses wherein the lower threshold value and the upper threshold value are determined based on the noise level of the image sensor (see Cote: e.g., --Once the pixel-to-pixel gradients have been determined, defective pixel detection may be performed by the DPR logic 474 as follows. First, it is assumed that a pixel is defective if a certain number of its gradients G.sub.k are at or below a particular threshold, denoted by the variable dprTh. Thus, for each pixel, a count (C) of the number of gradients for neighboring pixels inside the picture boundaries that are at or below the threshold dprTh is accumulated. By way of example, for each neighbor pixel inside the raw frame 310, the accumulated count C of the gradients G.sub.k that are at or below the threshold dprTh may be computed--, in [0368]-[0370], and, --lens shading correction statistics may collect a number of pixels that are above a programmable threshold value before and/or after the lens shading correction is applied. For example, in some embodiments, a programmable threshold value may be set to a sensor's saturation value. The lens shading correction statistics may count the number of pixels at or above the sensor's saturation value before lens shading correction is applied. Further, a second threshold value may be set to a desired clip level at the output of the lens shading correction. The lens shading correction statistics may count the number of pixels at or above the desired clip level after lens shading correction has been applied. The lens shading correction statistics may also count the number of pixels that both are above the sensor's saturation value before lens shading correction is applied and are above the desired clip level after the lens shading correction is applied.--, in [0382]).
Re Claim 9, Cote further discloses wherein the ISP is configured to: determine a representative value of each chrominance value of the RGB image data (see Cote: e.g., --[0254] The output of the raw block 150 may be stored in the memory 100 or continue to an RGB-format processing block 160 (also referred to as RgbProc or DMA destination D5). The RGB block 160 may receive one of two image data signals via selection logic 162, which may be processed by RGB image processing logic 164. The RGB image processing logic 164 may perform several image data processing operations, including demosaicing (DEM) to obtain RGB-format image data from raw image data. Having obtained RGB-format image data, the RGB image processing logic 164 may perform local tone mapping (LTM); color correction using a color correction matrix (CCM); color correction using a three-dimensional color lookup table (CLUT); gamma/degamma (GAM); gain, offset, and clipping (GOC); and/or color space conversion (CSC), producing image data in a YCC format (e.g., YCbCr or YUV).--, in [0253]-[0254]; also see: --[0229] The color filter array may be a Bayer color filter array, an example of which appears in FIG. 2. A Bayer color filter array provides a filter pattern that captures 50% green elements, 25% red elements, and 25% blue elements of light reaching the sensor. In the example of FIG. 2, 2 green elements (Gr and Gb), 1 red element (R), and 1 blue element (B) will repeat in the pattern shown across the full pixel array of the sensor(s) of the imaging device(s) 30. Thus, an image sensor with a Bayer color filter array may provide information regarding the intensity of the light received by the imaging device 30 at the green, red, and blue wavelengths, whereby each image pixel records only one of the three colors (RGB). This information, which may be referred to as “raw image data” or data in the “raw domain,” may be processed using one or more demosaicing techniques to convert the raw image data into a full color image, generally by interpolating a set of red, green, and blue values for each pixel. As will be discussed further below, such demosaicing techniques may be performed by the image processing circuitry 32.--, in [0229]-[0230]; also see: --[0019] FIG. 8 is a block diagram of an example of the image processing circuitry of FIG. 1, including statistics logic, a raw-format processing block, an RGB-format processing block, and a YCC-format processing block, in accordance with an embodiment;
[0020] FIG. 9 is flowchart depicting a method for processing image data in the ISP pipe processing logic 80 logic of FIG. 10, in accordance with an embodiment;
[0021] FIG. 10 is block diagram illustrating a configuration of double buffered registers and control registers that may be used for processing image data in the ISP pipe processing logic 80 logic, in accordance with an embodiment;--, in [0019]-[0021], and, --[0034] FIGS. 29-34 show examples of memory formats for full-color RGB image data that may be supported by the image processing circuitry of FIG. 7 or FIG. 8,--, in [0034]); and
correct each chrominance value of the each pixel based on the image distortion level of each pixel of the RGB image data, each chrominance value of each pixel of the RGB image data, and the representative value (see COTE: e.g., --[0123] FIG. 129 is an illustration of typical distortion curves for red, green, and blue color channels; [0124] FIG. 130 is an illustration of a 1920×1080 resolution RAW frame that simulates the lens distortion of FIG. 129 [0125] FIG. 131 is an image, illustrating the results of applying demosaic logic to a frame with chromatic aberrations;
[0126] FIG. 132 is a graph illustrating the relative distortion for chromatic aberration correction; [0127] FIG. 133 is a simulated image where chromatic aberrations are removed prior to demosaicing the image;--, in [0123]-[0127], and, --[0187] FIG. 195 is a block diagram of a YCC scaler with geometric distortion correction and scaling-formatting functions, in accordance with an embodiment; [0188] FIG. 196 is a flowchart describing a method for geometric distortion correction, in accordance with an embodiment;--, in [1087]-[0188]; --[0195] FIG. 206 is a block diagram of vertical luminance coordinate generation logic to determine displacement caused by geometric distortion--, in [0195]; and, --[0218] Acquired image data may undergo significant processing before appearing as a finished image. Accordingly, the disclosure below will describe image processing circuitry that can efficiently process image data. Statistics logic of the image processing circuitry may obtain statistics associated with an image in raw format in parallel with other image data processing. A raw-format processing block may also process the raw image data, using the statistics to correct fixed pattern noise, defective pixels, recover highlights lost by the sensor, and/or perform other operations. An RGB-format processing block may employ a more efficient organization, better demosaicing, improved local tone mapping, and/or color correction to correct colors from image data from more than one sensor vendor. A YCC-format processing block may similarly offer a more efficient organization, as well as improved sharpening, geometric distortion correction, and chroma noise reduction. Moreover, many operations may be performed using signed, rather than unsigned, pixel data. Using signed pixel data may preserve image data when operations produce interim negative pixel results, as well when a sensor produces black level noise in the negative direction.--, in [0218]; and, --[0378] The lens shading correction (LSC) gains may be represented in the same order as a Bayer image, with 16-bit gain per color component. The color of the first pixel in the LSC grid gain may be programmed by software…by utilizing the address of lens shading gain base 600 and the grid offsets, the same gain memory can be used while the sensor cropping region is changing. For example, instead of the ISP circuitry having to update grid gain values in internal memory, the ISP circuitry, by merely updating a few parameters (e.g., the grid point intervals 602 and 604), may align the proper grid points for the changed cropping region. By way of example only, this may be useful when cropping is used during digital zooming operations. Further, while the gain grid 600 shown in the embodiment of FIG. 56 is depicted as having generally equally spaced grid points, it should be understood that in other embodiments, the grid points may not necessarily be equally spaced. For instance, in some embodiments, the grid points may be distributed unevenly (e.g., logarithmically), such that the grid points are less concentrated in the center of the LSC region 588, but more concentrated towards the corners of the LSC region 588, typically where lens shading distortion is more noticeable.--, in [0378]-[0380]).
Re Claim 10, Cote further discloses wherein the ISP is configured to: convert the RGB image data into YUV image data (see Cote: e.g., --[0254] The output of the raw block 150 may be stored in the memory 100 or continue to an RGB-format processing block 160 (also referred to as RgbProc or DMA destination D5). The RGB block 160 may receive one of two image data signals via selection logic 162, which may be processed by RGB image processing logic 164. The RGB image processing logic 164 may perform several image data processing operations, including demosaicing (DEM) to obtain RGB-format image data from raw image data. Having obtained RGB-format image data, the RGB image processing logic 164 may perform local tone mapping (LTM); color correction using a color correction matrix (CCM); color correction using a three-dimensional color lookup table (CLUT); gamma/degamma (GAM); gain, offset, and clipping (GOC); and/or color space conversion (CSC), producing image data in a YCC format (e.g., YCbCr or YUV).--, in [0253]-[0254]);
calculate an absolute value of each chrominance value for each pixel of the YUV image data (see Cote: e.g., --The input offset value that is applied may be a signed value applied before the sensor linearization (SLIN) logic 470 looks up the new value of the pixel in the lookup tables 494. For negative pixel values, the pixel value selected from the lookup table 498, 500, 502, or 504 may be the absolute value of the input pixel. The sign of the image data may be applied after the resulting lookup table output value has been obtained. It may be appreciated that this is equivalent to miring the lookup tables 498, 500, 502, and 504 around zero.--, in [0357]; ,and,
--- The total absolute difference value is then normalized by the number of pixels in a window, which may be done here by dividing the total absolute difference value by 9. Similarly, when determining the absolute difference value between P24 and P11, the 3×3 window 1390 and the 3×3 window 1396 (centered about P11) are compared, and the absolute difference between each of P32 and P19, P31 and P18, P30 and P17, P25 and P12, P24 and P11, P23 and P10, P18 and P5, P17 and P6, and P16 and P7 are summed to determine a total absolute difference between the windows 1390 and 1396, and then divided by 9 to obtain a normalized absolute difference value between P24 and P11. As can be appreciated, this process may then be repeated for each neighbor pixel within the 7×7 block 1328 by comparing the 3×3 window 1390 with 3×3 windows centered about every other neighbor pixel within the 7×7 block 1328, with edge pixels being replicated for neighbor pixels located at the edges of the 7×7 block.--, in [0696]-[0698]); and
determine a minimum value or a median value of the absolute value of each chrominance value for each pixel of the YUV image data as the representative value (see Cote: e.g., --[0360] Returning to FIG. 48, the output of the sensor linearization (SLIN) logic 470 may be passed to the black level compensation (BLC) logic 472. The BLC logic 472 may provide for digital gain, offset, and clipping independently for each color component “c” (e.g., R, B, Gr, and Gb for Bayer) on the pixels used for statistics collection. For instance, as expressed by the following operation, the input value for the current pixel is first offset by a signed value, and then multiplied by a gain.
Y=(X+O[c])×G[c] (1),
where X represents the input pixel value for a given color component c (e.g., R, B, Gr, or Gb), O[c] represents a signed 16-bit offset for the current color component c, G[c] represents a gain value for the color component c, and Y represents the output pixel value. In one embodiment, the gain G[c] may be a 16-bit unsigned number with 2 integer bits and 14 fraction bits (e.g., 2.14 in floating point representation), and the gain G[c] may be applied with rounding. By way of example, the gain G[c] may have a range of between 0 to 4 (e.g., 4 times the input pixel value).
[0361] Next, as shown by Equation 2 below, the computed value Y, which is signed, may then be then clipped to a minimum and maximum range:
Y=(Y<min[c])?min[c]:(Y>max[c])?max[c]:Y) (2).--, in [0357], and [0360]-0361]; and,
-- The total absolute difference value is then normalized by the number of pixels in a window, which may be done here by dividing the total absolute difference value by 9. Similarly, when determining the absolute difference value between P24 and P11, the 3×3 window 1390 and the 3×3 window 1396 (centered about P11) are compared, and the absolute difference between each of P32 and P19, P31 and P18, P30 and P17, P25 and P12, P24 and P11, P23 and P10, P18 and P5, P17 and P6, and P16 and P7 are summed to determine a total absolute difference between the windows 1390 and 1396, and then divided by 9 to obtain a normalized absolute difference value between P24 and P11. As can be appreciated, this process may then be repeated for each neighbor pixel within the 7×7 block 1328 by comparing the 3×3 window 1390 with 3×3 windows centered about every other neighbor pixel within the 7×7 block 1328, with edge pixels being replicated for neighbor pixels located at the edges of the 7×7 block.--, in [0696]-[0698]).
Re Claim 11, Cote further discloses wherein with respect to each pixel of the RGB image data, the ISP is configured to determine a value of U based on a difference between a blue pixel value and a green pixel value, and a value of V based on a difference between a red pixel value and the green pixel value (see Cote: e.g., --[0734] As also depicted in FIG. 122, one effect of the binning process is that the spatial sampling of the binned pixels may not be equally spaced. This spatial distortion may, in some systems, result in aliasing (e.g., jagged edges), which is generally not desirable. Further, because certain image processing steps in the ISP pipe logic 82 may depend upon on the linear placement of the color samples in order to operate correctly, the raw scaler logic 1040 may be applied to perform re-sampling and re-positioning of the binned pixels such that the binned pixels are spatially evenly distributed.--, in [0733]-[0734]; and, --[0759] As discussed above, the raw scaler circuitry 1652 may also provide chromatic aberration correction logic 1737. Chromatic aberration refers generally to the spatial shift of blue and red components with respect to green components. These shifts may be caused by the chromatic aberration of the lens used to capture the image data. …This dependency results in differing geometric distortion for red, green, and blue color components. Longitudinal chromatic aberration causes different colors of light to focus on different planes. Lateral chromatic aberration results in a radial shift between the red, green, and blue wavelengths. [0760] Geometric distortion manifests as a radial variation in the magnification of the lens, resulting in barrel distortion if the magnification decreases radially or pincushion distortion if the magnification increases radially. Under certain circumstances, it may be possible for a lens to exhibit both barrel and pincushion distortion at the same time. For example, the magnification may first decrease radially and then increase near the edge of the lens. Such distortion may be referred to a moustache distortion. Both the geometric distortion and the chromatic aberrations may degrade the quality of the resultant image provided by the ISP. Thus, by either fully or partially correcting the geometric distortion, the chromatic aberration, or both, smaller, thinner, and cheaper lenses may be used while maintain sufficient visual quality in the video and still frames produced by the camera.
[0761] FIG. 129 illustrates typical distortion curves for red, green, and blue color channels 1738, 1739, and 1740, respectively. As illustrated, the graph plots the distortion, sometimes referred to as displacement, versus an ideal undistorted radius. The distortion, as a percentage of the maximum radius, may be represented by the following equation:--, in [0759]-[0762]).
Re Claim 12, Cote further discloses generate output RGB image data of which each chrominance of each pixel has each corrected chrominance value (see COTE: e.g., --[1022] The horizontal chrominance scaling module is very similar to the horizontal luminance scalers 4566, 4568. The differences may be as follows: [1023] 1. The input to the horizontal chrominance scaler 4582, 4584 is interleaved Cb and Cr samples. Pairs of Cb/Cr samples are cosited and are cosited with the even luminance samples. [1024] 2. If one of the output channels is in 4:2:0 format, there will be half as many chrominance lines as luminance lines output by the channel.
[1025] The coordinate generation logic of the horizontal chrominance scaling logic 4582, 4584 may operate in substantially the same way as the coordinate generation logic of the horizontal luminance scalers 4566, 4568, with slight modifications to accommodate the differences discussed above. Namely, the corrected x-coordinate determined by comparing the displacement and the SourceX coordinate may be divided by 2 (since half as many chrominance samples may be present as luminance samples) to obtain the ultimate distortion-corrected x-coordinate.
[1026] Similarly, the horizontal chrominance resampling logic of the horizontal chrominance scaling logic 4582, 4584 may operate in substantially the same way as the horizontal luminance resampling logic of the horizontal luminance scalers 4566, 4568, with a few exceptions. FIG. 210 illustrates an example of the horizontal chrominance resampling logic of the horizontal chrominance scaling logic 4582, 4584. In the example of FIG. 210, elements 5122, 5124, 5126, 5128, 5130, 5132, 5134, 5136, 5138, 5140, 5142, 5144, and 5146 may respectively operate in the same general way as elements 5022, 5024, 5026, 5028, 5030, 5032, 5034, 5036, 5038, 5040, 5042, 5044, and 5046 of FIG. 209. The control logic 5122 may also differ in that it may control two pipelines of buffers rather than just one—buffers 5120 for Cb chrominance data and buffers 5121 for Cr chrominance data. Data from the two pipelines of buffers thus may be selected by a cr_select signal to multiplexers 5137.
[1027] Essentially, for each input line to the horizontal chrominance scaler 4582, 4584, consisting of “inWidth” samples (inWidth/2 Cb/Cr pairs), the horizontal chrominance coordinate generator component will generate “outWidth/2” X coordinates, one per Cb/Cr pair of output samples. These coordinates define the position of the (geometric-distortion-corrected) output sample relative to the (non-geometric-distortion-corrected, in the horizontal coordinate) input samples, where the position of the input sample is implicit in their numbering (0-inWidth/2−1). The horizontal chrominance coordinate generator produces two output values, “xpointer” and “xphase”.--, in [1022]-[1028], and, --[1033] The image data output by the YCC scaler 4012 thus may be scaled to one or two desired resolutions, while also correcting for geometric distortion. When the upper-left-hand portion of the input image data generally appears as in FIG. 131 (which has been corrected for chromatic aberration but not geometric distortion), the YCC scaler 4012 may produce an output image with the upper-left-hand image shown in FIG. 228.--, in [1033]); and
generate the output image by converting the output RGB image data into a Bayer pattern (see COTE: e.g., --[0229] The color filter array may be a Bayer color filter array, an example of which appears in FIG. 2. A Bayer color filter array provides a filter pattern that captures 50% green elements, 25% red elements, and 25% blue elements of light reaching the sensor. In the example of FIG. 2, 2 green elements (Gr and Gb), 1 red element (R), and 1 blue element (B) will repeat in the pattern shown across the full pixel array of the sensor(s) of the imaging device(s) 30. Thus, an image sensor with a Bayer color filter array may provide information regarding the intensity of the light received by the imaging device 30 at the green, red, and blue wavelengths, whereby each image pixel records only one of the three colors (RGB). This information, which may be referred to as “raw image data” or data in the “raw domain,” may be processed using one or more demosaicing techniques to convert the raw image data into a full color image, generally by interpolating a set of red, green, and blue values for each pixel. As will be discussed further below, such demosaicing techniques may be performed by the image processing circuitry 32.--, in [0229]-[0230]; and, --[0364] The defective pixel replacement (DPR) logic 474 may correct defective pixels by replacing them with other values before the pixels are considered in statistics collection in the statistics core 146a. With reference again to FIG. 48, it may be seen that the DPR logic 474 appears after the BLC logic 472. By performing defective pixel replacement after, rather than before, black level compensation, the black levels may be more accurately represented…0365] In one embodiment, defective pixel correction is performed independently for each color component (e.g., R, B, Gr, and Gb for a Bayer pattern). Generally, the DPR logic 474 may provide for dynamic defect correction, wherein the locations of defective pixels are determined automatically based upon directional gradients computed using neighboring pixels of the same color. ….if the stuck pixel is in a region of the current image that is dominated by black or darker colors, then the stuck pixel may be identified as a defective pixel during processing by the DPR logic 474 and corrected accordingly.--, in [0362]-[0365], and, --Thereafter, at step 566, a count C of the number of gradients that are less than or equal to a particular threshold dprTh is determined. As shown at decision logic 568, if C is less than or equal to dprMaxC, then the process 560 continues to step 570, and the current pixel is identified as being defective. The defective pixel is then corrected at step 572 using a replacement value.--, in [0370]and, --[1022] The horizontal chrominance scaling module is very similar to the horizontal luminance scalers 4566, 4568. The differences may be as follows: [1023] 1. The input to the horizontal chrominance scaler 4582, 4584 is interleaved Cb and Cr samples. Pairs of Cb/Cr samples are cosited and are cosited with the even luminance samples. [1024] 2. If one of the output channels is in 4:2:0 format, there will be half as many chrominance lines as luminance lines output by the channel.
[1025] The coordinate generation logic of the horizontal chrominance scaling logic 4582, 4584 may operate in substantially the same way as the coordinate generation logic of the horizontal luminance scalers 4566, 4568, with slight modifications to accommodate the differences discussed above. Namely, the corrected x-coordinate determined by comparing the displacement and the SourceX coordinate may be divided by 2 (since half as many chrominance samples may be present as luminance samples) to obtain the ultimate distortion-corrected x-coordinate.
[1026] Similarly, the horizontal chrominance resampling logic of the horizontal chrominance scaling logic 4582, 4584 may operate in substantially the same way as the horizontal luminance resampling logic of the horizontal luminance scalers 4566, 4568, with a few exceptions. FIG. 210 illustrates an example of the horizontal chrominance resampling logic of the horizontal chrominance scaling logic 4582, 4584. In the example of FIG. 210, elements 5122, 5124, 5126, 5128, 5130, 5132, 5134, 5136, 5138, 5140, 5142, 5144, and 5146 may respectively operate in the same general way as elements 5022, 5024, 5026, 5028, 5030, 5032, 5034, 5036, 5038, 5040, 5042, 5044, and 5046 of FIG. 209. The control logic 5122 may also differ in that it may control two pipelines of buffers rather than just one—buffers 5120 for Cb chrominance data and buffers 5121 for Cr chrominance data. Data from the two pipelines of buffers thus may be selected by a cr_select signal to multiplexers 5137.
[1027] Essentially, for each input line to the horizontal chrominance scaler 4582, 4584, consisting of “inWidth” samples (inWidth/2 Cb/Cr pairs), the horizontal chrominance coordinate generator component will generate “outWidth/2” X coordinates, one per Cb/Cr pair of output samples. These coordinates define the position of the (geometric-distortion-corrected) output sample relative to the (non-geometric-distortion-corrected, in the horizontal coordinate) input samples, where the position of the input sample is implicit in their numbering (0-inWidth/2−1). The horizontal chrominance coordinate generator produces two output values, “xpointer” and “xphase”.--, in [1022]-[1028], and, --[1033] The image data output by the YCC scaler 4012 thus may be scaled to one or two desired resolutions, while also correcting for geometric distortion. When the upper-left-hand portion of the input image data generally appears as in FIG. 131 (which has been corrected for chromatic aberration but not geometric distortion), the YCC scaler 4012 may produce an output image with the upper-left-hand image shown in FIG. 228.--, in [1033]).
Re Claim 13, Cote further discloses wherein the raw image is an image of a tetra pattern or an image of a nona pattern (see COTE: e.g., --[0229] The color filter array may be a Bayer color filter array, an example of which appears in FIG. 2. A Bayer color filter array provides a filter pattern that captures 50% green elements, 25% red elements, and 25% blue elements of light reaching the sensor. In the example of FIG. 2, 2 green elements (Gr and Gb), 1 red element (R), and 1 blue element (B) will repeat in the pattern shown across the full pixel array of the sensor(s) of the imaging device(s) 30. Thus, an image sensor with a Bayer color filter array may provide information regarding the intensity of the light received by the imaging device 30 at the green, red, and blue wavelengths, whereby each image pixel records only one of the three colors (RGB). This information, which may be referred to as “raw image data” or data in the “raw domain,” may be processed using one or more demosaicing techniques to convert the raw image data into a full color image, generally by interpolating a set of red, green, and blue values for each pixel. As will be discussed further below, such demosaicing techniques may be performed by the image processing circuitry 32.--, in [0229]-[0230]; and, --[0364] The defective pixel replacement (DPR) logic 474 may correct defective pixels by replacing them with other values before the pixels are considered in statistics collection in the statistics core 146a. With reference again to FIG. 48, it may be seen that the DPR logic 474 appears after the BLC logic 472. By performing defective pixel replacement after, rather than before, black level compensation, the black levels may be more accurately represented…0365] In one embodiment, defective pixel correction is performed independently for each color component (e.g., R, B, Gr, and Gb for a Bayer pattern). Generally, the DPR logic 474 may provide for dynamic defect correction, wherein the locations of defective pixels are determined automatically based upon directional gradients computed using neighboring pixels of the same color. ….if the stuck pixel is in a region of the current image that is dominated by black or darker colors, then the stuck pixel may be identified as a defective pixel during processing by the DPR logic 474 and corrected accordingly.--, in [0362]-[0365], and, --Thereafter, at step 566, a count C of the number of gradients that are less than or equal to a particular threshold dprTh is determined. As shown at decision logic 568, if C is less than or equal to dprMaxC, then the process 560 continues to step 570, and the current pixel is identified as being defective. The defective pixel is then corrected at step 572 using a replacement value.--, in [0370], and,-- where interp3 denotes a 3D interpolation function. Tetrahedral interpolation is used instead of tri-linear interpolation to generate smoother transitions at the input points of the grid.
[0881] To complete the tetrahedral interpolation, a hexahedron (cube) of 3D color LUT 3698 space may be divided into six tetrahedra, and the closest four points may be used to perform the interpolation. FIG. 182 illustrates this interpolation, in which a color point P is defined by vectors u, v, and w within a hexahedron representing the 3D color LUT 3698. The hexahedron representing the 3D color LUT 3698 may have outer points L, Lu, Lw, Luw, Lv, Luv, Lvw, and H. Each tetrahedron that subdivides the 3D color LUT 3698 may extend between point L and point H. The following equations may describe tetrahedral interpolation as shown in FIG. 182:--, in [0880]-[0881]; and,
--[1022] The horizontal chrominance scaling module is very similar to the horizontal luminance scalers 4566, 4568. The differences may be as follows: [1023] 1. The input to the horizontal chrominance scaler 4582, 4584 is interleaved Cb and Cr samples. Pairs of Cb/Cr samples are cosited and are cosited with the even luminance samples. [1024] 2. If one of the output channels is in 4:2:0 format, there will be half as many chrominance lines as luminance lines output by the channel.
[1025] The coordinate generation logic of the horizontal chrominance scaling logic 4582, 4584 may operate in substantially the same way as the coordinate generation logic of the horizontal luminance scalers 4566, 4568, with slight modifications to accommodate the differences discussed above. Namely, the corrected x-coordinate determined by comparing the displacement and the SourceX coordinate may be divided by 2 (since half as many chrominance samples may be present as luminance samples) to obtain the ultimate distortion-corrected x-coordinate.
[1026] Similarly, the horizontal chrominance resampling logic of the horizontal chrominance scaling logic 4582, 4584 may operate in substantially the same way as the horizontal luminance resampling logic of the horizontal luminance scalers 4566, 4568, with a few exceptions. FIG. 210 illustrates an example of the horizontal chrominance resampling logic of the horizontal chrominance scaling logic 4582, 4584. In the example of FIG. 210, elements 5122, 5124, 5126, 5128, 5130, 5132, 5134, 5136, 5138, 5140, 5142, 5144, and 5146 may respectively operate in the same general way as elements 5022, 5024, 5026, 5028, 5030, 5032, 5034, 5036, 5038, 5040, 5042, 5044, and 5046 of FIG. 209. The control logic 5122 may also differ in that it may control two pipelines of buffers rather than just one—buffers 5120 for Cb chrominance data and buffers 5121 for Cr chrominance data. Data from the two pipelines of buffers thus may be selected by a cr_select signal to multiplexers 5137.
[1027] Essentially, for each input line to the horizontal chrominance scaler 4582, 4584, consisting of “inWidth” samples (inWidth/2 Cb/Cr pairs), the horizontal chrominance coordinate generator component will generate “outWidth/2” X coordinates, one per Cb/Cr pair of output samples. These coordinates define the position of the (geometric-distortion-corrected) output sample relative to the (non-geometric-distortion-corrected, in the horizontal coordinate) input samples, where the position of the input sample is implicit in their numbering (0-inWidth/2−1). The horizontal chrominance coordinate generator produces two output values, “xpointer” and “xphase”.--, in [1022]-[1028], and, --[1033] The image data output by the YCC scaler 4012 thus may be scaled to one or two desired resolutions, while also correcting for geometric distortion. When the upper-left-hand portion of the input image data generally appears as in FIG. 131 (which has been corrected for chromatic aberration but not geometric distortion), the YCC scaler 4012 may produce an output image with the upper-left-hand image shown in FIG. 228.--, in [1033]).
Re Claims 14-17, claims 14-17 are the corresponding system claim to claims 1, and 5-7 respectively. Thus, claims 14-17 are rejected for the similar reasons as for claims 1, and 5-7. Furthermore, Cote further discloses an image processing system comprising: an image sensor comprising a pixel array, and an image signal processor (ISP) configured to perform the steps (see COTE: e.g., --a digital camera, to acquire image data to be processed in the image processing circuitry 32. The input structures 14 may enable user input to the electronic device, and may include hardware keys, a touch-sensitive element of the display 28, and/or a microphone. [0224] The processor(s) 16 may control the general operation of the device 10. For instance, the processor(s) 16 may execute an operating system, programs, user and application interfaces, and other functions of the electronic device 10.--, in [0223]-[0224], and, --[0227] The imaging device(s) 30 of the electronic device 10 may represent a digital camera that may acquire both still images and video. Each imaging device 30 may include a lens and an image sensor capture and convert light into electrical signals. By way of example, the image sensor may include a CMOS image sensor (e.g., a CMOS active-pixel sensor (APS)) or a CCD (charge-coupled device) sensor. Generally, the image sensor of the imaging device 30 includes an integrated circuit with an array of photodetectors. The array of photodetectors may detect the intensity of light captured at specific locations on the sensor. Photodetectors are generally only able to capture intensity, however, and may not detect the particular wavelength of the captured light.
[0228] Accordingly, the image sensor may include a color filter array (CFA) that may overlay the pixel array of the image sensor to capture color information. The color filter array may include an array of small color filters, each of which may overlap a respective location—namely, a picture element, or pixel—of the image sensor and filter the captured light by wavelength. Thus, together, the color filter array and the photodetectors may detect both the wavelength and intensity of light through the lens. The resulting image information may represent a frame of raw image data.--, in [0227]-[0229], and, --[0734] As also depicted in FIG. 122, one effect of the binning process is that the spatial sampling of the binned pixels may not be equally spaced. This spatial distortion may, in some systems, result in aliasing (e.g., jagged edges), which is generally not desirable. Further, because certain image processing steps in the ISP pipe logic 82 may depend upon on the linear placement of the color samples in order to operate correctly, the raw scaler logic 1040 may be applied to perform re-sampling and re-positioning of the binned pixels such that the binned pixels are spatially evenly distributed.--, in [0733]-[0734]).
Re Claims 18-20, are the corresponding method claim to claims 1, 5 and 9 respectively. Thus, claims 18-20 are rejected for the similar reasons as for claims 1, 5 and 9. Furthermore, Cote further discloses image processing method (see COTE: e.g., --a digital camera, to acquire image data to be processed in the image processing circuitry 32. The input structures 14 may enable user input to the electronic device, and may include hardware keys, a touch-sensitive element of the display 28, and/or a microphone. [0224] The processor(s) 16 may control the general operation of the device 10. For instance, the processor(s) 16 may execute an operating system, programs, user and application interfaces, and other functions of the electronic device 10.--, in [0223]-[0224], and, --[0227] The imaging device(s) 30 of the electronic device 10 may represent a digital camera that may acquire both still images and video. Each imaging device 30 may include a lens and an image sensor capture and convert light into electrical signals. By way of example, the image sensor may include a CMOS image sensor (e.g., a CMOS active-pixel sensor (APS)) or a CCD (charge-coupled device) sensor. Generally, the image sensor of the imaging device 30 includes an integrated circuit with an array of photodetectors. The array of photodetectors may detect the intensity of light captured at specific locations on the sensor. Photodetectors are generally only able to capture intensity, however, and may not detect the particular wavelength of the captured light.
[0228] Accordingly, the image sensor may include a color filter array (CFA) that may overlay the pixel array of the image sensor to capture color information. The color filter array may include an array of small color filters, each of which may overlap a respective location—namely, a picture element, or pixel—of the image sensor and filter the captured light by wavelength. Thus, together, the color filter array and the photodetectors may detect both the wavelength and intensity of light through the lens. The resulting image information may represent a frame of raw image data.--, in [0227]-[0229], and, --[0734] As also depicted in FIG. 122, one effect of the binning process is that the spatial sampling of the binned pixels may not be equally spaced. This spatial distortion may, in some systems, result in aliasing (e.g., jagged edges), which is generally not desirable. Further, because certain image processing steps in the ISP pipe logic 82 may depend upon on the linear placement of the color samples in order to operate correctly, the raw scaler logic 1040 may be applied to perform re-sampling and re-positioning of the binned pixels such that the binned pixels are spatially evenly distributed.--, in [0733]-[0734]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEI WEN YANG whose telephone number is (571)270-5670. The examiner can normally be reached on 8:00 - 5:00 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEI WEN YANG/Primary Examiner, Art Unit 2662
Prosecution Insights