DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-20 are pending for examination in the application filed 03/22/2024.
Priority
Acknowledgement is made of Applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent application KR10-2023-0038184 filed on 03/23/2023.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/22/2024 has been considered by the examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 15, and 20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Malvar (US20080240559A1).
Regarding claim 1, Malvar teaches an image signal processor comprising: a stripe pattern determiner configured to calculate a gradient index for each of a plurality of directions in a target kernel, and determine whether the target kernel corresponds to a stripe pattern based on the gradient index for each of the plurality of directions (Fig. 9 [0039] In general, referring to FIG. 2, the adaptive filter interpolation architecture 200 inputs color information including raw, mosaic-patterned pixel data 202 produced by the image sensor array 204, processes the information, and outputs one or more missing color values for each pixel 230. More specifically, in one embodiment, the adaptive interpolation with artifact reduction architecture 200 includes a current pixel selector 206 that selects a current pixel having a current color from the raw pixel data 202 (e.g. an image array that represents a Bayer image). A gradient computation module 208 computes a gradient for both the horizontal and vertical directions at the selected pixel). [0064] for example, if the image has vertical stripes that change rapidly in intensity from a column to another (that is, with significant signal energy at the highest possible horizontal spatial frequency), the horizontal activity metric above may end up having a small value, as if it were in a nearly flat background region. Thus, in one embodiment, the adaptive interpolation with artifact reduction technique described herein includes an important variation: the technique first computes the horizontal and vertical estimates for the missing colors at the current pixel. Then, it includes in the activity estimate a measure the absolute difference between that interpolated value and its immediate neighbors);
a pattern direction determiner configured to determine a pattern direction of the target kernel when the target kernel corresponds to the stripe pattern; and a demosaicing component configured to perform an interpolation operation based on the pattern direction of the target kernel ([0007] The technique chooses the best interpolation direction, from one of three options: horizontal, vertical, or non-directed, based on the values of the horizontal and vertical gradients and the horizontal and vertical activity metrics. [0065] A key point for the reduction of demosaicing artifacts is choosing well in which direction to interpolate…A key idea in the adaptive interpolation with artifact reduction technique is to measure "high-frequency activity", or simply activity, as a metric of how fast pixel values are changing…for example, if the image has vertical stripes that change rapidly in intensity from a column to another (that is, with significant signal energy at the highest possible horizontal spatial frequency), the horizontal activity metric above may end up having a small value, as if it were in a nearly flat background region. Thus, in one embodiment, the adaptive interpolation with artifact reduction technique described herein includes an important variation: the technique first computes the horizontal and vertical estimates for the missing colors at the current pixel. Then, it includes in the activity estimate a measure the absolute difference between that interpolated value and its immediate neighbors…Additionally, seeking to interpolate in the direction of lower activity favors choosing the output interpolated value (from H or V directions) that leads to a smoother output image, thus directly addressing the reduction of artifacts in the output image. [0066] The technique computes horizontal and vertical activity measures, AH and AV respectively, using pixel values from the positions marked Og and Gy in FIG. 7. For pixels in a red or blue (i,j) location 702, 704, the technique computes the horizontal and vertical activity metrics (AH 706 and AV 708) by
PNG
media_image1.png
587
716
media_image1.png
Greyscale
Regarding claim 15, Malvar teaches an image signal processing method comprising: calculating a gradient index for each of a plurality of directions in a target kernel; determining whether the target kernel corresponds to a stripe pattern based on the gradient index for each of the plurality of directions ([0039] In general, referring to FIG. 2, the adaptive filter interpolation architecture 200 inputs color information including raw, mosaic-patterned pixel data 202 produced by the image sensor array 204, processes the information, and outputs one or more missing color values for each pixel 230. More specifically, in one embodiment, the adaptive interpolation with artifact reduction architecture 200 includes a current pixel selector 206 that selects a current pixel having a current color from the raw pixel data 202 (e.g. an image array that represents a Bayer image). A gradient computation module 208 computes a gradient for both the horizontal and vertical directions at the selected pixel). [0064] for example, if the image has vertical stripes that change rapidly in intensity from a column to another (that is, with significant signal energy at the highest possible horizontal spatial frequency), the horizontal activity metric above may end up having a small value, as if it were in a nearly flat background region. Thus, in one embodiment, the adaptive interpolation with artifact reduction technique described herein includes an important variation: the technique first computes the horizontal and vertical estimates for the missing colors at the current pixel. Then, it includes in the activity estimate a measure the absolute difference between that interpolated value and its immediate neighbors);
determining a pattern direction of the target kernel when the target kernel corresponds to the stripe pattern; and performing an interpolation operation based on the pattern direction of the target kernel ([0007] The technique chooses the best interpolation direction, from one of three options: horizontal, vertical, or non-directed, based on the values of the horizontal and vertical gradients and the horizontal and vertical activity metrics. [0065] A key point for the reduction of demosaicing artifacts is choosing well in which direction to interpolate…A key idea in the adaptive interpolation with artifact reduction technique is to measure "high-frequency activity", or simply activity, as a metric of how fast pixel values are changing…for example, if the image has vertical stripes that change rapidly in intensity from a column to another (that is, with significant signal energy at the highest possible horizontal spatial frequency), the horizontal activity metric above may end up having a small value, as if it were in a nearly flat background region. Thus, in one embodiment, the adaptive interpolation with artifact reduction technique described herein includes an important variation: the technique first computes the horizontal and vertical estimates for the missing colors at the current pixel. Then, it includes in the activity estimate a measure the absolute difference between that interpolated value and its immediate neighbors…Additionally, seeking to interpolate in the direction of lower activity favors choosing the output interpolated value (from H or V directions) that leads to a smoother output image, thus directly addressing the reduction of artifacts in the output image. [0066] The technique computes horizontal and vertical activity measures, AH and AV respectively, using pixel values from the positions marked Og and Gy in FIG. 7. For pixels in a red or blue (i,j) location 702, 704, the technique computes the horizontal and vertical activity metrics (AH 706 and AV 708) by
PNG
media_image1.png
587
716
media_image1.png
Greyscale
Regarding claim 20, Malvar teaches an electronic device comprising: a processor; and a memory storing instructions executed by the processor, wherein the instructions, when executed by the processor, cause the electronic device to (Fig 9):
calculate a gradient index for each of a plurality of directions in a target kernel; determine whether the target kernel corresponds to a stripe pattern based on the gradient index for each of the plurality of directions ([0039] In general, referring to FIG. 2, the adaptive filter interpolation architecture 200 inputs color information including raw, mosaic-patterned pixel data 202 produced by the image sensor array 204, processes the information, and outputs one or more missing color values for each pixel 230. More specifically, in one embodiment, the adaptive interpolation with artifact reduction architecture 200 includes a current pixel selector 206 that selects a current pixel having a current color from the raw pixel data 202 (e.g. an image array that represents a Bayer image). A gradient computation module 208 computes a gradient for both the horizontal and vertical directions at the selected pixel). [0064] for example, if the image has vertical stripes that change rapidly in intensity from a column to another (that is, with significant signal energy at the highest possible horizontal spatial frequency), the horizontal activity metric above may end up having a small value, as if it were in a nearly flat background region. Thus, in one embodiment, the adaptive interpolation with artifact reduction technique described herein includes an important variation: the technique first computes the horizontal and vertical estimates for the missing colors at the current pixel. Then, it includes in the activity estimate a measure the absolute difference between that interpolated value and its immediate neighbors);
determine a pattern direction of the target kernel when the target kernel corresponds to the stripe pattern; and perform an interpolation operation based on the pattern direction of the target kernel ([0007] The technique chooses the best interpolation direction, from one of three options: horizontal, vertical, or non-directed, based on the values of the horizontal and vertical gradients and the horizontal and vertical activity metrics. [0065] A key point for the reduction of demosaicing artifacts is choosing well in which direction to interpolate…A key idea in the adaptive interpolation with artifact reduction technique is to measure "high-frequency activity", or simply activity, as a metric of how fast pixel values are changing…for example, if the image has vertical stripes that change rapidly in intensity from a column to another (that is, with significant signal energy at the highest possible horizontal spatial frequency), the horizontal activity metric above may end up having a small value, as if it were in a nearly flat background region. Thus, in one embodiment, the adaptive interpolation with artifact reduction technique described herein includes an important variation: the technique first computes the horizontal and vertical estimates for the missing colors at the current pixel. Then, it includes in the activity estimate a measure the absolute difference between that interpolated value and its immediate neighbors…Additionally, seeking to interpolate in the direction of lower activity favors choosing the output interpolated value (from H or V directions) that leads to a smoother output image, thus directly addressing the reduction of artifacts in the output image. [0066] The technique computes horizontal and vertical activity measures, AH and AV respectively, using pixel values from the positions marked Og and Gy in FIG. 7. For pixels in a red or blue (i,j) location 702, 704, the technique computes the horizontal and vertical activity metrics (AH 706 and AV 708) by
PNG
media_image1.png
587
716
media_image1.png
Greyscale
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 2-3 are rejected under 35 U.S.C. 103 as being unpatentable over Malvar in view of Guo (CN115147278A).
Regarding claim 2, Malvar teaches the image signal processor of claim 1. Malvar further teaches wherein the target kernel includes pixel data of a plurality of pixels arranged according to an RGB pattern (Fig 5-7).
Malvar does not teach a plurality of pixels arranged according to an RGBW pattern.
Guo, in the same field of endeavor of image pixel processing, teaches a plurality of pixels arranged according to an RGBW pattern ([Abstract] the method comprising: performing W interpolation on the HexRGBW data obtained by the HexRGBW configuration colour filter array, generating a full resolution W image; performing a first sampling process according to the full resolution W image and the HexRGBW data, generating a half-resolution W image and a half-resolution RGB image; performing RGB interpolation processing to the half-resolution RGB image according to the half-resolution W image, generating the R image of the half-resolution, G image, B image; according to the R image of the half-resolution, G image, B image for the second sampling process, generating RGB image of full resolution. The embodiment of the application is capable of de-mosaic processing the colour filter array of HexRGBW configuration).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the image signal processor of Malvar with the teachings of Guo to use an RGBW pattern because "the color filter array of HexRGBW configuration has more brightness/gray scale information, so that it can keep the correct color saturation under the environment of low illumination, at the same time, it can accurately sense the gray scale information. Therefore, how to demosaic the color filter array of HexRGBW configuration, becoming a technical problem to be solved urgently" [Guo pg. 2 para. 2].
Regarding claim 3, Malvar and Guo teach the image signal processor of claim 2. Guo teaches wherein the RGBW pattern includes a pixel group formed in a (2×2) matrix including white pixels arranged in a first diagonal direction of the plurality of directions and color pixels arranged in a second diagonal direction of the plurality of directions ([pg. 7 para. 4 and 6] Specifically, as shown in FIG. 7, the application value in the diagonal direction of the embodiment is average to obtain the half-resolution W image of the down-sampling. the full resolution W image comprises a plurality of 2x2 pixel point sub-array, each 2x2 pixel point sub-array corresponding to one pixel point in the half-resolution W image. Specifically, as shown in FIG. 8, the application image of the half-resolution obtained by averaging the R, G, B values in the diagonal direction is Quad bayer image. HexRGBW data also comprises a plurality of 2x2 pixel point sub-array. in the 2x2 pixel point sub-array, the pixel point of the left lower corner and the right upper corner is a white pixel, the pixel points of the left upper corner and the right lower corner are red pixel points or green pixel points or blue pixel points. according to the arithmetic average of the pixel points of the left upper corner and the right lower corner as the pixel value of the corresponding pixel point in the RGB image of the half resolution).
PNG
media_image2.png
552
542
media_image2.png
Greyscale
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the image signal processor of Malvar with the teachings of Guo to use a 2x2 matrix of white pixels arranged in a first diagonal direction and color pixels arranged in a second diagonal direction because "the color filter array of HexRGBW configuration has more brightness/gray scale information, so that it can keep the correct color saturation under the environment of low illumination, at the same time, it can accurately sense the gray scale information. Therefore, how to demosaic the color filter array of HexRGBW configuration, becoming a technical problem to be solved urgently" [Guo pg. 2 para. 2].
Claims 4-5 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Malvar in view of Rebiere (US20210297610A1).
Regarding claim 4, Malvar teaches the image signal processor of claim 1. Malvar further teaches wherein: the plurality of directions includes a horizontal direction, a vertical direction, and gradient indices for the plurality of directions include a first gradient index for the horizontal direction, a second gradient index for the vertical direction ([0039] A gradient computation module 208 computes a gradient for both the horizontal and vertical directions at the selected pixel. An activity computation module 210 computes a horizontal activity metric for the horizontal 212 direction and a vertical activity metric for the vertical 214 direction that take into account candidate output interpolated pixels. A filter selection module 216 then selects horizontal 220, vertical 222 or non-directional 224 filters from a pre-determined set of filters 218 using the computed gradients and the vertical 214 and horizontal activity metrics 212).
Malvar does not teach a first diagonal direction, and a second diagonal direction, a third gradient index for the first diagonal direction, and a fourth gradient index for the second diagonal direction.
Rebiere, in the same field of endeavor of pixel interpolation, teaches a first diagonal direction, and a second diagonal direction, a third gradient index for the first diagonal direction, and a fourth gradient index for the second diagonal direction ([0031] Preferably, eight semi-gradients are calculated for eight given directions along four axes passing through the location of the missing or incorrect value. Advantageously, these four axes are one horizontal axis, one vertical axis, and two diagonal axes).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the image signal processor of Malvar with the teachings of Rebiere to use a first diagonal direction and a second diagonal direction and a third and fourth gradient index because "each semi-gradient being associated with a given direction with respect to the location of the missing or incorrect value to be determined…makes it possible to enhance the estimation of the missing or incorrect pixel. The semi-gradients provide information relating to the probability that the missing or incorrect value to be estimated belongs to one side of the kernel" [Rebiere 0029-0030].
Regarding claim 5, Malvar and Rebiere teach the image signal processor of claim 4. Malvar further teaches wherein each of the gradient indices is a sum of differences in pixel data between homogeneous pixels arranged adjacent to each other in a corresponding direction in the target kernel ([0065] A key idea in the adaptive interpolation with artifact reduction technique is to measure "high-frequency activity", or simply activity, as a metric of how fast pixel values are changing. The previous technique uses a simple gradient-like metric such as horizontal activity=abs{2*x[i, j]-x(i,j-2)-x(i,j+2)}, in which it is important to measure the difference two pixels away, so that changes in pixel values of the same color were compared. However, measuring the jump of two positions leads to aliasing problems at high frequencies; for example, if the image has vertical stripes that change rapidly in intensity from a column to another (that is, with significant signal energy at the highest possible horizontal spatial frequency), the horizontal activity metric above may end up having a small value, as if it were in a nearly flat background region. Thus, in one embodiment, the adaptive interpolation with artifact reduction technique described herein includes an important variation: the technique first computes the horizontal and vertical estimates for the missing colors at the current pixel. Then, it includes in the activity estimate a measure the absolute difference between that interpolated value and its immediate neighbors. Including deltas with only one pixel difference reduces the aliasing problems just discussed. Additionally, seeking to interpolate in the direction of lower activity favors choosing the output interpolated value (from H or V directions) that leads to a smoother output image, thus directly addressing the reduction of artifacts in the output image).
PNG
media_image3.png
591
588
media_image3.png
Greyscale
Malvar does not teach the first to fourth gradient indices.
Rebiere, in the same field of endeavor of pixel interpolation, teaches the first to fourth gradient indices.
PNG
media_image4.png
1
415
media_image4.png
Greyscale
PNG
media_image4.png
1
415
media_image4.png
Greyscale
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the image signal processor of Malvar with the teachings of Rebiere to use first to fourth gradient indices because "each semi-gradient being associated with a given direction with respect to the location of the missing or incorrect value to be determined…makes it possible to enhance the estimation of the missing or incorrect pixel. The semi-gradients provide information relating to the probability that the missing or incorrect value to be estimated belongs to one side of the kernel" [Rebiere 0029-0030].
Regarding claim 16, Malvar teaches the method of claim 15. Malvar further teaches wherein: the plurality of directions includes a horizontal direction, a vertical direction, and gradient indices for the plurality of directions include a first gradient index for the horizontal direction, a second gradient index for the vertical direction ([0039] A gradient computation module 208 computes a gradient for both the horizontal and vertical directions at the selected pixel. An activity computation module 210 computes a horizontal activity metric for the horizontal 212 direction and a vertical activity metric for the vertical 214 direction that take into account candidate output interpolated pixels. A filter selection module 216 then selects horizontal 220, vertical 222 or non-directional 224 filters from a pre-determined set of filters 218 using the computed gradients and the vertical 214 and horizontal activity metrics 212).
Malvar does not teach a first diagonal direction, and a second diagonal direction, a third gradient index for the first diagonal direction, and a fourth gradient index for the second diagonal direction.
Rebiere, in the same field of endeavor of pixel interpolation, teaches a first diagonal direction, and a second diagonal direction, a third gradient index for the first diagonal direction, and a fourth gradient index for the second diagonal direction ([0031] Preferably, eight semi-gradients are calculated for eight given directions along four axes passing through the location of the missing or incorrect value. Advantageously, these four axes are one horizontal axis, one vertical axis, and two diagonal axes).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Malvar with the teachings of Rebiere to use a first diagonal direction and a second diagonal direction and a third and fourth gradient index because "each semi-gradient being associated with a given direction with respect to the location of the missing or incorrect value to be determined…makes it possible to enhance the estimation of the missing or incorrect pixel. The semi-gradients provide information relating to the probability that the missing or incorrect value to be estimated belongs to one side of the kernel" [Rebiere 0029-0030].
Claims 6, 8, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Malvar in view of Rebiere and Saini (US20170223383A1).
Regarding claim 6, Malvar and Rebiere teach the image signal processor of claim 4. Malvar does not teach wherein the stripe pattern determiner is configured to determine whether the target kernel corresponds to the stripe pattern depending on whether a stripe condition based on the first to fourth gradient indices is satisfied.
Rebiere teaches the first to fourth gradient indices.
PNG
media_image4.png
1
415
media_image4.png
Greyscale
PNG
media_image4.png
1
415
media_image4.png
Greyscale
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the image signal processor of Malvar with the teachings of Rebiere to use first to fourth gradient indices because "each semi-gradient being associated with a given direction with respect to the location of the missing or incorrect value to be determined…makes it possible to enhance the estimation of the missing or incorrect pixel. The semi-gradients provide information relating to the probability that the missing or incorrect value to be estimated belongs to one side of the kernel" [Rebiere 0029-0030].
Saini, in the same field of endeavor of stripe determination, teaches wherein the stripe pattern determiner is configured to determine whether the target kernel corresponds to the stripe pattern depending on whether a stripe condition based on the gradient indices is satisfied ([0063] In an example embodiment, edges that may correspond to the erroneous regions and/or stripes may be determined by the electronic device 104 in order to detect a candidate region. At step 308, vertical gradient may be calculated on the eroded TMB to determine the horizontal edges that correspond to the potential erroneous region and/or stripe. The vertical gradient may be calculated by taking pixel-by-pixel absolute difference of adjacent rows of the eroded TMB. The vertical gradient may be determined based on the equation (1): V(x,y) = abs(I(x,y)-I(x,y-1)). [0064] where, V(x, y) corresponds to a vertical gradient at a column ‘x’ and a row ‘y’ of the eroded TMB, and I(x, y) corresponds to an intensity value at a column ‘x’ and a row ‘y’ of the eroded TMB. The eroded TMB after determination of the vertical gradient may be referred to a Difference Image Vertical Gradient Buffer (DIVGB). In an aspect, the region between the validated edges within DIVGB where the values of the pixels is less than a fourth predefined threshold such as “5”, may correspond to a potential candidate region. [0067] In an example embodiment, instead of using one or more horizontal rows for detection of candidate region, one or more vertical columns may be used. In another embodiment, a combination of a first pair of a horizontal row and a vertical row, and a second pair of horizontal row and a vertical row, may be used for detection of the candidate region, in accordance with the foregoing description).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the image signal processor of Malvar with the teachings of Saini to determine that a kernel corresponds to a stripe pattern depending on whether a condition based on the gradient indices is satisfied because "During transmission of a video stream over a lossy transmission channel, data corresponding to one or more regions of the video stream may be lost. When such a video with lost data is decoded by a decoder at the receiver, the rendered video content may be riddled with one or more artifacts such as blocking artifacts, corrupted content, and the like. For example, when multiple regions of data are lost while transmission, the rendered video may comprise gray regions that correspond to the lost regions…The data corresponding to the lost region may be retrieved based on directional interpolation in the direction of the calculated edge orientations". [Saini 0003-0004].
Regarding claim 8, Malvar and Rebiere teach the image signal processor of claim 4. Malvar does not teach further comprising an interpolation direction determiner configured to determine an interpolation direction by comparing the first to fourth gradient indices with each other, when the target kernel does not correspond to the stripe pattern.
Rebiere teaches the first to fourth gradient indices.
PNG
media_image4.png
1
415
media_image4.png
Greyscale
PNG
media_image4.png
1
415
media_image4.png
Greyscale
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the image signal processor of Malvar with the teachings of Rebiere to use first to fourth gradient indices because "each semi-gradient being associated with a given direction with respect to the location of the missing or incorrect value to be determined…makes it possible to enhance the estimation of the missing or incorrect pixel. The semi-gradients provide information relating to the probability that the missing or incorrect value to be estimated belongs to one side of the kernel" [Rebiere 0029-0030].
Saini, in the same field of endeavor of stripe determination, teaches further comprising an interpolation direction determiner configured to determine an interpolation direction by comparing the gradient indices with each other (Fig. 5 step 512, compare first count and second count), when the target kernel does not correspond to the stripe pattern (Fig. 5 step 508, count is greater than gradient threshold) ([0035] In an aspect, the one or more characteristics may correspond to a formation of an edge that corresponds to the detected candidate region, in the current image and/or the previously decoded image. In such a scenario, the electronic device 104 may determine a gradient image of the current image and a gradient image of the previously decoded image. The electronic device 104 may further determine a first region in the gradient image of the current image corresponding to one or more edges of the detected candidate region and a second region in the gradient image of the previously decoded image corresponding to the one or more edges of the detected candidate region. The electronic device 104. The electronic device 104 may further determine a first count of pixels for which a gradient value in the pair of edges of the determined first region of the gradient image of the current image is greater than a gradient threshold. The electronic device 104 may further determine a second count of pixels for which a gradient value in the pair of edges of the determined second region of the gradient image of the previously decoded image is greater than the gradient threshold. The electronic device 104 may further validate detected candidate region as the artifact if the first count of pixels is greater than the second count of pixels by a fifth predefined threshold, as explained in FIG. 5. [0063] In an example embodiment, edges that may correspond to the erroneous regions and/or stripes may be determined by the electronic device 104 in order to detect a candidate region…In an aspect, the region between the validated edges within DIVGB where the values of the pixels is less than a fourth predefined threshold such as “5”, may correspond to a potential candidate region. [0004] The data corresponding to the lost region may be retrieved based on directional interpolation in the direction of the calculated edge orientations).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the image signal processor of Malvar with the teachings of Saini to determine interpolation direction by comparing gradient indices when the target kernel does not correspond to the stripe because "The determined candidate region that comprises the erroneous regions and/or stripes may be validated in order to detect the artifacts. The artifacts may be detected based on a comparison of one or more characteristics of the detected candidate region with one or more characteristics of the current image and/or one or more previously decoded images". [Saini 0034].
Regarding claim 19, Malvar and Rebiere teach the method of claim 16. Malvar does not teach further comprising an interpolation direction determiner configured to determine an interpolation direction by comparing the first to fourth gradient indices with each other, when the target kernel does not correspond to the stripe pattern.
Rebiere teaches the first to fourth gradient indices.
PNG
media_image4.png
1
415
media_image4.png
Greyscale
PNG
media_image4.png
1
415
media_image4.png
Greyscale
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Malvar with the teachings of Rebiere to use first to fourth gradient indices because "each semi-gradient being associated with a given direction with respect to the location of the missing or incorrect value to be determined…makes it possible to enhance the estimation of the missing or incorrect pixel. The semi-gradients provide information relating to the probability that the missing or incorrect value to be estimated belongs to one side of the kernel" [Rebiere 0029-0030].
Saini, in the same field of endeavor of stripe determination, teaches further comprising determining an interpolation direction by comparing the gradient indices with each other (Fig. 5 step 512, compare first count and second count), when the target kernel does not correspond to the stripe pattern (Fig. 5 step 508, count is greater than gradient threshold) ([0035] In an aspect, the one or more characteristics may correspond to a formation of an edge that corresponds to the detected candidate region, in the current image and/or the previously decoded image. In such a scenario, the electronic device 104 may determine a gradient image of the current image and a gradient image of the previously decoded image. The electronic device 104 may further determine a first region in the gradient image of the current image corresponding to one or more edges of the detected candidate region and a second region in the gradient image of the previously decoded image corresponding to the one or more edges of the detected candidate region. The electronic device 104. The electronic device 104 may further determine a first count of pixels for which a gradient value in the pair of edges of the determined first region of the gradient image of the current image is greater than a gradient threshold. The electronic device 104 may further determine a second count of pixels for which a gradient value in the pair of edges of the determined second region of the gradient image of the previously decoded image is greater than the gradient threshold. The electronic device 104 may further validate detected candidate region as the artifact if the first count of pixels is greater than the second count of pixels by a fifth predefined threshold, as explained in FIG. 5. [0063] In an example embodiment, edges that may correspond to the erroneous regions and/or stripes may be determined by the electronic device 104 in order to detect a candidate region…In an aspect, the region between the validated edges within DIVGB where the values of the pixels is less than a fourth predefined threshold such as “5”, may correspond to a potential candidate region. [0004] The data corresponding to the lost region may be retrieved based on directional interpolation in the direction of the calculated edge orientations).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Malvar with the teachings of Saini to determine interpolation direction by comparing gradient indices when the target kernel does not correspond to the stripe because "The determined candidate region that comprises the erroneous regions and/or stripes may be validated in order to detect the artifacts. The artifacts may be detected based on a comparison of one or more characteristics of the detected candidate region with one or more characteristics of the current image and/or one or more previously decoded images". [Saini 0034].
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Malvar in view of Takahashi (US20210099682A1).
Regarding claim 9, Malvar teaches the image signal processor of claim 1. Malvar does not teach wherein the stripe pattern includes one of: a horizontal stripe pattern having a horizontal pattern direction; and a vertical stripe pattern having a vertical pattern direction.
Takahashi, in the same field of endeavor of pixel interpolation, teaches wherein the stripe pattern includes one of: a horizontal stripe pattern having a horizontal pattern direction; and a vertical stripe pattern having a vertical pattern direction ([0019] The test pattern is assumed to be a pattern image having horizontal stripes or a pattern image having vertical stripes, for example. In this case, a direction of correlation of pixel signal values (correlation direction) is the horizontal direction in a case of the pattern image having the horizontal stripes and is the vertical direction in a case of the pattern image having the vertical stripes. The interpolation direction information is information indicating such a correlation direction).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the image signal processor of Malvar with the teachings of Takahashi to include a horizontal stripe pattern having a horizontal pattern direction and a vertical stripe pattern having a vertical pattern direction because "the interpolation processor may perform vertical interpolation processing using pixels aligned in the vertical direction in a case where the interpolation direction information indicates the vertical direction and performs horizontal interpolation processing using pixels aligned in the horizontal direction in a case where the interpolation direction information indicates the horizontal direction" [Takahashi 0020].
Allowable Subject Matter
Claims 7, 10-14, and 17-18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claims 7 and 17, Rebiere teaches the first to fourth gradient indices.
PNG
media_image4.png
1
415
media_image4.png
Greyscale
PNG
media_image4.png
1
415
media_image4.png
Greyscale
Saini teaches wherein the stripe pattern determiner is configured to determine whether the target kernel corresponds to the stripe pattern depending on whether a stripe condition based on the gradient indices is satisfied ([0063] In an example embodiment, edges that may correspond to the erroneous regions and/or stripes may be determined by the electronic device 104 in order to detect a candidate region. At step 308, vertical gradient may be calculated on the eroded TMB to determine the horizontal edges that correspond to the potential erroneous region and/or stripe. The vertical gradient may be calculated by taking pixel-by-pixel absolute difference of adjacent rows of the eroded TMB. The vertical gradient may be determined based on the equation (1): V(x,y) = abs(I(x,y)-I(x,y-1)). [0064] where, V(x, y) corresponds to a vertical gradient at a column ‘x’ and a row ‘y’ of the eroded TMB, and I(x, y) corresponds to an intensity value at a column ‘x’ and a row ‘y’ of the eroded TMB. The eroded TMB after determination of the vertical gradient may be referred to a Difference Image Vertical Gradient Buffer (DIVGB). In an aspect, the region between the validated edges within DIVGB where the values of the pixels is less than a fourth predefined threshold such as “5”, may correspond to a potential candidate region. [0067] In an example embodiment, instead of using one or more horizontal rows for detection of candidate region, one or more vertical columns may be used. In another embodiment, a combination of a first pair of a horizontal row and a vertical row, and a second pair of horizontal row and a vertical row, may be used for detection of the candidate region, in accordance with the foregoing description).
The following limitations of claims 7 and 17 were not found to be taught in the art: wherein the stripe condition is a condition in which the third gradient index is greater than a value obtained by multiplying a sum of the first gradient index and the second gradient index by a preset weight, and the fourth gradient index is greater than a value obtained by multiplying a sum of the first gradient index and the second gradient index by the preset weight.
Regarding claims 10-14 and 18, the following limitations of claims 10 and 18 were not found to be taught in the art: wherein the pattern direction determiner is configured to: calculate a first color ratio index indicating a degree to which a color ratio between pixel data values of color and white pixels belonging to an upper row within a horizontal pixel block of the target kernel and another color ratio between pixel data values of color and white pixels belonging to a lower row within the horizontal pixel block of the target kernel, are maintained; and calculate a second color ratio index indicating a degree to which a color ratio between pixel data values of color and white pixels belonging to a left column within a vertical pixel block of the target kernel and another color ratio between pixel data values of color and white pixels belonging to a right column within the vertical pixel block of the target kernel, are maintained. Claims 11-14 depend from claim 10 and therefore also include allowable subject matter.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Saito (US20120257821A1) teaches RGBW pixel interpolation according to edge direction.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jacqueline R Zak whose telephone number is (571)272-4077. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACQUELINE R ZAK/Examiner, Art Unit 2666
/EMILY C TERRELL/Supervisory Patent Examiner, Art Unit 2666