DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/05/2026 has been entered. Claims 1-20 remain pending in the application.
Response to Arguments
Applicant's arguments have been fully considered but are moot because they do not apply to the new combination of references being used in the current rejection, as necessitated by amendment to the claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Trejo et al (US 2020/0211161), in view of Pinhasov et al (US 2022/0060619), and further in view of Wang et al (US 2019/0301861).
Regarding Claim 1, Trejo teaches a method, comprising: for a first portion of an image, calculating a weighted metric ([0134], for a pixel, the adaptive spatial noise filter adjusts the weights used in the filter not only based on the Euclidean distance of adjacent pixels from the pixel, but also based on differences in signal levels between the adjacent pixels and the current pixel and based on the noise component of the current pixel. Specifically, as explained more completely below, adaptive spatial noise filter 570 takes a weighted sum of the pixels in a neighborhood of the current pixel. The weights depend on the spatial distance and on the signal levels of the neighboring pixels as well as the noise component of the current pixel itself. Thus, the signal value at each pixel in a frame is replaced by a noise-based weighted average of signal values from nearby pixels)
based on a plurality of second portions of the image ([0022], adaptive noise filter is configured to determine a difference between the signal level of the first pixel and the signal level of each of a plurality of neighboring pixels. The adaptive spatial noise filter also uses a noise dependent signal level parameter, where the noise dependent signal level parameter is a function of the noise of the first pixel, [0075], When adaptive noise filter 210 is an adaptive spatial noise filter, a value of a current pixel at a location in a current frame of demosiaced and color transformed pixel data is compared with the values of neighboring pixels in the current frame. The percentage of the spatial change of current pixel that is passed through the filter, as an adaptively spatial noise filtered pixel, is weighted based on the spatial relationship of the current pixel to the neighboring pixels) and a range filter function ([0022], the adaptive spatial noise filter includes a distance filter and a signal level range filter. The signal level range filter is configured to filter the pixel based on a difference between a signal level of a pixel and each of the signal levels of the plurality of neighboring pixels and based on the noise dependent signal level parameter); and
processing the image based on the weighted metric for the first portion of the image ([0155-0158], COMBINE FILTERS process 607 multiplies a value at a location in the n by n block of the distance filter values by the value at the same location in the n by a block of range filter values for each location in the two blocks, and so generates a combined filter value for each location in a n by n block of combined filter values. COMBINE FILTERS process 607 transfers to NORMALIZE process 608, GENERATE PIXEL process 609 generates the adaptively spatial noise filtered pixel and writes the adaptively spatial noise filtered pixel to output frame of filtered pixel data 614, sometimes referred to as output frame of pixel data or filtered pixel output frame).
Trejo fails to teach the following, which in the same field of endeavor, Pinhasov teaches wherein the weighted metric is a confidence weighted metric ([0062], The confidence map 235 identifies a degree of confidence that the classification engine 220 has as to its classification of a given pixel in the category map 230. The classification engine 220 sends the category map 230 and the confidence map 235 to the ISP 240, [0117], Neighboring pixels with lower levels of confidence in the confidence map 235 can have lower weights in the neighbor weights 915 than neighboring pixels with higher levels of confidence in the confidence map 235, and can therefore contribute less to the weight sum. Neighboring pixels that are farther from the pixel(s) being upscaled can also have lower weights than neighboring pixels that are closer to the pixel(s) being upscaled, and can therefore contribute less to the weight sum. In other words, neighboring pixels that are closer to the pixel(s) being upscaled can have higher weights than neighboring pixels that are farther from the pixel(s) being upscaled. In operation 930, the category with the maximum weight is used for the upscaled pixel in the upscaled category map 950) based on identifying a maximum confidence value among a plurality of confidence values ([0062], Pixels illustrated in white in the confidence map 235 represent a high confidence level, such as a confidence level exceeding a high threshold percentage, such as 90%. Pixels illustrated in black in the confidence map 235 represent a low confidence level, such as a confidence level falling below a low threshold percentage, such as 10%. The confidence map 235 also includes six different shades of grey (other than black and white), each representing confidence levels falling into different ranges of confidence between the high threshold percentage and the low threshold percentage (~where exceeding 90% is the maximum confidence value, and each shade of black/white/grey is the plurality of confidence values)).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the determination of a category map and a confidence score, as taught in Pinhasov, in the weighted metrics of Trejo, in order to better apply different image processing technologies to different portions of the image, enhancing overall accuracy.
Trejo, as modified by Pinhasov, fails to explicitly teach that the plurality of confidence values are from corresponding pixels of a plurality of channels of a confidence volume.
In the same field of endeavor, Wang teaches wherein the plurality of confidence values are from corresponding pixels of a plurality of channels of a confidence volume ([0092], The computational costs of all disparity values in a disparity dimension for each pixel point in the cost volume are normalized, to obtain a confidence volume Z∈R.sup.(h×w×d). Two branches of processes are applied to the confidence volume. In a first branch, a maximum value is selected from confidence levels of all the disparity values in the disparity dimension for each pixel point in the confidence volume, to obtain a confidence map, then each pixel point having a confidence level higher than a predetermined threshold in the confidence map is mapped to 1, and each pixel point having a confidence level lower than or equal to the threshold in the confidence map is mapped to 0, to obtain a mask map. In a second branch, an argmax value is calculated for the confidence levels of all disparity values in the disparity dimension for each pixel point in the confidence volume using an argmax function (i.e., selecting the sequence number of the maximum confidence level in the disparity dimension), to obtain a complete, dense disparity map which contains some pixel points having low confidence levels or matching errors. The mask map obtained in the first branch is multiplied with the disparity map obtained in the second branch to filter out those pixel points having low confidence levels or matching errors in the disparity map, so as to obtain a target disparity map. Finally, a distance may be estimated based on the target disparity map).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to further modify the confidence-weighted image processing of Trejo and Pinhasov, to arrange the plurality of confidence values corresponding to each pixel location in respective channels of a confidence volume, as suggested by Wang’s confidence volume having a plurality of confidence values along a third dimension for each pixel point, and to select the maximum confidence value therefrom, in order to identify the highest confidence value associated with each pixel for subsequent image processing, thereby enhancing overall accuracy.
Regarding Claim 2, Trejo, as modified by Pinhasov and Wang, teaches the invention of Claim 1 above, where Pinhasov further teaches wherein the confidence weighted metric is based on a maximum confidence value ([0062], Pixels illustrated in white in the confidence map 235 represent a high confidence level, such as a confidence level exceeding a high threshold percentage, such as 90%. Pixels illustrated in black in the confidence map 235 represent a low confidence level, such as a confidence level falling below a low threshold percentage, such as 10%. The confidence map 235 also includes six different shades of grey (other than black and white), each representing confidence levels falling into different ranges of confidence between the high threshold percentage and the low threshold percentage (~where exceeding 90% is the maximum confidence value, and each shade of black/white/grey is the plurality of confidence values)) among each of the one or more second portions of the image ([0117], Neighboring pixels with lower levels of confidence in the confidence map 235 can have lower weights in the neighbor weights 915 than neighboring pixels with higher levels of confidence in the confidence map 235, and can therefore contribute less to the weight sum. Neighboring pixels that are farther from the pixel(s) being upscaled can also have lower weights than neighboring pixels that are closer to the pixel(s) being upscaled, and can therefore contribute less to the weight sum. In other words, neighboring pixels that are closer to the pixel(s) being upscaled can have higher weights than neighboring pixels that are farther from the pixel(s) being upscaled. In operation 930, the category with the maximum weight is used for the upscaled pixel in the upscaled category map 950).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the determination of a category map and a confidence score, as taught in Pinhasov, in the weighted metrics of Trejo, in order to better apply different image processing technologies to different portions of the image, enhancing overall accuracy.
Regarding Claim 3, Trejo, as modified by Pinhasov and Wang, teaches the invention of Claim 2 above, where Trejo further teaches wherein the one or more second portions of the image are neighboring portions of the first portion of the image ([0022], The adaptive spatial noise filter is configured to determine a difference between the signal level of the first pixel and the signal level of each of a plurality of neighboring pixels).
Regarding Claim 4, Trejo, as modified by Pinhasov and Wang, teaches the invention of Claim 1 above, where Trejo further teaches wherein the confidence weighted metric is further based on a spatial filter function ([0132], Adaptive spatial noise filter 570 uses a square block of pixels, e.g., a plurality of neighboring pixels, around a current pixel to determine how to filter the current pixel. The distance filter filters the current pixel based on weights determined from the Euclidean distance of adjacent pixels in the block from the current pixel. The signal range filter, sometimes called a range filter or an intensity filter, is an adaptive noise signal range filter based on the value of the pixel noise in frame imNoiseY corresponding to the pixel being filtered. Thus, the weights used in the range filter are based not only on differences in signal levels between the adjacent pixels in the block and the current pixel, but also on the noise component of the current pixel. The pixel noise in the pixel noise input frame at the same location as the location of current pixel in the input pixel frame is said to correspond to the current pixel and is referred to as the noise component of the current pixel).
Regarding Claim 5, Trejo, as modified by Pinhasov and Wang, teaches the invention of Claim 4 above, where Trejo further teaches wherein the one or more second portions of the image comprise a first neighboring portion and a second neighboring portion, and wherein the spatial filter function has a greater value for the first neighboring portion than for the second neighboring portion, the first portion being closer to the first neighboring portion than to the second neighboring portion ([0132-0134], adaptive spatial noise filter adjusts the weights used in the filter not only based on the Euclidean distance of adjacent pixels from the pixel, but also based on differences in signal levels between the adjacent pixels and the current pixel and based on the noise component of the current pixel. Specifically, as explained more completely below, adaptive spatial noise filter 570 takes a weighted sum of the pixels in a neighborhood of the current pixel. The weights depend on the spatial distance and on the signal levels of the neighboring pixels as well as the noise component of the current pixel itself. Thus, the signal value at each pixel in a frame is replaced by a noise-based weighted average of signal values from nearby pixels, [0075], The percentage of the spatial change of current pixel that is passed through the filter, as an adaptively spatial noise filtered pixel, is weighted based on the spatial relationship of the current pixel to the neighboring pixels; based on the signal level relationship of the current pixel to the signal level of neighboring pixels; and based on the noise component of the current pixel).
Regarding Claim 6, Trejo, as modified by Pinhasov and Wang, teaches the invention of Claim 5 above, where Trejo further teaches wherein: the spatial filter function is within 30% of (x2 – x) (y2 – y) / ((x2 – x1) (y2 – y1)), x1 and y1 are the coordinates of the first neighboring portion, x2 and y2 are the coordinates of the second neighboring portion, and x and y are the coordinates of the first portion ([0131], Ĩ(x)=(Distance Filter)*(Range Filter(Noise)), [0150-0152], DISTANCE FILTER process 604 generates a value of the spatial Gaussian function defined above for each pixel in the input block of pixels, i.e., for each pixel in the input block of pixels, and so generates a distance filter value for each location in an n by n block of distance filter values. DISTANCE FILTER process 604 transfers to a RANGE DIFFERENCE process 605, which evaluates I(y)−I(x) where x is the location of the center pixel in the block, I(x) is the value of the center pixel, I(y) is the value of the pixel at location y in the input block of pixels, and y ranges over the locations in the input block of pixels, and generates a range difference for each location in an n by n block of range differences, the adaptive spatial noise filter compares the signal level of the current pixel to the signal levels of a plurality of neighboring pixels and to an estimated pixel noise parameter by dividing the square of the absolute value of the difference between the signal level of the pixel at location y and the signal level of the pixel at location x by the square of signal level range parameter σ(Noise)r. Here, signal level range parameter σ(Noise)r is the estimated pixel noise parameter for this aspect. This comparison is used in outputting the adaptively spatial noise filtered pixel value).
Regarding Claim 7, Trejo, as modified by Pinhasov and Wang, teaches the invention of Claim 6 above, where Trejo further teaches wherein the spatial filter function is, for each of the one or more second portions of the image, within 30% of a Gaussian function of coordinate differences between the first portion and the one or more second portions of the image ([0131], Ĩ(x)=(Distance Filter)*(Range Filter(Noise)), [0150-0152], DISTANCE FILTER process 604 generates a value of the spatial Gaussian function defined above for each pixel in the input block of pixels, i.e., for each pixel in the input block of pixels, and so generates a distance filter value for each location in an n by n block of distance filter values. DISTANCE FILTER process 604 transfers to a RANGE DIFFERENCE process 605, which evaluates I(y)−I(x) where x is the location of the center pixel in the block, I(x) is the value of the center pixel, I(y) is the value of the pixel at location y in the input block of pixels, and y ranges over the locations in the input block of pixels, and generates a range difference for each location in an n by n block of range differences, the adaptive spatial noise filter compares the signal level of the current pixel to the signal levels of a plurality of neighboring pixels and to an estimated pixel noise parameter by dividing the square of the absolute value of the difference between the signal level of the pixel at location y and the signal level of the pixel at location x by the square of signal level range parameter σ(Noise)r. Here, signal level range parameter σ(Noise)r is the estimated pixel noise parameter for this aspect. This comparison is used in outputting the adaptively spatial noise filtered pixel value).
Regarding Claim 8, Trejo, as modified by Pinhasov and Wang, teaches the invention of Claim 1 above, where Trejo further teaches wherein the one or more second portions of the image comprise a first neighboring portion and a second neighboring portion, and wherein the range filter function has a greater value for the first neighboring portion than for the second neighboring portion, the first portion being closer, in intensity, to the first neighboring portion than to the second neighboring portion ([0132-0134], The distance filter filters the current pixel based on weights determined from the Euclidean distance of adjacent pixels in the block from the current pixel. The signal range filter, sometimes called a range filter or an intensity filter, is an adaptive noise signal range filter based on the value of the pixel noise in frame imNoiseY corresponding to the pixel being filtered. Thus, the weights used in the range filter are based not only on differences in signal levels between the adjacent pixels in the block and the current pixel, but also on the noise component of the current pixel. The pixel noise in the pixel noise input frame at the same location as the location of current pixel in the input pixel frame is said to correspond to the current pixel and is sometimes referred to as the noise component of the current pixel, adaptive spatial noise filter adjusts the weights used in the filter not only based on the Euclidean distance of adjacent pixels from the pixel, but also based on differences in signal levels between the adjacent pixels and the current pixel and based on the noise component of the current pixel. Specifically, as explained more completely below, adaptive spatial noise filter 570 takes a weighted sum of the pixels in a neighborhood of the current pixel. The weights depend on the spatial distance and on the signal levels of the neighboring pixels as well as the noise component of the current pixel itself. Thus, the signal value at each pixel in a frame is replaced by a noise-based weighted average of signal values from nearby pixels, [0075], The percentage of the spatial change of current pixel that is passed through the filter, as an adaptively spatial noise filtered pixel, is weighted based on the spatial relationship of the current pixel to the neighboring pixels; based on the signal level relationship of the current pixel to the signal level of neighboring pixels; and based on the noise component of the current pixel).
Regarding Claim 9, Trejo, as modified by Pinhasov and Wang, teaches the invention of Claim 1 above, where Pinhasov further teaches wherein the confidence weighted metric corresponds to a category of the first portion of the image, and wherein the processing of the image is further performed based on the category of the first portion of the image ([0061-0062], The classification engine 220 can generate a category map 230 and a confidence map 235 using the downscaled second copy of the raw image data 215. For example, the classification engine 220 can partition the downscaled second copy of the raw image data 215 into different image regions based on detection of different categories of objects within the different image regions in the downscaled second copy of the raw image data 215. The confidence map 235 identifies a degree of confidence that the classification engine 220 has as to its classification of a given pixel in the category map 230).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the determination of a category map and a confidence score, as taught in Pinhasov, in the weighted metrics of Trejo, in order to better apply different image processing technologies to different portions of the image, enhancing overall accuracy.
Regarding Claim 10, Trejo teaches a system comprising a processing circuit (Figs. 1-2), the processing circuit being configured to: for a first portion of an image, calculate a confidence weighted metric ([0134], for a pixel, the adaptive spatial noise filter adjusts the weights used in the filter not only based on the Euclidean distance of adjacent pixels from the pixel, but also based on differences in signal levels between the adjacent pixels and the current pixel and based on the noise component of the current pixel. Specifically, as explained more completely below, adaptive spatial noise filter 570 takes a weighted sum of the pixels in a neighborhood of the current pixel. The weights depend on the spatial distance and on the signal levels of the neighboring pixels as well as the noise component of the current pixel itself. Thus, the signal value at each pixel in a frame is replaced by a noise-based weighted average of signal values from nearby pixels) based on a plurality of second portions of the image ([0022], adaptive noise filter is configured to determine a difference between the signal level of the first pixel and the signal level of each of a plurality of neighboring pixels. The adaptive spatial noise filter also uses a noise dependent signal level parameter, where the noise dependent signal level parameter is a function of the noise of the first pixel, [0075], When adaptive noise filter 210 is an adaptive spatial noise filter, a value of a current pixel at a location in a current frame of demosiaced and color transformed pixel data is compared with the values of neighboring pixels in the current frame. The percentage of the spatial change of current pixel that is passed through the filter, as an adaptively spatial noise filtered pixel, is weighted based on the spatial relationship of the current pixel to the neighboring pixels) and a range filter function ([0022], the adaptive spatial noise filter includes a distance filter and a signal level range filter. The signal level range filter is configured to filter the pixel based on a difference between a signal level of a pixel and each of the signal levels of the plurality of neighboring pixels and based on the noise dependent signal level parameter); and
process the image based on the confidence weighted metric for the first portion of the image ([0155-0158], COMBINE FILTERS process 607 multiplies a value at a location in the n by n block of the distance filter values by the value at the same location in the n by a block of range filter values for each location in the two blocks, and so generates a combined filter value for each location in a n by n block of combined filter values. COMBINE FILTERS process 607 transfers to NORMALIZE process 608, GENERATE PIXEL process 609 generates the adaptively spatial noise filtered pixel and writes the adaptively spatial noise filtered pixel to output frame of filtered pixel data 614, sometimes referred to as output frame of pixel data or filtered pixel output frame).
Trejo fails to teach the following, which in the same field of endeavor, Pinhasov teaches wherein the weighted metric is a confidence weighted metric ([0062], The confidence map 235 identifies a degree of confidence that the classification engine 220 has as to its classification of a given pixel in the category map 230. The classification engine 220 sends the category map 230 and the confidence map 235 to the ISP 240, [0117], Neighboring pixels with lower levels of confidence in the confidence map 235 can have lower weights in the neighbor weights 915 than neighboring pixels with higher levels of confidence in the confidence map 235, and can therefore contribute less to the weight sum. Neighboring pixels that are farther from the pixel(s) being upscaled can also have lower weights than neighboring pixels that are closer to the pixel(s) being upscaled, and can therefore contribute less to the weight sum. In other words, neighboring pixels that are closer to the pixel(s) being upscaled can have higher weights than neighboring pixels that are farther from the pixel(s) being upscaled. In operation 930, the category with the maximum weight is used for the upscaled pixel in the upscaled category map 950) based on identifying a maximum confidence value among a plurality of confidence values ([0062], Pixels illustrated in white in the confidence map 235 represent a high confidence level, such as a confidence level exceeding a high threshold percentage, such as 90%. Pixels illustrated in black in the confidence map 235 represent a low confidence level, such as a confidence level falling below a low threshold percentage, such as 10%. The confidence map 235 also includes six different shades of grey (other than black and white), each representing confidence levels falling into different ranges of confidence between the high threshold percentage and the low threshold percentage (~where exceeding 90% is the maximum confidence value, and each shade of black/white/grey is the plurality of confidence values)).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the determination of a category map and a confidence score, as taught in Pinhasov, in the weighted metrics of Trejo, in order to better apply different image processing technologies to different portions of the image, enhancing overall accuracy.
Trejo, as modified by Pinhasov, fails to explicitly teach that the plurality of confidence values are from corresponding pixels of a plurality of channels of a confidence volume.
In the same field of endeavor, Wang teaches wherein the plurality of confidence values are from corresponding pixels of a plurality of channels of a confidence volume ([0092], The computational costs of all disparity values in a disparity dimension for each pixel point in the cost volume are normalized, to obtain a confidence volume Z∈R.sup.(h×w×d). Two branches of processes are applied to the confidence volume. In a first branch, a maximum value is selected from confidence levels of all the disparity values in the disparity dimension for each pixel point in the confidence volume, to obtain a confidence map, then each pixel point having a confidence level higher than a predetermined threshold in the confidence map is mapped to 1, and each pixel point having a confidence level lower than or equal to the threshold in the confidence map is mapped to 0, to obtain a mask map. In a second branch, an argmax value is calculated for the confidence levels of all disparity values in the disparity dimension for each pixel point in the confidence volume using an argmax function (i.e., selecting the sequence number of the maximum confidence level in the disparity dimension), to obtain a complete, dense disparity map which contains some pixel points having low confidence levels or matching errors. The mask map obtained in the first branch is multiplied with the disparity map obtained in the second branch to filter out those pixel points having low confidence levels or matching errors in the disparity map, so as to obtain a target disparity map. Finally, a distance may be estimated based on the target disparity map).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to further modify the confidence-weighted image processing of Trejo and Pinhasov, to arrange the plurality of confidence values corresponding to each pixel location in respective channels of a confidence volume, as suggested by Wang’s confidence volume having a plurality of confidence values along a third dimension for each pixel point, and to select the maximum confidence value therefrom, in order to identify the highest confidence value associated with each pixel for subsequent image processing, thereby enhancing overall accuracy.
Regarding Claim 11, Trejo, as modified by Pinhasov and Wang, teaches the invention of Claim 10 above, where Pinhasov further teaches wherein the confidence weighted metric is based on a maximum confidence value ([0062], Pixels illustrated in white in the confidence map 235 represent a high confidence level, such as a confidence level exceeding a high threshold percentage, such as 90%. Pixels illustrated in black in the confidence map 235 represent a low confidence level, such as a confidence level falling below a low threshold percentage, such as 10%. The confidence map 235 also includes six different shades of grey (other than black and white), each representing confidence levels falling into different ranges of confidence between the high threshold percentage and the low threshold percentage (~where exceeding 90% is the maximum confidence value, and each shade of black/white/grey is the plurality of confidence values)) among each of the one or more second portions of the image ([0117], Neighboring pixels with lower levels of confidence in the confidence map 235 can have lower weights in the neighbor weights 915 than neighboring pixels with higher levels of confidence in the confidence map 235, and can therefore contribute less to the weight sum. Neighboring pixels that are farther from the pixel(s) being upscaled can also have lower weights than neighboring pixels that are closer to the pixel(s) being upscaled, and can therefore contribute less to the weight sum. In other words, neighboring pixels that are closer to the pixel(s) being upscaled can have higher weights than neighboring pixels that are farther from the pixel(s) being upscaled. In operation 930, the category with the maximum weight is used for the upscaled pixel in the upscaled category map 950).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the determination of a category map and a confidence score, as taught in Pinhasov, in the weighted metrics of Trejo, in order to better apply different image processing technologies to different portions of the image, enhancing overall accuracy.
Regarding Claim 12, Trejo, as modified by Pinhasov and Wang, teaches the invention of Claim 11 above, where Trejo further teaches wherein the one or more second portions of the image are neighboring portions of the first portion of the image ([0022], The adaptive spatial noise filter is configured to determine a difference between the signal level of the first pixel and the signal level of each of a plurality of neighboring pixels).
Regarding Claim 13, Trejo, as modified by Pinhasov and Wang, teaches the invention of Claim 10 above, where Trejo further teaches wherein the confidence weighted metric is further based on a spatial filter function ([0132], Adaptive spatial noise filter 570 uses a square block of pixels, e.g., a plurality of neighboring pixels, around a current pixel to determine how to filter the current pixel. The distance filter filters the current pixel based on weights determined from the Euclidean distance of adjacent pixels in the block from the current pixel. The signal range filter, sometimes called a range filter or an intensity filter, is an adaptive noise signal range filter based on the value of the pixel noise in frame imNoiseY corresponding to the pixel being filtered. Thus, the weights used in the range filter are based not only on differences in signal levels between the adjacent pixels in the block and the current pixel, but also on the noise component of the current pixel. The pixel noise in the pixel noise input frame at the same location as the location of current pixel in the input pixel frame is said to correspond to the current pixel and is referred to as the noise component of the current pixel).
Regarding Claim 14, Trejo, as modified by Pinhasov and Wang, teaches the invention of Claim 13 above, where Trejo further teaches wherein the one or more second portions of the image comprise a first neighboring portion and a second neighboring portion, and wherein the spatial filter function has a greater value for the first neighboring portion than for the second neighboring portion, the first portion being closer to the first neighboring portion than to the second neighboring portion ([0132-0134], adaptive spatial noise filter adjusts the weights used in the filter not only based on the Euclidean distance of adjacent pixels from the pixel, but also based on differences in signal levels between the adjacent pixels and the current pixel and based on the noise component of the current pixel. Specifically, as explained more completely below, adaptive spatial noise filter 570 takes a weighted sum of the pixels in a neighborhood of the current pixel. The weights depend on the spatial distance and on the signal levels of the neighboring pixels as well as the noise component of the current pixel itself. Thus, the signal value at each pixel in a frame is replaced by a noise-based weighted average of signal values from nearby pixels, [0075], The percentage of the spatial change of current pixel that is passed through the filter, as an adaptively spatial noise filtered pixel, is weighted based on the spatial relationship of the current pixel to the neighboring pixels; based on the signal level relationship of the current pixel to the signal level of neighboring pixels; and based on the noise component of the current pixel).
Regarding Claim 15, Trejo, as modified by Pinhasov and Wang, teaches the invention of Claim 14 above, where Trejo further teaches wherein: the spatial filter function is within 30% of (x2 – x) (y2 – y) / ((x2 – x1) (y2 – y1)), x1 and y1 are the coordinates of the first neighboring portion, x2 and y2 are the coordinates of the second neighboring portion, and x and y are the coordinates of the first portion ([0131], Ĩ(x)=(Distance Filter)*(Range Filter(Noise)), [0150-0152], DISTANCE FILTER process 604 generates a value of the spatial Gaussian function defined above for each pixel in the input block of pixels, i.e., for each pixel in the input block of pixels, and so generates a distance filter value for each location in an n by n block of distance filter values. DISTANCE FILTER process 604 transfers to a RANGE DIFFERENCE process 605, which evaluates I(y)−I(x) where x is the location of the center pixel in the block, I(x) is the value of the center pixel, I(y) is the value of the pixel at location y in the input block of pixels, and y ranges over the locations in the input block of pixels, and generates a range difference for each location in an n by n block of range differences, the adaptive spatial noise filter compares the signal level of the current pixel to the signal levels of a plurality of neighboring pixels and to an estimated pixel noise parameter by dividing the square of the absolute value of the difference between the signal level of the pixel at location y and the signal level of the pixel at location x by the square of signal level range parameter σ(Noise)r. Here, signal level range parameter σ(Noise)r is the estimated pixel noise parameter for this aspect. This comparison is used in outputting the adaptively spatial noise filtered pixel value).
Regarding Claim 16, Trejo, as modified by Pinhasov and Wang, teaches the invention of Claim 15 above, where Trejo further teaches wherein the spatial filter function is, for each of the one or more second portions of the image, within 30% of a Gaussian function of coordinate differences between the first portion and the one or more second portions of the image ([0131], Ĩ(x)=(Distance Filter)*(Range Filter(Noise)), [0150-0152], DISTANCE FILTER process 604 generates a value of the spatial Gaussian function defined above for each pixel in the input block of pixels, i.e., for each pixel in the input block of pixels, and so generates a distance filter value for each location in an n by n block of distance filter values. DISTANCE FILTER process 604 transfers to a RANGE DIFFERENCE process 605, which evaluates I(y)−I(x) where x is the location of the center pixel in the block, I(x) is the value of the center pixel, I(y) is the value of the pixel at location y in the input block of pixels, and y ranges over the locations in the input block of pixels, and generates a range difference for each location in an n by n block of range differences, the adaptive spatial noise filter compares the signal level of the current pixel to the signal levels of a plurality of neighboring pixels and to an estimated pixel noise parameter by dividing the square of the absolute value of the difference between the signal level of the pixel at location y and the signal level of the pixel at location x by the square of signal level range parameter σ(Noise)r. Here, signal level range parameter σ(Noise)r is the estimated pixel noise parameter for this aspect. This comparison is used in outputting the adaptively spatial noise filtered pixel value).
Regarding Claim 17, Trejo, as modified by Pinhasov and Wang, teaches the invention of Claim 10 above, where Trejo further teaches wherein the one or more second portions of the image comprise a first neighboring portion and a second neighboring portion, and wherein the range filter function has a greater value for the first neighboring portion than for the second neighboring portion, the first portion being closer, in intensity, to the first neighboring portion than to the second neighboring portion ([0132-0134], The distance filter filters the current pixel based on weights determined from the Euclidean distance of adjacent pixels in the block from the current pixel. The signal range filter, sometimes called a range filter or an intensity filter, is an adaptive noise signal range filter based on the value of the pixel noise in frame imNoiseY corresponding to the pixel being filtered. Thus, the weights used in the range filter are based not only on differences in signal levels between the adjacent pixels in the block and the current pixel, but also on the noise component of the current pixel. The pixel noise in the pixel noise input frame at the same location as the location of current pixel in the input pixel frame is said to correspond to the current pixel and is sometimes referred to as the noise component of the current pixel, adaptive spatial noise filter adjusts the weights used in the filter not only based on the Euclidean distance of adjacent pixels from the pixel, but also based on differences in signal levels between the adjacent pixels and the current pixel and based on the noise component of the current pixel. Specifically, as explained more completely below, adaptive spatial noise filter 570 takes a weighted sum of the pixels in a neighborhood of the current pixel. The weights depend on the spatial distance and on the signal levels of the neighboring pixels as well as the noise component of the current pixel itself. Thus, the signal value at each pixel in a frame is replaced by a noise-based weighted average of signal values from nearby pixels, [0075], The percentage of the spatial change of current pixel that is passed through the filter, as an adaptively spatial noise filtered pixel, is weighted based on the spatial relationship of the current pixel to the neighboring pixels; based on the signal level relationship of the current pixel to the signal level of neighboring pixels; and based on the noise component of the current pixel).
Regarding Claim 18, Trejo, as modified by Pinhasov and Wang, teaches the invention of Claim 10 above, where Pinhasov further teaches wherein the confidence weighted metric corresponds to a category of the first portion of the image, and wherein the processing circuit is further configured to process the image based on the category of the first portion of the image ([0061-0062], The classification engine 220 can generate a category map 230 and a confidence map 235 using the downscaled second copy of the raw image data 215. For example, the classification engine 220 can partition the downscaled second copy of the raw image data 215 into different image regions based on detection of different categories of objects within the different image regions in the downscaled second copy of the raw image data 215. The confidence map 235 identifies a degree of confidence that the classification engine 220 has as to its classification of a given pixel in the category map 230).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the determination of a category map and a confidence score, as taught in Pinhasov, in the weighted metrics of Trejo, in order to better apply different image processing technologies to different portions of the image, enhancing overall accuracy.
Regarding Claim 19, Trejo teaches a system comprising means for processing, the means for processing being configured to: for a first portion of an image, calculate a confidence weighted metric ([0134], for a pixel, the adaptive spatial noise filter adjusts the weights used in the filter not only based on the Euclidean distance of adjacent pixels from the pixel, but also based on differences in signal levels between the adjacent pixels and the current pixel and based on the noise component of the current pixel. Specifically, as explained more completely below, adaptive spatial noise filter 570 takes a weighted sum of the pixels in a neighborhood of the current pixel. The weights depend on the spatial distance and on the signal levels of the neighboring pixels as well as the noise component of the current pixel itself. Thus, the signal value at each pixel in a frame is replaced by a noise-based weighted average of signal values from nearby pixels) based on a plurality of second portions of the image ([0022], adaptive noise filter is configured to determine a difference between the signal level of the first pixel and the signal level of each of a plurality of neighboring pixels. The adaptive spatial noise filter also uses a noise dependent signal level parameter, where the noise dependent signal level parameter is a function of the noise of the first pixel, [0075], When adaptive noise filter 210 is an adaptive spatial noise filter, a value of a current pixel at a location in a current frame of demosiaced and color transformed pixel data is compared with the values of neighboring pixels in the current frame. The percentage of the spatial change of current pixel that is passed through the filter, as an adaptively spatial noise filtered pixel, is weighted based on the spatial relationship of the current pixel to the neighboring pixels) and a range filter function ([0022], the adaptive spatial noise filter includes a distance filter and a signal level range filter. The signal level range filter is configured to filter the pixel based on a difference between a signal level of a pixel and each of the signal levels of the plurality of neighboring pixels and based on the noise dependent signal level parameter); and
process the image based on the confidence weighted metric for the first portion of the image ([0155-0158], COMBINE FILTERS process 607 multiplies a value at a location in the n by n block of the distance filter values by the value at the same location in the n by a block of range filter values for each location in the two blocks, and so generates a combined filter value for each location in a n by n block of combined filter values. COMBINE FILTERS process 607 transfers to NORMALIZE process 608, GENERATE PIXEL process 609 generates the adaptively spatial noise filtered pixel and writes the adaptively spatial noise filtered pixel to output frame of filtered pixel data 614, sometimes referred to as output frame of pixel data or filtered pixel output frame).
Trejo fails to teach the following, which in the same field of endeavor, Pinhasov teaches wherein the weighted metric is a confidence weighted metric ([0062], The confidence map 235 identifies a degree of confidence that the classification engine 220 has as to its classification of a given pixel in the category map 230. The classification engine 220 sends the category map 230 and the confidence map 235 to the ISP 240, [0117], Neighboring pixels with lower levels of confidence in the confidence map 235 can have lower weights in the neighbor weights 915 than neighboring pixels with higher levels of confidence in the confidence map 235, and can therefore contribute less to the weight sum. Neighboring pixels that are farther from the pixel(s) being upscaled can also have lower weights than neighboring pixels that are closer to the pixel(s) being upscaled, and can therefore contribute less to the weight sum. In other words, neighboring pixels that are closer to the pixel(s) being upscaled can have higher weights than neighboring pixels that are farther from the pixel(s) being upscaled. In operation 930, the category with the maximum weight is used for the upscaled pixel in the upscaled category map 950) based on identifying a maximum confidence value among a plurality of confidence values ([0062], Pixels illustrated in white in the confidence map 235 represent a high confidence level, such as a confidence level exceeding a high threshold percentage, such as 90%. Pixels illustrated in black in the confidence map 235 represent a low confidence level, such as a confidence level falling below a low threshold percentage, such as 10%. The confidence map 235 also includes six different shades of grey (other than black and white), each representing confidence levels falling into different ranges of confidence between the high threshold percentage and the low threshold percentage (~where exceeding 90% is the maximum confidence value, and each shade of black/white/grey is the plurality of confidence values)).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the determination of a category map and a confidence score, as taught in Pinhasov, in the weighted metrics of Trejo, in order to better apply different image processing technologies to different portions of the image, enhancing overall accuracy.
Trejo, as modified by Pinhasov, fails to explicitly teach that the plurality of confidence values are from corresponding pixels of a plurality of channels of a confidence volume.
In the same field of endeavor, Wang teaches wherein the plurality of confidence values are from corresponding pixels of a plurality of channels of a confidence volume ([0092], The computational costs of all disparity values in a disparity dimension for each pixel point in the cost volume are normalized, to obtain a confidence volume Z∈R.sup.(h×w×d). Two branches of processes are applied to the confidence volume. In a first branch, a maximum value is selected from confidence levels of all the disparity values in the disparity dimension for each pixel point in the confidence volume, to obtain a confidence map, then each pixel point having a confidence level higher than a predetermined threshold in the confidence map is mapped to 1, and each pixel point having a confidence level lower than or equal to the threshold in the confidence map is mapped to 0, to obtain a mask map. In a second branch, an argmax value is calculated for the confidence levels of all disparity values in the disparity dimension for each pixel point in the confidence volume using an argmax function (i.e., selecting the sequence number of the maximum confidence level in the disparity dimension), to obtain a complete, dense disparity map which contains some pixel points having low confidence levels or matching errors. The mask map obtained in the first branch is multiplied with the disparity map obtained in the second branch to filter out those pixel points having low confidence levels or matching errors in the disparity map, so as to obtain a target disparity map. Finally, a distance may be estimated based on the target disparity map).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to further modify the confidence-weighted image processing of Trejo and Pinhasov, to arrange the plurality of confidence values corresponding to each pixel location in respective channels of a confidence volume, as suggested by Wang’s confidence volume having a plurality of confidence values along a third dimension for each pixel point, and to select the maximum confidence value therefrom, in order to identify the highest confidence value associated with each pixel for subsequent image processing, thereby enhancing overall accuracy.
Regarding Claim 20, Trejo, as modified by Pinhasov and Wang, teaches the invention of Claim 19 above, where Pinhasov further teaches wherein the confidence weighted metric is based on a maximum confidence value ([0062], Pixels illustrated in white in the confidence map 235 represent a high confidence level, such as a confidence level exceeding a high threshold percentage, such as 90%. Pixels illustrated in black in the confidence map 235 represent a low confidence level, such as a confidence level falling below a low threshold percentage, such as 10%. The confidence map 235 also includes six different shades of grey (other than black and white), each representing confidence levels falling into different ranges of confidence between the high threshold percentage and the low threshold percentage (~where exceeding 90% is the maximum confidence value, and each shade of black/white/grey is the plurality of confidence values)) among each of the one or more second portions of the image ([0117], Neighboring pixels with lower levels of confidence in the confidence map 235 can have lower weights in the neighbor weights 915 than neighboring pixels with higher levels of confidence in the confidence map 235, and can therefore contribute less to the weight sum. Neighboring pixels that are farther from the pixel(s) being upscaled can also have lower weights than neighboring pixels that are closer to the pixel(s) being upscaled, and can therefore contribute less to the weight sum. In other words, neighboring pixels that are closer to the pixel(s) being upscaled can have higher weights than neighboring pixels that are farther from the pixel(s) being upscaled. In operation 930, the category with the maximum weight is used for the upscaled pixel in the upscaled category map 950).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the determination of a category map and a confidence score, as taught in Pinhasov, in the weighted metrics of Trejo, in order to better apply different image processing technologies to different portions of the image, enhancing overall accuracy.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARGARET G WEBB whose telephone number is (571)270-7803. The examiner can normally be reached M-F 9:00-6:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Charles Appiah can be reached at (571) 272-7904. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MARGARET G WEBB/ Primary Examiner, Art Unit 2641
