DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-19 are rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Ttofis and Theocharides (“High-quality real-time hardware stereo matching based on guided image filtering”).
Regarding Claim 1, Ttofis and Theocharides teach a computing device for image processing, the computing device comprising:
Abstract: “This paper proposes a hardware-based stereo matching architecture that aims to provide high accuracy and concurrently high performance in embedded vision applications.”
a memory;
A. Hardware Implementation of the Guided Image Filter, pg. 3: “The mean filtering process is implemented in hardware with simple arithmetic operations (addition, subtraction and fixed-point multiplication) and a series of read/write operations to a memory buffer (stores the column sums) using the architecture shown in Fig. 3(b).”
B. Proposed GIF-based Stereo Matcher (GIF-SM), pg. 4: “It stores 2r+1 lines of both RGB and gradient data in BRAMs working in read-first mode; this allows access on both the new and old pixels.”
and one or more processors implemented in circuitry, coupled to the memory, and configured to:
B. Proposed GIF-based Stereo Matcher (GIF-SM), pg. 4: “The proposed GIF-based Stereo Matching architecture shown in Fig. 4(a) is decomposed into four major hardware units: the Gradients Computation & Memory Management Unit (GCMMU), the Cost Volume Construction Unit (CVCU), the Cost Volume Filtering & Disparity Selection Unit (CVFDSU) and the Disparity Refinement Unit (DRU). These hardware units are fully pipelined in order to obtain best processing throughput.”
Explanation: The reference discloses hardware processing units implemented in circuitry interacting with memory. The architecture includes processing cores and therefore discloses processors implemented in circuitry coupled to memory.
determine a plurality of filtered values for a plurality of sub-kernels of a kernel of image data, the plurality of filter values corresponding to performing an image filtering operation on each of the plurality of sub-kernels, including:
2) Cost Volume Filtering, pg. 2: “This step utilizes the GIF to smooth each slice of the SCV…Typically, the filtered cost value at p and disparity d is a weighted average of the pixels in the same slice of the SCV…”
A. Hardware Implementation of the Guided Image Filter, pg. 3: “The mean intensity of pixels over rectangular windows in the image can be implemented in a fast way using the integral image technique.”
Explanation: The reference discloses computing filtered values using a guided image filter applied across kernel windows of image data. The rectangular windows correspond to kernel/sub-kernel filtering operations producing filtered values.
using one or more filtered values for one or more sub-kernels of a neighboring kernel that corresponds to performing the image filtering operation on each of the one or more sub-kernels of the neighboring kernel as a first one or more filtered values for a first one or more sub-kernels of the kernel;
A. Hardware Implementation of the Guided Image Filter, pg. 3: “Each column sum accumulates 2r+1 pixels, while the window sum is computed by adding 2r+1 adjacent column sums…When the window is moved to the right from one pixel to the next, the column sum to the right of the window is yet to be computed for the current row, so it is centered one row above… The second step moves the window to the right and updates the window sum by subtracting its leftmost column sum (old column sum), and adding the updated column sum computed in step 1 (new column sum).”
Explanation: The reference discloses reuse of filtered window values when the kernel moves across the image. Thus, filtered results from neighboring windows (kernels) are reused to compute filtered values for the current kernel. This corresponds to using filtered values from neighboring kernels as first filtered values.
and performing the image filtering operation on each of a second one or more sub-kernels of the kernel to generate a second one or more filtered values for the second one or more sub-kernels of the kernel;
A. Hardware Implementation of the Guided Image Filter, pg. 3: “The mean filter architecture receives the new pixel and the old pixel, and outputs the mean corresponding to the window being filtered.”
Explanation: The reference discloses computing new filter outputs for windows as the filter processes new pixels. Thus, the system performs filtering on the next window/sub-kernel to generate additional filtered values.
and determine, using the plurality of filtered values, a filtered value for the kernel that corresponds to performing the image filtering operation on the kernel.
2) Cost Volume Filtering, pg. 2: “Typically, the filtered cost value at p and disparity d is a weighted average of the pixels in the same slice of the SCV…”
A. Hardware Implementation of the Guided Image Filter, pg. 3: “The final mean value is computed by multiplying the window sum with 1/(2r+1)2.”
Explanation: The reference discloses computing the final filtered output based on filtered window values. Thus, the architecture determines a final filtered value for the kernel based on filtered values.
Regarding Claim 2, Ttofis and Theocharides teach the computing device of claim 1, wherein to use the one or more filtered values for the one or more sub-kernels of the neighboring kernel as the first one or more filtered values for the first one or more sub-kernels for the kernel, the one or more processors are further configured to:
determine, for each respective sub-kernel of the first one or more sub-kernels, a filtered value for the respective sub-kernel as a filtered value of a corresponding sub-kernel of the one or more sub-kernels of the neighboring kernel that covers the same pixels in the image data as the respective sub-kernel (Fig. 3a (shown below)).
PNG
media_image1.png
241
881
media_image1.png
Greyscale
A. Hardware Implementation of the Guided Image Filter, pg. 3: “Each column sum accumulates 2r+1 pixels, while the window sum is computed by adding 2r+1 adjacent column sums…When the window is moved to the right from one pixel to the next, the column sum to the right of the window is yet to be computed for the current row, so it is centered one row above… The second step moves the window to the right and updates the window sum by subtracting its leftmost column sum (old column sum), and adding the updated column sum computed in step 1 (new column sum).”
Explanation: The reference discloses that filtering results for windows/sub-kernels are reused from neighboring windows that cover overlapping image pixels. The disclosure shows a window (kernel) sliding across image pixels, reuse of filtered values from neighboring windows, and overlapping pixel coverage between kernels. The mean filtering process shown in Figure 3a shows the overlapping windows and reuse of values between neighboring kernels.
Regarding Claim 3, Ttofis and Theocharides teach the computing device of claim 1, wherein the kernel partially overlaps the neighboring kernel in the image data, and wherein the kernel is one of: shifted to the right by one pixel in the image data compared to the neighboring kernel or shifted down by one pixel in the image data compared to the neighboring kernel (Fig. 3a (shown above)).
A. Hardware Implementation of the Guided Image Filter, pg. 3: “When the window is moved to the right from one pixel to the next, the column sum to the right of the window is yet to be computed for the current row, so it is centered one row above… The second step moves the window to the right and updates the window sum by subtracting its leftmost column sum (old column sum), and adding the updated column sum computed in step 1 (new column sum).”
Explanation: The reference explicitly discloses a filtering window shifting across pixels. The disclosure describes kernels/windows shifting one pixel at a time and overlapping windows sharing image pixels. The mean filtering process diagram shown in Figure 3a visually illustrates the kernel shifting operation across image data.
Regarding Claim 4, Ttofis and Theocharides teach the computing device of claim 1, wherein the image filtering operation comprises a neighborhood filtering operation.
A. Hardware Implementation of the Guided Image Filter, pg. 3: “The mean intensity of pixels over rectangular windows in the image can be implemented in a fast way using the integral image technique.”
2) Cost Volume Filtering, pg. 2: “Typically, the filtered cost value at p and disparity d is a weighted average of the pixels in the same slice of the SCV…The filter weights are defined as in (5), where μk and σk are the mean and the variance of I in a squared window ωk with dimensions r×r, centered at pixel k, |ω| is the number of pixels in the window and ε is a smoothness parameter.”
Explanation: The rectangular windows correspond to pixel neighborhoods used to compute the filtered value, and a squared window corresponds to a neighborhood filtering operation.
Regarding Claim 5, Ttofis and Theocharides teach the computing device of claim 1, wherein the one or more processors are further configured to:
perform the image filtering operation on a plurality of sub-kernels of the neighboring kernel to generate a second plurality of filtered values for the plurality of sub-kernels of the neighboring kernel;
A. Hardware Implementation of the Guided Image Filter, pg. 4: “The architecture consists of four mean filters that compute the values of meanI, meanp, corrI and corrIp.”
2) Cost Volume Filtering, pg. 2: “This step utilizes the GIF to smooth each slice of the SCV…Typically, the filtered cost value at p and disparity d is a weighted average of the pixels in the same slice of the SCV…”
Explanation: This describes filtering multiple sub-regions (sub-kernels) and generating multiple filtered outputs.
and determine, using the second plurality of filtered values, a filtered value for the neighboring kernel that corresponds to performing the image filtering operation on the neighboring kernel.
A. Hardware Implementation of the Guided Image Filter, pg. 3: “The final mean value is computed by multiplying the window sum with 1/(2r+1)2.”
2) Cost Volume Filtering, pg. 2: “Typically, the filtered cost value at p and disparity d is a weighted average of the pixels in the same slice of the SCV…”
Explanation: The reference discloses computing a final filtered output from the filtered values and thus determines the final filtered kernel value using multiple filtered values.
Regarding Claim 6, Ttofis and Theocharides teach the computing device of claim 1, wherein the neighboring kernel is a horizontal neighboring kernel to the left of the kernel in the image data and wherein the neighboring kernel partially overlaps the kernel in the image data (Fig. 3a (shown above)).
A. Hardware Implementation of the Guided Image Filter, pg. 3: “When the window is moved to the right from one pixel to the next, the column sum to the right of the window is yet to be computed for the current row, so it is centered one row above… The second step moves the window to the right and updates the window sum by subtracting its leftmost column sum (old column sum), and adding the updated column sum computed in step 1 (new column sum).”
Explanation: The reference explicitly discloses horizontal sliding windows that overlap with neighboring windows. The reference shows a kernel/window sliding horizontally, the previous window to the left acting as the neighboring kernel, and overlapping pixels between consecutive windows. Figure 3a illustrates two horizontally adjacent windows sharing overlapping pixels.
Regarding Claim 7, Ttofis and Theocharides teach the computing device of claim 6, wherein the one or more sub-kernels of the neighboring kernel form one or more right-most columns of sub-kernels of the neighboring kernel, and wherein the first one or more sub-kernels of the kernel forms one or more left-most columns of sub-kernels of the kernel (Fig. 3a (shown above)).
A. Hardware Implementation of the Guided Image Filter, pg. 3: “The main idea is to maintain a sum for each column in the image to be filtered. Each column sum accumulates 2r+1 pixels, while the window sum is computed by adding 2r+1 adjacent column sums…The second step moves the window to the right and updates the window sum by subtracting its leftmost column sum (old column sum), and adding the updated column sum computed in step 1 (new column sum).”
Explanation: The reference explicitly discloses the reuse of column sums between neighboring sliding windows. The disclosure corresponds to right-most columns of the neighboring kernel becoming left-most columns of the new kernel. Figure 3a shows overlapping column regions between adjacent windows and reuse of column sums when shifting the window horizontally.
Regarding Claim 8, Ttofis and Theocharides teach the computing device of claim 1, wherein the neighboring kernel is a vertical neighboring kernel to the top of the kernel in the image data, and wherein the neighboring kernel partially overlaps the kernel in the image data (Fig. 3a (shown above)).
A. Hardware Implementation of the Guided Image Filter, pg. 3: “Each column sum accumulates 2r+1 pixels, while the window sum is computed by adding 2r+1 adjacent column sums…Therefore, the first step consists of updating the column sum to the right of the window, by subtracting its topmost old pixel and adding one new pixel below it.”
Explanation: The disclosure describes top pixel removal and new pixel added below which corresponds to a vertical sliding window relationship between neighboring kernels. Additionally, the architecture description indicates that windows extend vertically and share overlapping rows. Figure 3a depicts vertical stacks of pixels contributing to each column sum.
Regarding Claim 9, Ttofis and Theocharides teach the computing device of claim 8, wherein the one or more sub-kernels of the neighboring kernel form one or more bottom rows of sub-kernels of the neighboring kernel, and wherein the first one or more sub-kernels of the kernel forms one or more top rows of sub-kernels of the kernel (Fig. 3a (shown above)).
A. Hardware Implementation of the Guided Image Filter, pg. 3: “Each column sum accumulates 2r+1 pixels, while the window sum is computed by adding 2r+1 adjacent column sums…Therefore, the first step consists of updating the column sum to the right of the window, by subtracting its topmost old pixel and adding one new pixel below it.”
Explanation: The reference discloses that column sums are updated by removing top pixels and adding bottom pixels when processing the image vertically. The disclosure corresponds to bottom rows of the neighboring kernel becoming top rows of the new kernel when the kernel moves through the image. This disclosure also shows overlapping vertical windows composed of rows of pixels. Figure 3a demonstrates row reuses between neighboring windows.
Regarding Claim 10, Ttofis and Theocharides teach the computing device of claim 1, wherein to determine the plurality of filtered values for the plurality of sub-kernels of a kernel of image data, the one or more processors are further configured to:
determine a third one or more filtered values for a third one or more sub-kernels of the kernel as one or more filtered values for one or more sub-kernels of a second neighboring kernel that corresponds to performing the image filtering operation on each of the one or more sub-kernels of the second neighboring kernel, wherein the neighboring kernel is a horizontal neighboring kernel, and wherein the second neighboring kernel is a vertical neighboring kernel.
A. Hardware Implementation of the Guided Image Filter, pg. 3: “Each column sum accumulates 2r+1 pixels, while the window sum is computed by adding 2r+1 adjacent column sums…When the window is moved to the right from one pixel to the next, the column sum to the right of the window is yet to be computed for the current row, so it is centered one row above. Therefore, the first step consists of updating the column sum to the right of the window, by subtracting its topmost old pixel and adding one new pixel below it. The second step moves the window to the right and updates the window sum by subtracting its leftmost column sum (old column sum), and adding the updated column sum computed in step 1 (new column sum).”
Explanation: The reference discloses computing filtered values by reusing neighboring window values, which shows that filtered values for the current kernel are determined from filtered values associated with neighboring kernels. The reference also discloses filtering operations occurring in both horizontal and vertical directions. Figure 3a illustrates the reuse of neighboring kernels in both directions.
Regarding Claim 11, Ttofis and Theocharides teach the computing device of claim 10,
wherein the one or more sub-kernels of the neighboring kernel form one or more right-most columns of sub-kernels of the neighboring kernel,
A. Hardware Implementation of the Guided Image Filter, pg. 3: “The main idea is to maintain a sum for each column in the image to be filtered. Each column sum accumulates 2r+1 pixels, while the window sum is computed by adding 2r+1 adjacent column sums.”
Explanation: The reference explicitly describes column-based sub-kernels which indicate that the kernel is composed of column-based sub-kernels.
wherein the first one or more sub-kernels of the kernel forms one or more left-most columns of sub-kernels of the kernel,
A. Hardware Implementation of the Guided Image Filter, pg. 3: “The second step moves the window to the right and updates the window sum by subtracting its leftmost column sum (old column sum), and adding the updated column sum computed in step 1 (new column sum).”
Explanation: The reference discloses leftmost column removal when the kernel moves.
wherein the one or more sub-kernels of the second neighboring kernel are in a right-most column of sub-kernels of the second neighboring kernel,
A. Hardware Implementation of the Guided Image Filter, pg. 3: “The second step moves the window to the right and updates the window sum by subtracting its leftmost column sum (old column sum), and adding the updated column sum computed in step 1 (new column sum).”
Explanation: The reference describes the column entering the kernel when sliding horizontally. This new column corresponds to the right-most column of the neighboring kernel.
and wherein the third one or more sub-kernels of the kernel are in a right-most columns of sub-kernels of the kernel (Fig. 3a (shown above)).
Explanation: The column that is added becomes the right-most column of the updated kernel. This is illustrated in Figure 3a which shows the sliding window update where a new right-most column replaces the previous one.
Regarding Claim 12, Ttofis and Theocharides teach all of the limitations of claim 1 above because claim 12 recites a method that performs substantially the same steps.
Regarding Claim 13, Ttofis and Theocharides teach the method of claim 12, and additional limitations are met as in the consideration of claim 2 above.
Regarding Claim 14, Ttofis and Theocharides teach the method of claim 12, and additional limitations are met as in the consideration of claim 6 above.
Regarding Claim 15, Ttofis and Theocharides teach the method of claim 14, and additional limitations are met as in the consideration of claim 7 above.
Regarding Claim 16, Ttofis and Theocharides teach the method of claim 12, and additional limitations are met as in the consideration of claim 8 above.
Regarding Claim 17, Ttofis and Theocharides teach the method of claim 16, and additional limitations are met as in the consideration of claim 9 above.
Regarding Claim 18, Ttofis and Theocharides teach the method of claim 12, and additional limitations are met as in the consideration of claim 10 above.
Regarding Claim 19, Ttofis and Theocharides teach the method of claim 18, and additional limitations are met as in the consideration of claim 11 above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Ttofis and Theocharides in view of Worthington (US2018268530A1).
Regarding Claim 20, Ttofis and Theocharides teach all of the limitations of claim 1 above because claim 20 recites a computer-readable storage medium storing instructions thereon that when executed cause one or more processors to perform the same steps as claim 1. However, Ttofis and Theocharides do not disclose a computer-readable storage medium storing instructions, as the system is described primarily as a hardware implementation.
However, Worthington teaches implementing image filtering operations using a computing device executing an image filtering system, including kernel-based filtering of digital images. Worthington describes both software and hardware implementations, stating that “the components 306 - 314 and their corresponding elements can comprise software, hardware, or both. For example, the components 306 - 314 and their corresponding elements can comprise one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices” (paragraph [0103]).
Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to implement the image filtering architecture of Ttofis and Theocharides using instructions stored on a computer-readable medium. Worthington explains that software-based implementation of image filtering systems provides improved efficiency in performing filtering operations on digital images, stating that “the systems and methods determine filtered pixel values in a manner that avoids processing some or all of the entries in a histogram with a zero count…by selectively considering entries of the histogram, the systems and methods reduce computations and speed up the process of determining filtered pixel values” (paragraph [0004]). Implementing the filtering operations of Ttofis and Theocharides using software instructions stored on a computer-readable medium represents the substitution of one known implementation technique (software-based image filtering) for another known technique (hardware-based implementation) to obtain predictable results. Both references perform kernel-based filtering of image data, and the substitution would merely involve implementing the known filtering operations using software stored on a computer-readable medium. Thus, a person of ordinary skill in the art would have recognized that implementing the filtering operations of Ttofis and Theocharides using software instructions on a computing device would improve flexibility and computational efficiency when performing digital image filtering operations.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Worthington and Cox (US 7792386 B1) teach a method and apparatus, including computer program products, for filtering an image. A filter kernel is received to determine one or more filtered values for each pixel in a sequence of pixels, where adjacent pixels are separated by a characteristic distance in the image. A difference kernel is defined based on local differences between a first kernel and a second kernel that are defined by the filter kernel centered at a first location and a second location, respectively. The difference kernel is used to determine a difference between filtered values of adjacent pixels in the sequence.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM ADU-JAMFI whose telephone number is (571)272-9298. The examiner can normally be reached M-T 8:00-6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM ADU-JAMFI/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677