DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 5, 6, 9, 13, 14, 16, 17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Motten (NPL: Adaptive Memory Architecture for Real-Time Image Warping) in view of Wikipedia (NPL: Bilinear interpolation) and Staudenmaier (US 20160247255 A1).
Regarding claim 1:
Motten teaches:
A display driver (Motten: The architecture and methods presented in this paper have been implemented on an FPGA system, based on an Altera Cyclone IV with 114,480 logic elements and 432 memory blocks, Pg. 6, Section 5: Implementation), comprising:
an image warping circuit configured to perform image warping processing on input image data corresponding to an input image to generate resulting image data corresponding to a resulting image (Motten: Fig. 3. Warping of input to output pixel stream using line buffer storage. Pg. 2; see Note 1A); and
wherein performing the image warping processing comprises:
performing bilinear interpolation corresponding to a target pixel in a quadrangular target cell defined in the resulting image (Motten: Bilinear interpolation is used to reconstruct the mapping coordinates for the complete image (Fig. 2), Pg. 2, Section A: Reverse Mapping Coordinates; see Note 1B),
determining pixel data of the target pixel based on pixel data of one or more pixels selected from pixels of the input image (Motten: For each pixel of the warped image a pixel of the source image is selected, Pg. 2, Section A: Reverse Mapping Coordinates).
Note 1A: Motten showcases in Fig. 3 that an image warping process involves an Input Image, Line Buffer, and Output Image.
Note 1B: Motten showcases in Fig. 2 that a pixel p may be interpolated from a quadrangular cell of pixels a, b, c, and d in the rectangular grid.
Motten fails to explicitly teach:
drive circuitry configured to drive a display panel based on the resulting image data,
determining a first ratio and a second ratio corresponding to a target pixel in a quadrangular target cell defined in the resulting image, wherein the target pixel is located at an intersection between a first line segment and a second line segment,
wherein the first line segment connects a first point on a first side of the target cell and a second point on a second side of the target cell opposite the first side, the first point dividing the first side according to the first ratio, the second point dividing the second side according to the first ratio, and
wherein the second line segment connects a third point on a third side of the target cell and a fourth point on a fourth side of the target cell opposite the third side, the third point dividing the third side according to the second ratio, the fourth point dividing the fourth side according to the second ratio; and
determining pixel data of the target pixel based on pixel data of one or more pixels selected from pixels of the input image based on the first ratio and the second ratio.
Wikipedia teaches:
determining a first ratio and a second ratio corresponding to a target pixel in a quadrangular target cell defined in the resulting image, wherein the target pixel is located at an intersection between a first line segment and a second line segment (Wikipedia: As seen in the example on the right, the intensity value at the pixel computed to be at row 20.2, column 14.5 can be calculated by first linearly interpolating between the values at column 14 and 15 on each rows 20 and 21, Pg. 4, par. 5),
wherein the first line segment connects a first point on a first side of the target cell and a second point on a second side of the target cell opposite the first side, the first point dividing the first side according to the first ratio, the second point dividing the second side according to the first ratio (Wikipedia: Figures on Pg. 1 and Pg. 4; see Note 1C), and
wherein the second line segment connects a third point on a third side of the target cell and a fourth point on a fourth side of the target cell opposite the third side, the third point dividing the third side according to the second ratio, the fourth point dividing the fourth side according to the second ratio Wikipedia: Figures on Pg. 1 and Pg. 4; see Note 1C); and
determining pixel data of the target pixel based on pixel data of one or more pixels selected from pixels of the input image based on the first ratio and the second ratio (Wikipedia: A weighted average of the attributes (color, transparency, etc.) of the four surrounding texels is computed and applied to the screen pixel. This process is repeated for each pixel forming the object being textured, Pg. 4, par. 1).
Note 1C: Wikipedia showcases a first line segment (vertical dotted line segment at 14.5) that connects a first point (denoted by 150.5) on a first side of a target cell and a second point (denoted by 128.5) opposite to the first side, the first point dividing the first side according to the first ratio (0.5), the second point dividing the second side according to the first ratio, and a second line segment (horizontal dotted line segment at 20.2) that connects a third point (intersection between the horizontal line and column 14) and a fourth point (further intersection of the horizontal line with column 15, see below) of the target cell opposite the third side, the third point dividing the third side according to the second ratio (0.2) the fourth point dividing the fourth side according to the second ratio.
The Examiner notes that the horizontal line is cut off in the Figure on Pg. 4, but it would be obvious to one of ordinary skill in the art that the line could continue to column 15 and have a corresponding intersection, as Wikipedia shows a similar figure on Pg. 1 which does include both intersections of the horizontal line “y”.
PNG
media_image1.png
275
274
media_image1.png
Greyscale
Figure depicted on Pg. 4 of the Wikipedia reference.
PNG
media_image2.png
553
562
media_image2.png
Greyscale
Figure depicted on Pg. 1 of Wikipedia.
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Wikipedia with Motten, because Wikipedia teaches known methods of bilinear interpolation, which Motten explicitly teaches the use of: “Bilinear interpolation is used to reconstruct the mapping coordinates for the complete image (Fig. 2)” (Motten, Pg. 2, Section A: Reverse Mapping Coordinates).
Motten in view of Wikipedia still fails to teach:
drive circuitry configured to drive a display panel based on the resulting image data,
Staudenmaier teaches:
drive circuitry configured to drive a display panel based on the resulting image data (Staudenmaier: A script-driven head-up display controller comprising an image warping unit and an image projection unit wherein the image warping unit is coupled to the image projection unit, Abstract)
It would be obvious to combine the teachings of Staudenmaier with Motten because Staudenmaier explicitly refers back to the teachings of Motten: “The details of the foregoing implementation can be found in the following document: Andy Motten, Luc Claesen, Yun Pan, “Adaptive memory architecture for real-time image warping”, ICCD 2012, pages 466-471.” [0003]
Regarding claim 2:
Motten in view of Wikipedia and Staudenmaier teaches:
The display driver of claim 1 (as shown above), wherein determining the pixel data of the target pixel includes:
determining a target pixel corresponding position in a quadrangular source cell defined in the input image (Motten: a better result can be obtained when making use of mapping coordinates which contain fractional values (Fig.4) When using mapping coordinates with sub-pixel accuracy, a window of four pixels is used from the source image, Pg. 2, Section C: Sub-Pixel Resampling), wherein the target pixel corresponding position is a position of an intersection between a third line segment and a fourth line segment (Motten: Bilinear interpolation is hence used to calculate the resulting warped pixel, Pg. 2, Section C: Sub-Pixel Resampling; see Note 2A and Note 1C),
wherein the third line segment connects a fifth point on a fifth side of the source cell and a sixth point on a sixth side of the source cell opposite the fifth side, the fifth point dividing the fifth side according to the first ratio, the sixth point dividing the sixth side according to the first ratio (Motten: Bilinear interpolation is hence used to calculate the resulting warped pixel, Pg. 2, Section C: Sub-Pixel Resampling; see Note 2A and Note 1C),
wherein the fourth line segment connects a seventh point on a seventh side of the source cell and an eighth point on an eighth side of the source cell opposite the seventh side, the seventh point dividing the seventh side according to the second ratio, the eighth point dividing the eighth side according to the second ratio (Motten: Bilinear interpolation is hence used to calculate the resulting warped pixel, Pg. 2, Section C: Sub-Pixel Resampling; see Note 2A and Note 1C), and
wherein selecting the one or more pixels is based on the target pixel corresponding position (see Note 2B).
Note 2A: Motten teaches that similar to the reverse mapping, bilinear interpolation is utilized for sampling from the source image. It follows that one of ordinary skill in the art would re-use the method taught by Wikipedia in this context as well. Specifically (referring back to the Pg. 4 figure of Wikipedia as shown in Note 1C):
Wikipedia teaches a position of an intersection between a third line segment (vertical dotted line at 14.5) and a fourth line segment (horizontal dotted line at 20.2) wherein the third line segment connects a fifth point (point at 150.5) on a fifth side of the source cell and a sixth point (point at 128.5) on a sixth side of the source cell opposite the fifth side, the fifth point dividing the fifth side according to the first ratio (ratio of 0.5), the sixth point dividing the sixth side according to the first ratio and the fourth line segment connects a seventh point (intersection between horizontal dotted line and column 14) on a seventh side of the source cell and an eighth point (further intersection of the horizontal line with column 15, see Note 1C) on an eighth side of the source cell opposite the seventh side, the seventh point dividing the seventh side according to the second ratio (ratio of 0.2), the eighth point dividing the eighth side according to the second ratio.
Note 2B: In Fig. 4, Motten showcases that a target pixel (the singular dark square) may be mapped to four source pixels based on its position, in contrast to integer mapping, which is one-to-one.
Note 2C: As best understood by the Examiner, the limitations of claim 1 are directed to bilinear interpolation for target pixel in the resulting image only (not based on the input image), whereas the limitations of claim 2 are directed to bilinear interpolation based on the input image pixels. The Examiner mapped sections A and C on Pg. 2 of Motten accordingly. See also Fig. 17 of Motten on Pg. 5.
Regarding claim 5:
Motten in view of Wikipedia and Staudenmaier teaches:
The display driver of claim 1 (as shown above), wherein performing the image warping processing further comprises:
defining a target grid that divides the resulting image into a plurality of cells (Motten: These pixels are chosen to be located on a regular grid. The desired grid size depends on the amount of distortion in the image, Pg. 2, Section A: Reverse Mapping Coordinates, par. 2);
determining positions of intersection points between the target grid (Motten: Formula (1) shows how the mapping coordinates for pixel ‘p’ (mapp) are calculated from the mapping coordinates of pixel ‘a’ (mapa), ’b’ (mapb), ’c’ (mapc) and ‘d’ (mapd), which are located on the rectangular grid, Pg. 2, Section A: Reverse Mapping Coordinates, par. 2; see Note 5C) and a horizontal line in which the target pixel is located (Motten: Instead of using a line buffer with a fixed vertical offset across the image line, it is more memory efficient to split up the line buffer into two (or more) slices; […] For every pixel write, it is determined in which memory slice the pixel needs to be written, Pg. 3, col. 2, par. 1; see also Motten Pg. 3, Fig. 9);
storing intersection point information that indicates the positions of the intersection points in a storage (Motten: Storing the mapping coordinates for each pixel uses a large amount of memory. When the mapping coordinates do not change drastically from pixel to pixel, it suffices to only store the mapping coordinates of certain pixels, Pg. 2, Section A: Reverse Mapping Coordinates); and
identifying the target cell from the plurality of cells based on intersection point information (see Note 5B).
Note 5A: The specification of the present application teaches: “The "horizontal line" of the resulting image, as referred to herein, is a row of pixels of the resulting image arrayed in the horizontal direction.” [0072]. Motten teaches a “split circular buffer” that is depicted as a horizontal line of pixels in Fig. 9 on Pg. 3.
Note 5B: Motten showcases the “target cell” derived from the mapping coordinates in the right half of Fig. 2.
Note 5C: The Examiner interpreted the intersection points as claimed in claim 5 to be the mapping coordinates for the pixel as discussed in Motten, because 1) the mapping coordinates are located on a horizontal line (i.e., the line buffer associated with the pixel) and 2) they are located on the rectangular grid.
Regarding claim 6:
Motten in view of Wikipedia and Staudenmaier teaches:
The display driver of claim 5 (as shown above),
wherein the image warping circuit is configured to perform the image warping processing such that the determining of the positions of the intersection points and the storing of the intersection point information are performed (Staudenmaier: the line-based warping descriptor 302 may further comprise a fourth information describing an interpolation scheme to be used by the line-buffer-based memory 112 in order to generate the corresponding output pixel. For instance, bilinear interpolation [0046]; see Note 6A) in a line period prior to a line period during which pixel data of the resulting image data for pixels on the horizontal line are generated (Staudenmaier: The image warping unit 110 is also further adapted to generate at least one output line of the output image, the at least one output line being associated with an electronic image warping of one or more pixels of the one or more input lines. Finally, the image warping unit 110 is further adapted to output the at least one output line to the image projection unit 120 for projection onto the non-flat display unit. [0026]; see Note 6B).
Note 6A: In claim 1, it was shown that Motten in view of Wikipedia teaches a bilinear interpolation process that determines intersection points and intersection point information. Staudenmaier teaches in [0046] that bilinear interpolation may be used during a warping process to generate output pixels.
Note 6B: Staudenmaier teaches: “In fact, in order to generate one output line, an electronic image warping process usually requires one or more inputs lines since the distortion due to the non-flat display 200 for a given output pixel may originate from several input pixels coming from one or more different input lines. This is due to the fact that an input image is usually distorted vertically and horizontally while being projected on the non-flat display 200.” [0029]. In other words, Staudenmaier teaches that prior to generating an output pixel, the locations of multiple input pixels may need to be determined first.
On Pg. 2 of Motten, Fig. 4 similarly showcases that multiple rows of pixels may be used in a “Window Mapping” process to determine an output pixel value. Therefore, the Examiner submits that when the teachings of Staudenmaier are combined with Motten in view of Wikipedia, it would be obvious to one of ordinary skill in the art to determining of the positions of the intersection points and store the intersection point information in a period prior to a second period where a line of output pixel data is generated.
Regarding claim 9:
Motten in view of Wikipedia and Staudenmaier teaches:
The display driver of claim 1 (as shown above), wherein the drive circuitry is configured to drive the display panel to cause a corrected image corresponding to the resulting image data to be displayed on a curved display screen (Staudenmaier: The image projection unit 120 is adapted to project lines of an output image, referred to as output lines, onto the non-flat display unit 200. [0026]).
Regarding claim 13:
Claim 13 is substantially similar to claim 1, and is therefore rejected for similar reasons. Claim 13 contains the following notable differences:
Claim 13 claims a method instead of a display driver. In the rejection of claim 1, it was shown that Motten in view of Wikipedia and Staudenmaier teaches the claimed display driver. It follows that Motten in view of Wikipedia and Staudenmaier teaches the corresponding method.
Regarding claim 14:
Claim 14 is substantially similar to claim 2, and is therefore rejected for similar reasons. Claim 14 contains the following notable differences:
Claim 14 claims a method instead of a display driver. In the rejection of claim 2, it was shown that Motten in view of Wikipedia and Staudenmaier teaches the claimed display driver. It follows that Motten in view of Wikipedia and Staudenmaier teaches the corresponding method.
Regarding claim 16:
Claim 16 is substantially similar to claim 5, and is therefore rejected for similar reasons. Claim 16 contains the following notable differences:
Claim 16 claims a method instead of a display driver. In the rejection of claim 5, it was shown that Motten in view of Wikipedia and Staudenmaier teaches the claimed display driver. It follows that Motten in view of Wikipedia and Staudenmaier teaches the corresponding method.
Regarding claim 17:
Claim 17 is substantially similar to claim 6, and is therefore rejected for similar reasons. Claim 17 contains the following notable differences:
Claim 17 claims a method instead of a display driver. In the rejection of claim 6, it was shown that Motten in view of Wikipedia and Staudenmaier teaches the claimed display driver. It follows that Motten in view of Wikipedia and Staudenmaier teaches the corresponding method.
Regarding claim 20:
Claim 20 is substantially similar to claim 9, and is therefore rejected for similar reasons. Claim 20 contains the following notable differences:
Claim 20 claims a method instead of a display driver. In the rejection of claim 9, it was shown that Motten in view of Wikipedia and Staudenmaier teaches the claimed display driver. It follows that Motten in view of Wikipedia and Staudenmaier teaches the corresponding method.
Claims 3 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Motten (NPL: Adaptive Memory Architecture for Real-Time Image Warping) in view of Wikipedia (NPL: Bilinear interpolation), Staudenmaier (US 20160247255 A1) and StackOverflow (NPL: Looking for an algorithm (version of 2-dimensional binary search)).
Regarding claim 3:
Motten in view of Wikipedia and Staudenmaier teaches:
The display driver of claim 1 (as shown above),
Motten in view of Wikipedia and Staudenmaier fails to teach:
wherein the image warping circuit is configured to determine the first ratio and the second ratio using a binary search.
StackOverflow teaches:
determine the first ratio and the second ratio using a binary search (StackOverflow: What the problem fundamentally consists in is to find a point in a space that contains 100*100 elements. The best you can do is to divide at each step this space in two. […] But if you realize that a binary search on the X axis still divides the research space in two at each step, (the same goes for the Y axis) then you understand that it's optimal, Pg. 1; see also Note 3A).
Note 3A: On Pg. 1, StackOverflow teaches a two-dimensional search algorithm that searches for a location of a point in a two-dimensional quadrilateral by performing a binary search along the x axis, and a binary search along the y axis. StackOverflow also showcases that a point in the quadrilateral has a corresponding horizontal and vertical line:
PNG
media_image3.png
329
394
media_image3.png
Greyscale
Figure from Pg. 1 of Stack Overflow.
In Note 1C, it was shown that the first and second ratios correspond to horizontal and vertical lines intersecting a point. Therefore, it would be obvious to one of ordinary skill in the art to determine the first ratio and the second ratio using a binary search.
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of StackOverflow with Motten in view of Wikipedia and Staudenmaier. Performing a placeholder action, as in StackOverflow, would benefit the Motten in view of Wikipedia and Staudenmaier teachings because binary search is the optimal method of determining the two ratios: “The optimal solution consists in two simple binary search.” (Pg. 1).
Regarding claim 15:
Claim 15 is substantially similar to claim 3, and is therefore rejected for similar reasons. Claim 15 contains the following notable differences:
Claim 15 claims a method instead of a display driver. In the rejection of claim 1, it was shown that Motten in view of Wikipedia, Staudenmaier, and StackOverflow teaches the claimed display driver. It follows that Motten in view of Wikipedia, Staudenmaier, and StackOverflow teaches the corresponding method.
Claims 7, 8, 18, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Motten (NPL: Adaptive Memory Architecture for Real-Time Image Warping) in view of Wikipedia (NPL: Bilinear interpolation), Staudenmaier (US 20160247255 A1) and Gribbon (NPL: A Novel Approach to Real-time Bilinear Interpolation).
Regarding claim 7:
Motten in view of Wikipedia and Staudenmaier teaches:
The display driver of claim 5 (as shown above), wherein performing the image warping processing further comprises:
Motten in view of Wikipedia and Staudenmaier fails to teach:
determining a cell adjacent to each of the intersection points from the plurality of cells; and
storing adjacent cell information that indicates the cell adjacent to each of the intersection points in the storage,
wherein the identifying of the target cell is further based on the adjacent cell information.
Gribbon teaches:
determining a cell adjacent to each of the intersection points from the plurality of cells (Gribbon: Figure 3. Possible scenarios derived from the relationship between the current and previous coordinate, Pg. 4; see Note 7A); and
storing adjacent cell information that indicates the cell adjacent to each of the intersection points in the storage (Gribbon: The integer part of the x-coordinate is used as the address with the y-offset and pixel value stored at the corresponding cache location […] The pixel value to be written into the cache on each clock cycle must be chosen carefully in order to maximise the likelihood of it being used for interpolation on the next line. Pg. 4, Section 3.2: Caching, par. 1-2; see Note 7B),
wherein the identifying of the target cell is further based on the adjacent cell information (Gribbon: The pixel values in the left column of the neighbourhood are obtained from the previous interpolation. After reading the cache contents of the address corresponding to the current location, the y offset indicates that it corresponds to the bottom-right pixel of the neighbourhood (Pg. 4, col 2, par. 1; see Note 7B).
Note 7A: Gribbon teaches a method of caching results from previous bilinear interpolation iterations: “Assuming that as the output image is scanned, the distorted image is traversed in a predominantly left to right and top to bottom direction, there are six possible relationships between the current and previous coordinates as shown in figure 3.” (Pg. 3, Section 3.1, par. 1). Similarly, the specification of the present application teaches: “The cell identification circuit 410 may be further configured to determine the cell adjacent to each intersection point in the direction in which the pixels on the N-th horizontal line are scanned.” [0079]
Note 7B: In other words, Gribbon teaches caching cell information that is adjacent to a cell to be calculated in a future bilinear interpolation. Because the coordinates of the pixel are stored in the cache, when Gribbon reads from the cache, the Examiner submits that Gribbon teaches identifying the target cell based on the adjacent cell information.
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Gribbon with Motten in view of Wikipedia and Staudenmaier, because Motten explicitly refers to the teachings of Gribbon: “Bilinear interpolation is hence used to calculate the resulting warped pixel [8]” (Pg. 2, Section C: Sub-Pixel Resampling, par. 1) where [8] refers to “K.T. Gribbon and D.G. Bailey, ‘A novel approach to real-time bilinear interpolation,’” as seen on Pg. 6.
Regarding claim 8:
Motten in view of Wikipedia, Staudenmaier, and Gribbon teaches:
The display driver of claim 7 (as shown above), wherein the image warping circuit is configured to perform the image warping processing such that the determining of the cell adjacent to each of the intersection points and the storing of the adjacent cell information (Staudenmaier: the line-based warping descriptor 302 may further comprise a fourth information describing an interpolation scheme to be used by the line-buffer-based memory 112 in order to generate the corresponding output pixel. For instance, bilinear interpolation [0046]; see Note 8A) are performed in a line period prior to a line period during which pixel data of the resulting image data for pixels on the horizontal line are generated (Staudenmaier: The image warping unit 110 is also further adapted to generate at least one output line of the output image, the at least one output line being associated with an electronic image warping of one or more pixels of the one or more input lines. Finally, the image warping unit 110 is further adapted to output the at least one output line to the image projection unit 120 for projection onto the non-flat display unit. [0026]; see Note 6B).
Note 8A: In claim 1, it was shown that Motten in view of Wikipedia teaches a bilinear interpolation process that determines intersection points and intersection point information. Staudenmaier teaches in [0046] that bilinear interpolation may be used during a warping process to generate output pixels.
When describing that “Bilinear interpolation is hence used to calculate the resulting warped pixel” on Pg. 2, Motten cites the Gribbon reference, and therefore, it would be obvious to one of ordinary skill in the art to implement the methods taught by Gribbon when utilizing bilinear interpolation. Gribbon teaches that their implementation of bilinear interpolation determines the cell adjacent to each of the intersection points and stores of the adjacent cell information, as discussed in the rejection of claim 7 above.
Therefore, because Staudenmaier teaches that bilinear interpolation may be performed prior to generating a line of output pixels (see also Note 6B), when the teachings of Staudenmaier are combined with Motten in view of Wikipedia and Gribbon, it would be obvious to one of ordinary skill in the art to determine of the cell adjacent to each of the intersection points and store the adjacent cell information in a period prior to a period where a line of output pixel data is generated.
Regarding claim 18:
Claim 18 is substantially similar to claim 7, and is therefore rejected for similar reasons. Claim 18 contains the following notable differences:
Claim 18 claims a method instead of a display driver. In the rejection of claim 1, it was shown that Motten in view of Wikipedia and Staudenmaier teaches the claimed display driver. It follows that Motten in view of Wikipedia and Staudenmaier teaches the corresponding method.
Regarding claim 19:
Claim 19 is substantially similar to claim 8, and is therefore rejected for similar reasons. Claim 19 contains the following notable differences:
Claim 19 claims a method instead of a display driver. In the rejection of claim 8, it was shown that Motten in view of Wikipedia and Staudenmaier teaches the claimed display driver. It follows that Motten in view of Wikipedia and Staudenmaier teaches the corresponding method.
Claims 10 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Motten (NPL: Adaptive Memory Architecture for Real-Time Image Warping) in view of Staudenmaier (US 20160247255 A1).
Regarding claim 10:
Motten teaches:
A display driver (Motten: The architecture and methods presented in this paper have been implemented on an FPGA system, based on an Altera Cyclone IV with 114,480 logic elements and 432 memory blocks, Pg. 6, Section 5: Implementation), comprising:
an image warping circuit configured to perform image warping processing on input image data corresponding to an input image to generate resulting image data corresponding to a resulting image (Motten: Fig. 3. Warping of input to output pixel stream using line buffer storage. Pg. 2; see Note 1A); and
wherein performing the image warping processing comprises:
defining a target grid that divides the resulting image into a plurality of first cells (Motten: These pixels are chosen to be located on a regular grid. The desired grid size depends on the amount of distortion in the image, Pg. 2, Section A: Reverse Mapping Coordinates, par. 2; see Note 10A);
defining a source grid that divides the input image into a plurality of second cells that correspond to the plurality of first cells (see Note 10A), respectively;
determining positions of intersection points between the target grid (Motten: Formula (1) shows how the mapping coordinates for pixel ‘p’ (mapp) are calculated from the mapping coordinates of pixel ‘a’ (mapa), ’b’ (mapb), ’c’ (mapc) and ‘d’ (mapd), which are located on the rectangular grid, Pg. 2, Section A: Reverse Mapping Coordinates, par. 2) and a horizontal line in which a target pixel of the resulting image is located (Motten: Instead of using a line buffer with a fixed vertical offset across the image line, it is more memory efficient to split up the line buffer into two (or more) slices; […] For every pixel write, it is determined in which memory slice the pixel needs to be written, Pg. 3, col. 2, par. 1; see also Motten Pg. 3, Fig. 9);
storing intersection point information that indicates the positions of the intersection points in a storage (Motten: Storing the mapping coordinates for each pixel uses a large amount of memory. When the mapping coordinates do not change drastically from pixel to pixel, it suffices to only store the mapping coordinates of certain pixels, Pg. 2, Section A: Reverse Mapping Coordinates);
identifying a target cell in which the target pixel is located from the plurality of first cells based on the intersection point information (see Note 5B); and
determining pixel data of the target pixel based on pixel data of the input image data for one or more pixels in a source cell of the plurality of the first cells corresponding to the target cell (Motten: When using mapping coordinates with sub-pixel accuracy, a window of four pixels is used from the source image, Pg. 2, Section C: Sub-Pixel Resampling; see also Fig. 4, Pg. 2).
Note 10A: Motten teaches that “pixels are chosen to be located on a regular grid” (Pg. 2, Section A: Reverse Mapping Coordinates). Motten teaches that both the input and output images may be stored: “the input pixel stream needs to be stored in order to select the output pixels from” (Pg. 2, Section 2: Warping Overview, par. 1); “Off-chip memories are used to store the output image.” (Pg. 1, col. 2, par. 2). Therefore, Motten teaches input and output images that have corresponding grids. Furthermore, because Motten teaches that the “input pixel stream needs to be stored in order to select the output pixels from”, one of ordinary skill in the art would conclude that the input and output correspond to each other.
Motten fails to explicitly teach:
drive circuitry configured to drive a display panel based on the resulting image data,
Staudenmaier teaches:
drive circuitry configured to drive a display panel based on the resulting image data (Staudenmaier: A script-driven head-up display controller comprising an image warping unit and an image projection unit wherein the image warping unit is coupled to the image projection unit, Abstract)
It would be obvious to combine the teachings of Staudenmaier with Motten because Staudenmaier explicitly refers back to the teachings of Motten: “The details of the foregoing implementation can be found in the following document: Andy Motten, Luc Claesen, Yun Pan, “Adaptive memory architecture for real-time image warping”, ICCD 2012, pages 466-471.” [0003]
Regarding claim 11:
Motten in view of Staudenmaier teaches:
The display driver of claim 10, wherein the image warping circuit is configured to perform the image warping processing such that the determining of the positions of the intersection points and the storing of the intersection point information are performed (Staudenmaier: the line-based warping descriptor 302 may further comprise a fourth information describing an interpolation scheme to be used by the line-buffer-based memory 112 in order to generate the corresponding output pixel. For instance, bilinear interpolation [0046]; see Note 11A) in a line period prior to a line period during which pixel data of the resulting image data for pixels on the horizontal line are generated (Staudenmaier: The image warping unit 110 is also further adapted to generate at least one output line of the output image, the at least one output line being associated with an electronic image warping of one or more pixels of the one or more input lines. Finally, the image warping unit 110 is further adapted to output the at least one output line to the image projection unit 120 for projection onto the non-flat display unit. [0026]; see Note 11B).
Note 11A: In claim 10, it was shown that Motten teaches a bilinear interpolation process that determines intersection points and intersection point information. Staudenmaier teaches in [0046] that bilinear interpolation may be used during a warping process to generate output pixels.
Note 11B: Staudenmaier teaches: “In fact, in order to generate one output line, an electronic image warping process usually requires one or more inputs lines since the distortion due to the non-flat display 200 for a given output pixel may originate from several input pixels coming from one or more different input lines. This is due to the fact that an input image is usually distorted vertically and horizontally while being projected on the non-flat display 200.” [0029]. In other words, Staudenmaier teaches that prior to generating an output pixel, the locations of multiple input pixels may need to be determined first.
On Pg. 2 of Motten, Fig. 4 similarly showcases that multiple rows of pixels may be used in a “Window Mapping” process to determine an output pixel value. Therefore, the Examiner submits that when the teachings of Staudenmaier are combined with Motten, it would be obvious to one of ordinary skill in the art to determining of the positions of the intersection points and store the intersection point information in a period prior to a second period where a line of output pixel data is generated.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Motten (NPL: Adaptive Memory Architecture for Real-Time Image Warping) in view of Staudenmaier (US 20160247255 A1) and Gribbon (NPL: A Novel Approach to Real-time Bilinear Interpolation).
Regarding claim 12:
Motten in view of Staudenmaier teaches:
The display driver of claim 10 (as shown above), wherein performing the image warping processing further comprises:
Motten in view of Staudenmaier fails to teach:
determining a cell adjacent to each of the intersection points from the plurality of first cells; and
storing adjacent cell information that indicates the cell adjacent to each of the intersection points in the storage,
wherein the identifying of the target cell is further based on the adjacent cell information.
Gribbon teaches:
determining a cell adjacent to each of the intersection points from the plurality of cells (Gribbon: Figure 3. Possible scenarios derived from the relationship between the current and previous coordinate, Pg. 4; see Note 7A); and
storing adjacent cell information that indicates the cell adjacent to each of the intersection points in the storage (Gribbon: The integer part of the x-coordinate is used as the address with the y-offset and pixel value stored at the corresponding cache location […] The pixel value to be written into the cache on each clock cycle must be chosen carefully in order to maximise the likelihood of it being used for interpolation on the next line. Pg. 4, Section 3.2: Caching, par. 1-2; see Note 7B),
wherein the identifying of the target cell is further based on the adjacent cell information (Gribbon: The pixel values in the left column of the neighbourhood are obtained from the previous interpolation. After reading the cache contents of the address corresponding to the current location, the y offset indicates that it corresponds to the bottom-right pixel of the neighbourhood (Pg. 4, col 2, par. 1; see Note 7B).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Gribbon with Motten in view of Staudenmaier, because Motten explicitly refers to the teachings of Gribbon: “Bilinear interpolation is hence used to calculate the resulting warped pixel [8]” (Pg. 2, Section C: Sub-Pixel Resampling, par. 1) where [8] refers to “K.T. Gribbon and D.G. Bailey, ‘A novel approach to real-time bilinear interpolation,’” as seen on Pg. 6.
Allowable Subject Matter
Claim 4 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Claim 4 recites: “a plurality of serially-coupled iteration circuits, wherein each iteration circuit is configured to perform a respective iteration of the binary search, wherein each respective iteration comprises calculation of midpoints located on the first, second, third, and fourth sides of the target cell; and
a ratio calculation circuit configured to calculate the first ratio and the second ratio based on outputs of a final iteration circuit of the serially-coupled iteration circuits.”
The cited prior art, Motten, Wikipedia, Staudenmaier, Gribbon, and StackOverflow fail to teach serially coupled circuits where each circuit performs a respective iteration of binary search.
Pepper (US 20130124491 A1) teaches:
a plurality of serially-coupled iteration circuits (Pepper: The pipelined binary search engine 200 includes four stages, identified as stage 1 through stage 4, [0020]; Pepper: Each of the key storage registers KS (203, 213, 223, 233) and the result registers R (204, 214, 224, 234) may be one or more flips-flops, a register, a latch, or other circuit [0023]), wherein each iteration circuit is configured to perform a respective iteration of the binary search (Pepper: Each stage has twice as much memory as the preceding stage, and each stage reads a single value from its respective memory using the collective results of the comparisons performed in the previous stages, [0028]), wherein each respective iteration comprises calculation of midpoints (Pepper: At the first step in a binary search, the key is compared to the middle value in the array [0006]) located on the first, second, third, and fourth sides of the target cell;
However, Pepper fails to teach “a ratio calculation circuit configured to calculate the first ratio and the second ratio based on outputs of a final iteration circuit of the serially-coupled iteration circuits”.
None of the other prior art searched or on the record teaches, suggests, or renders obvious the limitations of claim 4.
Conclusion
The Examiner identified potential limitation(s) in the specification that would overcome the prior art rejections under 103 if amended into the claims. Note that in such a situation, further search and consideration would be required:
Claims 6 and 8 discuss a “line period”. The specification of the present application teaches “In one or more embodiments, the image warping circuit 400 is configured to generate pixel data of one horizontal line of the resulting image during one line period (or horizontal synchronization period)” [0073] (emphasis added). Specifying that the line period is a horizontal synchronization period in the claims would overcome the respective prior art rejections.
“the storage of the cell identification circuit 410 may include a first first-in-first-out (FIFO)”, as discussed in paragraph [0080] of the specification of the present application.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VINCENT ALEXANDER PROVIDENCE whose telephone number is (571)270-5765. The examiner can normally be reached Monday-Thursday 8:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571)270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VINCENT ALEXANDER PROVIDENCE/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights