Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner’s Note
The examiner believes the claims are directed to a specific improvement in image correction and not to a mathematical concept. Therefore a 101 rejection does not apply.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-6 and 9-11 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Su et al. (“Rolling Shutter Motion Deblurring”).
Regarding claim 1, Su teaches a method for correcting image blurring in a captured digital image of an object, wherein the captured image is captured with an image sensor of a camera (abstract and section 1, a single rolling shutter motion blurred image), and,
due to a relative movement between the camera and the object during an exposure time T of the captured image, the mapping of an object point of the object onto an image point in the captured image changes, such that the mapping moves along an image trajectory during the exposure time between an exposure start time and an exposure end time, and thereby image blurring arises in the captured image (section 1 and Fig. 2, the camera moves along a motion trajectory during the exposure, and each scanline integrates over a segment of the trajectory during its exposure window),
wherein a relationship between an image point in the blurred captured image b(p) and an image point of a sharpened captured image l(p) is modeled using a mathematical model (equation 2, each row of the blurred image is expressed in terms of the latent sharpened image, the camera motion during the exposure window and a noise term),
wherein the model takes into account the relative movement between the camera and the object during the exposure time T by means of a transformation operator H (equation 4, the warping function maps image positions according to the camera pose using the homography matrix H)
and contains a density function ω which describes an influence of the camera on the exposure during the exposure time T (equation (2), the weighting 1/te applied over the exposure window), and η describes noise occurring on the image sensor during the exposure (equations (1)-(3), the noise terms N),
and wherein, for image correction, a sharpened image l(p) at the image point is ascertained from the blurred captured image b(p) and the model (section 4 and equation (8), the latent image l is recovered from the blurred input b by minimizing the data error term based on the model).
an image-point-dependent density function ω(p,t) is used by means of which a different influence of the camera on the exposure of different image points of the captured image is taken into account during the image correction (equation 2 and section 1 and Fig. 2, “different scanlines integrate over a slightly different segment of the trajectory”).
Regarding claim 2, see equation (2) of Su, the blurred image point is the integral, over the exposure window from the exposure start time to the exposure end time of the row, of the density function times the sharpened image transformed by the camera pose, plus the noise η(p) = ni.
Regarding claim 3, see section 2 and equation (3) of Su, the blurred image B and the sharpened image L are discretized into (M+1)×(N+1) pixels, with M+1 scanlines in the height and N+1 pixels in the width of the image sensor.
Regarding claim 4, see equations (3) and (5) of Su, the integral of the model is discretized by a numerical integration over a finite set Ti of uniformly spaced time samples in the exposure window (section 5.3, 30 samples), whereby the discretized weights and the discretized warping (the homographies at the sampled poses) form the sparse matrix K and the model is converted into the linear system of equations b = Kl + n.
Regarding claim 5, see section 2 of Su following equation (3), the function bilinearly interpolates the intensity at sub-pixel positions of the sharpened image L.
Regarding claim 6, see section 4.3, equation (15) and Algorithm 2 of Su, the linear system is solved to obtain the sharpened image l.
Regarding claim 9, see section 4.2.2 and Figs. 4 and 9 of Su, the image area is divided into a plurality of horizontal regions (blocks of scanlines), and the model with the image-point-dependent (row-dependent) density function is used for the image points of a section (equations (2)-(3)), such that a linear system of equations of the claimed form is obtained for the rows of the section (equation (5)).
Regarding claim 10, see equation (14) and section 4.2.2 of Su, inside a block of several scanlines the blur kernel is assumed to be approximately constant, which is an image-point-constant density function with constant image trajectory, such that the model for that block becomes the convolution k∗l with the mathematical convolution operator ∗, and the sharpened image of the block is ascertained by a mathematical deconvolution (blind deblurring of each region).
Regarding claim 11, see Fig. 9 and section 5.2 of Su, the sharpened image is composed of the deblurred blocks, which are stitched together.
Claim(s) 15 is rejected under 35 U.S.C. 103 as being unpatentable over Su in view of Sakurai (US 9998668).
Regarding claim 15, Su teaches the image-point-dependent density function (see the rejection of claim 1 above), where the different exposure of different image points results from the electronic rolling shutter readout of the image sensor.
Su does not teach a mechanical shutter device.
Sakurai teaches that an opening or closing movement of a mechanical shutter device of the camera has a different influence on the exposure of different image points of the captured image (Figs. 10-11, Fig. 15, the unevenness of exposure is measured in an upper area 122a and a lower area 122b of the sensor).
It would have been obvious prior to the effective filing date of the invention to one of ordinary skill in the art to include in Su the ability to use the image-point-dependent density function to map the influence of the opening or closing movement of a mechanical shutter device on the exposure of different image points as taught by Sakurai. The reason is to correct image blurring for cameras with a mechanical focal plane shutter.
Allowable Subject Matter
Claims 7-8 and 12-14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The linear systems in claim 7-8 with the specific iteration is not found in the prior art.
For claims 12-14, the prior art of record does not teach ascertaining the image-point-dependent density function by holding the camera stationary and pointing it at a constant object and taking a series of individual captures with different start times of the exposure, such that a plurality of observations of the density function is obtained for one image point, in combination with the other limitations of claim 1.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Lapstun teaches A system for capturing aerial images, the system comprising at least one steerable camera module, the steerable camera module comprising a camera and a beam-steering mechanism in the optical path of the camera module whereby the pointing direction of the camera is time-multiplexed to provide a wider effective field of view, the beam-steering mechanism comprising a steerable mirror tilted with respect to an optical axis of the camera module, the steerable mirror adapted to spin about the optical axis to effect beam steering.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HADI AKHAVANNIK whose telephone number is (571)272-8622. The examiner can normally be reached 9 AM - 5 PM Monday to Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HADI AKHAVANNIK/Primary Examiner, Art Unit 2676
Prosecution Insights