DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3, 11, and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou ‘865 (US 20170310865) in view of Zhou ‘970 (US 20170187970).
Regarding claim 1:
Zhou ‘865 discloses: a method for enabling mitigation of rolling shutter effect for an image captured by a camera (Zhou ’865 teaches that for progressive scanning such as a rolling shutter sensor, exposure may be controlled on a row-by-row basis to reduce blur associated with a moving object. Zhou ’865 ¶¶ [0042]–[0052], [0062], and FIG. 1a), the method comprising:
analyzing preview images to identify an object in motion depicted in the preview images (Zhou ’865 ¶ [0055] teaches that the controller may: “analyze a preview image of the object to be shot, and acquire motion information of the object to be shot according to an analysis result”, wherein the preview image comprises a series of view finding images acquired before formal image capture, and determine a motion trend of the object from those images);
calculating an exposure time for each of a plurality of sensor portions of an image sensor of a camera, wherein the exposure time is shorter for sensor portions capturing the object in motion than for sensor portions not capturing the object in motion (Zhou ’865 determines, based on the object’s motion information, the relative displacement of the object’s image on the sensor and identifies an imaging region comprising corresponding pixels or a corresponding pixel row. Zhou ’865 ¶¶ [0042]–[0048]. Zhou ’865 then determines a second exposure duration that is less than the first or default exposure duration, applies the second exposure duration to the row or pixels capturing the moving object, and retains the longer default exposure for the other sensor region. Zhou ’865 ¶¶ [0049]–[0052], [0057]–[0058], [0062]. Thus, Zhou ’865 teaches: preview image analysis to determine object motion; mapping the moving object to corresponding sensor rows or pixels; calculating a shorter exposure for those rows or pixels; and retaining a longer exposure for the remaining sensor region);
Zhou ‘865 does not specifically teach: determining a midpoint of the exposure times for each of the plurality of sensor portions; aligning the exposure times such that the midpoint of the exposure times for each of the plurality of sensor portions occur concurrently and that a readout time for processing captured image data of any of the plurality of sensor portions do not overlap with a readout time of any other of the plurality of sensor portions;
However, in the same field of endeavor, Zhou 970 teaches: determining a midpoint of the exposure times for each of the plurality of sensor portions; aligning the exposure times such that the midpoint of the exposure times for each of the plurality of sensor portions occur concurrently (Zhou ’970 configures different exposure periods for at least two photosensitive areas, each area comprising one or more sensor rows or pixels. Zhou ’970 ¶¶ [0040]–[0045], [0052]–[0053]. Zhou ’970 ¶ [0049] expressly teaches: “aligning midpoints of the exposure periods to determine the start exposure times of the photosensitive areas.” As shown in Zhou ’970 Figure 2d, the longer exposure begins earlier, the shorter exposure begins later, and the different-duration exposure intervals share the same temporal midpoint. Zhou ’970 explains that such alignment reduces the offset of the average temporal center of portions of a moving object and reduces motion blur. Zhou ’970 ¶¶ [0047]–[0049], Fig. 2d.); and
that a readout time for processing captured image data of any of the plurality of sensor portions do not overlap with a readout time of any other of the plurality of sensor portions (Zhou ’970 further teaches non-overlapping readout times. In the shared-read-circuit embodiment, data from a shorter-exposure area is read after that area completes exposure and before the next longer-exposure area completes exposure. Data from the next area is read only after the preceding read has completed. Zhou ’970 states that the read operations are staggered to avoid data-reading conflicts. Zhou ’970 ¶¶ [0054], [0058], [0072]); and
exposing the plurality of sensor portions of the image sensor according to the aligned exposure times (Zhou ’970 further controls exposure of the respective photosensitive areas according to the determined exposure periods and aligned start-exposure times. Zhou ’970 ¶¶ [0042]–[0045], [0049], [0061]–[0066]).
Therefore, It would have been obvious to a person of ordinary skill in the art before the effective filing date to modify Zhou ’865’s motion-dependent sensor-region exposure method using Zhou ’970’s midpoint-aligned exposure and sequential-readout scheduling. Zhou ’865 teaches assigning a shorter exposure to sensor rows or pixels capturing a moving object while retaining a longer exposure for other portions, but does not specify the relative temporal placement of those unequal exposures. Zhou ’970 recognizes that unequal exposures under relative motion can capture different portions of an object at different average temporal positions and teaches aligning the exposure midpoints to reduce such displacement and motion blur. Zhou ’970 further teaches staggering the corresponding readouts to permit use of a shared read circuit without readout conflicts. A skilled artisan therefore would have applied Zhou ’970’s timing arrangement to Zhou ’865’s unequal motion-dependent exposures to obtain the predictable benefits of: reducing blur within the moving-object region through Zhou ’865’s shorter exposure; reducing temporal displacement among differently exposed sensor portions through Zhou ’970’s midpoint alignment; and avoiding shared-circuit readout conflicts through Zhou ’970’s sequential readout schedule.
Regarding claim 3:
Zhou ‘970 further teaches: wherein the midpoint of an exposure time for each of the plurality of sensor portions is between a start exposure time of the exposure time and an end exposure time for the exposure time, and a first time between the start exposure time and the midpoint is equal to a second time between the midpoint and the end exposure time (Zhou ‘970 teaches aligning midpoints to the exposure periods to determine the start exposure times for photosensitive areas relative to a reference time. Zhou ‘970 ¶ [0049]. Zhou ‘970 further teaches exposure periods T1, T2, and T3 centered about aligned temporal midpoints in Fig. 2d, and repeats the midpoint alignment operation in ¶ [0066]. A midpoint of a temporal interval necessarily lies between the start and end of that interval and divides the interval into two equal duration).
Regarding claims 11 and 13: the claims limitations are similar to those of claims 1 and 3; therefore, rejected in the same manner as applied above. Zhou ‘865 discloses the structure in claim 11 in Figs. 3-5.
Claim(s) 2 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou ‘865 (US 20170310865) in view of Zhou ‘970 (US 20170187970) and Dang (US 20170048441).
Regarding claim 2:
Zhou ‘865 further teaches: wherein analyzing preview images to identify the object in motion depicted in the preview images (Zhou ’865 ¶ [0055] teaches that the controller may: “analyze a preview image of the object to be shot, and acquire motion information of the object to be shot according to an analysis result”, where the preview image comprises a series of view finding images acquired before formal image capture, and determine a motion trend of the object from those images);
Zhou ‘865 does not specifically teach: determining that a speed of the object exceeds a threshold speed.
However, in the same field, Dang teaches: determining that a speed of the object exceeds a threshold speed (Dang teaches that method 200 further includes determining whether the speed exceeds the speed threshold, at 216. Dang ¶ [0035], FIG. 2, steps 212-216).
Therefore, it would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Zhou ‘865 and Zhou ‘970 to incorporate the teachings of Dang by including: speed threshold comparison in order to trigger the controller’s motion responsive exposure adjustment when the detected object speed exceeds a predetermined threshold.
Regarding claim 12: the claims limitations are similar to those of claim 2; therefore, rejected in the same manner as applied above.
Claim(s) 4-5 and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou ‘865 (US 20170310865) in view of Zhou ‘970 (US 20170187970) and Kalevo (WO 2007045714).
Regarding claim 4:
Zhou ‘865 in view of Zhou ‘970 does not specifically teach: determining a predominant relative motion direction between the camera and a current scene from a plurality of relative motion directions, wherein the predominant relative motion direction is a direction of camera movement along a horizontal axis or along a vertical axis in relation to the current scene; and determining a scanning direction for the image sensor from a plurality of scanning directions based on the predominant relative motion direction, wherein the scanning direction is a horizontal direction when the predominant relative motion direction is along a horizontal axis, and wherein the scanning direction is a vertical direction when the predominant relative motion direction is along a vertical axis.
However, in the same field, Kalevo teaches: determining a predominant relative motion direction between the camera and a current scene from a plurality of relative motion directions, wherein the predominant relative motion direction is a direction of camera movement along a horizontal axis or along a vertical axis in relation to the current scene (Kalevo teaches detecting relative movement caused by movement if the camera, movement of objects, or both, using image data motion estimation and/or camera mounted motion sensors. Kalevo further teaches determining whether objects in successive images are moving primarily upward, downward, rightward, or leftward. Kalevo pp. 7-10; FIG. 6, and claims 1, 4-5); and
determining a scanning direction for the image sensor from a plurality of scanning directions based on the predominant relative motion direction, wherein the scanning direction is a horizontal direction when the predominant relative motion direction is along a horizontal axis, and wherein the scanning direction is a vertical direction when the predominant relative motion direction is along a vertical axis (Kalevo also determines a main horizontal or vertical axis based on motion component having highest velocity magnitude. The main axis therefore corresponds to the claimed predominant relative motion direction. Kalevo pp. 9-11 and claims 9 and 22. Kalevo further selects the sensor scan direction based on the determined motion axis, and teaches that horizontal motion results in selection of horizontal scan direction and vertical motion results in selection od a vertical scan directions. Kalevo pp. 9-11 and claims 6-9, and 20-22).
Therefore, it would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Zhou ‘865 and Zhou ‘970 to incorporate the teachings of Kalevo to match the scan orientation to the detected motion axis to reduce rolling shutter motion distortion. The modification would predictably further reduce motion induced skew while retaining the motion adaptive exposure and midpoint alignment functions of Zhou combination.
Regarding claim 5:
Kalevo further teaches: wherein the plurality of scanning directions includes at least one of: a) a vertical scanning direction comprising scanning horizontal rows of the image sensor from a top row to a bottom row, b) a vertical scanning direction comprising scanning horizontal rows of the image sensor from a bottom row to a top row, c) a horizontal scanning direction comprising scanning vertical columns of the image sensor from a left column to a right column, or d) a horizontal scanning direction comprising scanning vertical columns of the image sensor from a right column to a left column (Kalevo Fig. 2 discloses reading horizontal rows from the topmost row to the bottommost row and, in reverse from the bottommost row to the topmost row. Kalevo also discloses reading vertical columns from the leftmost column to the rightmost column and, in reverse from the rightmost column to the leftmost column. Kalevo FIG. 2; p 5, lines 25-30, and p. 6 lines 1-6. Kalevo also lists the 4 selectable readout directions as top to bottom, bottom to top, left to right, right to left. Kalevo p. 8, lines 20-30; p. 9 line 1. )
Regarding claims 14-15: the claims limitations are similar to those of claims 4-5; therefore, rejected in the same manner as applied above.
Claim(s) 7-8 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou ‘865 (US 20170310865) in view of Zhou ‘970 (US 20170187970) and Gu (US 20130070121).
Regarding claim 7:
Zhou ‘970 further teaches: determining exposure differences among exposure times for each of the plurality of sensor portions (Zhou ‘970 teaches assigning exposure periods to respective photosensitive areas or sensor portions. Zhou ‘970 ¶¶ [0040] – [0045], [0049]);
Zhou ‘865 in view of Zhou ‘970 does not specifically teach: compensating for brightness variances among each of the plurality of sensor portions based on the exposure differences.
However, in the same field, Gu teaches: compensating for brightness variances among each of the plurality of sensor portions based on the exposure differences (Gu teaches determining respective exposure times for individual rows of an image sensor and capturing an image using different row wise exposures. Gu further normalizes the captured pixel values with respect to the exposure time applied to each row. Specifically, Gu teaches generating a final output image by dividing the captured image values by the corresponding row wise exposure. Gu ¶¶ [0095] – [0103], Figs 15 and 17-18. This normalization accounts for differences among the row exposure times and compensates the resulting differences in pixel intensity or brightness).
Therefore, it would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Zhou ‘865 and Zhou ‘970 to incorporate the teachings of Gu to include an exposure based normalization to the differently exposed sensor portions to reduce brightness discontinuities caused by the different exposures.
Regarding claim 8:
Gu further teaches: wherein compensating for brightness variances is performed by a post-processing imaging algorithm (Gu teaches capturing an image using different row wise exposure times and subsequently normalizing the captured image to generate a final output image. Specifically, Gu teaches dividing each captures pixel value by the exposure time assigned to its corresponding row (see the equation in ¶ [0102]). Gu further explains that the resulting normalized images constitute the final output images and expressly contemplated additional image prost processing. Gu ¶¶ [0098] – [0103], Figs. 15, 17, and 18.).
Regarding claims 17-18: the claims limitations are similar to those of claims 7-8; therefore, rejected in the same manner as applied above.
Allowable Subject Matter
Claims 6, 9-10, 16, and 19-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. McCarten (US 20230134194) teaches: time centering module 100 determines a start of the integration time and an end of the integration time in each image capture stored in the time centering memory in the rolling shutter implementation. The time centering module 100 use the start and end of the integration time to derive the midpoint in time for each image capture to be merged into the HDR image capture
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WASSIM MAHROUKA whose telephone number is (571)272-2945. The examiner can normally be reached Monday-Thursday 8:00-5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen Koziol can be reached at (408) 918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WASSIM MAHROUKA/Primary Examiner, Art Unit 2665
Prosecution Insights