DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-10, 13-14, 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Su et al (Improved Cross-Ratio Invariant-Based Intrinsic Calibration of A Hyperspectral Line-Scan Camera) in view of Kroeger (US 2020/0005489).
Regarding Claim 1, Su et al teaches an apparatus for image processing (Pika XC-2 Hyperspectral line scan camera and analysis; Fig 1-3 and 3. Calibration Method) configured to:
determine, from a first image and a second image received from a sensor (first image and second image from a hyperspectral line scan camera are obtained; Fig 1-3 and 3.1 Camera Model for a Line-Scan Camera ¶ 1, 4.1 Simulation ¶ 2-3), a first plurality of two-dimensional points (a plurality of camera observation points are obtained for the first image and second image (X is identified as a fixed number with the y identified as the variable plurality of observations y1, y2.. yn); Fig 2, 3 and 3.2 Calibration Target ¶ 2-3),
wherein the first plurality of two-dimensional points are represented in the first image and correspond to two-dimensional points represented in the second image (the camera observation points y1, y2..yn correspond between the images from the different camera view angles; Fig 2-4 and 3.3 Signal Processing , 4.1 Simulation ¶ 2-3); and
process an additional image received from the sensor using the at least one estimated intrinsic parameter of the sensor (the camera intrinsic parameters are considered in the analysis of the camera view angled images; Fig 2-4 and 3.5 Nonlinear Optimization for Each Camera Pose ¶ 2-4, 4.1 Simulation ¶ 2-3).
Su et al does not explicitly teach the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: apply a non-linear lens distortion function associated with the sensor to the first plurality of two-dimensional points to adjust the first plurality of two-dimensional points; determine a plurality of epipolar constraints based on the adjusted first plurality of two-dimensional points; and estimate at least one intrinsic parameter of the sensor based on the plurality of epipolar constraints.
Kroeger is analogous art pertinent to the technological problem addressed in the current application and teaches the apparatus (vehicle 502 with computing device 504; Fig 5 and ¶ [0054]-[0056]) comprising: at least one memory (memory 518; Fig 5 and ¶ [0056]); and at least one processor (processor 516; Fig 5 and ¶ [0056]) coupled to the at least one memory and configured (the memory 518 is communicatively coupled with processors 516 to execute components 520-538; Fig 5 and ¶ [0056]) to:
apply a non-linear lens distortion function associated with the sensor (images may be captured by a single camera ¶ [0105]) to the first plurality of two-dimensional points to adjust the first plurality of two-dimensional points (point pairs from the first image and the second image to calibrate data for non-linear camera lens distortion and create reprojection of the first points 212a, 216, 228 to the second points corresponding to the first 208a and second image 208b with estimated depths 238, 240, 242 performed by the calibration data component 534 and extrinsic calibration component 536; Fig 2, 5 and ¶ [0038], [0040]-[0041], [0064]-[0065]);
determine a plurality of epipolar constraints based on the adjusted first plurality of two-dimensional points (calibration data 534 may be obtained including calibration angles, mounting locations, height, direction, yaw, tilt, pan, and similar (epipolar constraints) to determine the epipolar geometry by extrinsic calibration component 536; Fig 5 and ¶ [0065]-[0066]); and
estimate at least one intrinsic parameter of the sensor based on the plurality of epipolar constraints (an intrinsic calibration component 538 can determine a correction function to calibrate intrinsic characteristics of the camera, which may be based on the measured points to undistorted points (epipolar derived constraints); Fig 5 and ¶ [0068]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Su et al with Kroeger including the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: apply a non-linear lens distortion function associated with the sensor to the first plurality of two-dimensional points to adjust the first plurality of two-dimensional points; determine a plurality of epipolar constraints based on the adjusted first plurality of two-dimensional points; and estimate at least one intrinsic parameter of the sensor based on the plurality of epipolar constraints. By calibrating non-linear cameras by considering points representative of the whole image, and performing calibration using epipolar geometry constraints, calibration is performed for both intrinsic and extrinsic parameters of non-linear cameras, thereby improving the functioning of the calibration resulting in improved functionality and use of the sensor data, such as for a perception system of an autonomous vehicle, as recognized by Kroeger (¶ [0014]-[0017]).
Regarding Claim 2, Su et al in view of Kroeger teach the apparatus of claim 1 (as described above), wherein the at least one processor is configured to time filter the at least one intrinsic parameter of the sensor (Kroeger, the calibration data components 534 can store a log of sensor calibration (intrinsic parameters) and the correction function to calibrate intrinsic sensor 506 components via the intrinsic calibration component 538 may be based on a particular time frequency (thereby a BRI interpretation of time filtering); Fig 5 and ¶ [0064], [0068], [0084]).
Regarding Claim 4, Su et al in view of Kroeger teach the apparatus of claim 1 (as described above), wherein the at least one processor (Kroeger, processor 516; Fig 5 and ¶ [0056]) is configured to:
determine, from a third image and a fourth image received from the sensor, a second plurality of two-dimensional points (Su et al, additional 2D frame camera view angle images are acquired (including a third and fourth image) with corresponding 2D (fixed x, variable y) camera observation points; Fig 2, 3 and 3.1 Camera Model for a Line-Scan Camera § 1-2, 3.2 Calibration Target ¶ 2-3),
wherein the second plurality of two-dimensional points are represented in the third image and correspond to two-dimensional points represented in the fourth image (Su et al, the camera observation points y1, y2 correspond between the images from the different camera view angles; Fig 2-4 and 3.3 Signal Processing , 4.1 Simulation ¶ 2-3; and
apply the non-linear lens distortion function associated with the sensor to the second plurality of two-dimensional points to adjust the second plurality of two-dimensional points (Kroeger, non-linear camera data lens distortion correction (¶ [0038]) may be applied to the first and second images for reprojection (adjustment correction) of the point pairs and is repeated for additional images (third and fourth images) through an iterative process; Fig 2, 5 and ¶ [0040], [0043]-[0047], [0061], [0064]-[0066]);
wherein the determination of the plurality of epipolar constraints is additionally based on the adjusted second plurality of two-dimensional points (Kroeger, calibration data 534 may be obtained including calibration angles, mounting locations, height, direction, yaw, tilt, pan, and similar to determine the epipolar geometry by extrinsic calibration component 536 based on the calibration points; Fig 2, 5 and ¶ [0043]-[0047], [0061]-[0066]).
Regarding Claim 5, Su et al in view of Kroeger teach the apparatus of claim 1 (as described above), wherein the sensor is an image sensor (Su et al, the sensor is a line scan camera that captures images; Fig 1, 2 and 1. Introduction ¶ 5, 3. Calibration Method ¶ 1).
Regarding Claim 6, Su et al in view of Kroeger teach the apparatus of claim 5 (as described above), wherein the at least one intrinsic parameter includes at least one of a focal length or a principal point of the image sensor (Su et al, the sensor intrinsic parameters include focal lengths in the horizontal and vertical axis; 3.1 Camera Model for a Line-Scan Camera ¶ 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Su et al with Kroeger including wherein the at least one intrinsic parameter includes at least one of a focal length or a principal point of the image sensor. By managing intrinsic parameters of the sensor during calibration, the optical image system may perform better after calibration resulting in more accurate range photogrammetry, as recognized by Su et al (2. Related Work ¶ 6-7).
Regarding Claim 7, Su et al in view of Kroeger teach the apparatus of claim 1 (as described above), wherein the at least one processor is configured to: receive temperature data associated with a temperature of the sensor (Kroeger, calibration characteristics of the sensor data may include temperature; ¶ [0015], [0035], [0075]); and determine to update the at least one intrinsic parameter of the sensor based on the temperature (Kroeger, the intrinsic calibration may be based on temperature fluctuations and serve as a source for correcting distortion parameters based on temperature fluctuations; ¶ [0035]).
Regarding Claim 8, Su et al in view of Kroeger teach the apparatus of claim 1 (as described above), wherein the at least one processor (Kroeger, processor 516; Fig 5 and ¶ [0056]) is configured to: determine to update the at least one intrinsic parameter of the sensor based on a period of elapsed time from a previous update exceeding a predetermined threshold (Kroeger, the vehicle can send sensor data to the computing devices 542 at a particular frequency after a laps of a predetermined (threshold) period of time; Fig 5 and ¶ [0084]).
Regarding Claim 9, Su et al in view of Kroeger teach the apparatus of claim 1 (as described above), wherein the at least one processor (Kroeger, processor 516; Fig 5 and ¶ [0056]) is configured to: process the plurality of epipolar constraints to reduce a goal function associated with the at least one intrinsic parameter of the sensor (Kroeger, epipolar data from the extrinsic calibration may be used for the intrinsic calibration with a goal to exclude point pairs with error equal or above a threshold error; Fig 5 and ¶ [0064]-[0065], [0068]-[0069]),
wherein the at least one estimated intrinsic parameter is based on the plurality of epipolar constraints and the reduced goal function (Kroeger, an intrinsic calibration component 538 can determine a correction function to calibrate intrinsic characteristics of the camera, which may be based on the measured points to undistorted points (epipolar derived constraints); Fig 5 and ¶ [0065]-[0069]).
Regarding Claim 10, Su et al in view of Kroeger teach the apparatus of claim 9 (as described above), wherein the at least one processor is configured to: process the plurality of epipolar constraints to reduce the goal function associated with the at least one intrinsic parameter of the sensor (Kroeger, the goal may be to reduce the number of error point pairs by optimizing a correction matrix for the intrinsic calibration, with the point pairs based on epipolar geometry-based calibration; Fig 5 and ¶ [0065]-[0066], [0069]), wherein the goal function is additionally associated with a relative motion of the sensor (Kroeger, sensor data may be acquired while the vehicle is in motion, thereby data is generated based on a relative motion of the sensor; Fig 5, 6 and ¶ [0087]-[0088]).
Regarding Claim 13, Su et al in view of Kroeger teach a method for image processing (Su et al, sensor calibration based on image data; Fig 1-3 and 3. Calibration Method; Kroeger, execution of components 520-538; Fig 5 and ¶ [0056]), the method comprising: steps identical to claim 1 (as described above).
Regarding Claim 14, Su et al in view of Kroeger teach the method of claim 13 (as described above), further comprising steps identical to claim 2 (as described above).
Regarding Claim 16, Su et al in view of Kroeger teach the method of claim 13 (as described above), further comprising steps identical to claim 4 (as described above).
Regarding Claim 17, Su et al in view of Kroeger teach the method of claim 13 (as described above), wherein limitations are claimed identical to claim 5 (as described above).
Regarding Claim 18, Su et al in view of Kroeger teach the method of claim 17 (as described above), f wherein limitations are claimed identical to claim 6 (as described above).
Regarding Claim 19, Su et al in view of Kroeger teach the method of claim 13 (as described above), further comprising steps identical to claim 7 (as described above).
Regarding Claim 20, Su et al in view of Kroeger teach a non-transitory computer-readable medium having stored thereon instructions (Kroeger, the memory 518 stores components 520-538; Fig 5 and ¶ [0056]) that, when executed by at least one processor (Kroeger, processor 516 executes components 520-538; Fig 5 and ¶ [0056]), cause the at least one processor to: method for image processing, the method comprising: steps identical to claim 1 (as described above).
Claims 3, 15 are rejected under 35 U.S.C. 103 as being unpatentable over Su et al (Improved Cross-Ratio Invariant-Based Intrinsic Calibration of A Hyperspectral Line-Scan Camera) in view of Kroeger (US 2020/0005489) and Rasheed et al (US 12,159,467).
Regarding Claim 3, Su et al in view of Kroeger teach the apparatus of claim 1 (as described above), including the at least one processor (Kroeger, processor 516; Fig 5 and ¶ [0056]).
Su et al in view of Kroeger does not teach to time filter the at least one intrinsic parameter of the sensor using at least one of an average filter, a Kalman filter, or a histogram based filter.
Rasheed et al is analogous art pertinent to the technological problem addressed in the current application and teaches to time filter the at least one intrinsic parameter of the sensor using at least one of an average filter, a Kalman filter, or a histogram based filter (a Kalman filter is used to estimate optimal time offset based on sensor clocks for imaging and sensor data synchronization, influencing imaging timing (therefore the Kalman filter is the time filter); col 21 ln 14-24).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Su et al in view of Kroeger with Rasheed et al including to time filter the at least one intrinsic parameter of the sensor using at least one of an average filter, a Kalman filter, or a histogram based filter. By using Kalman filtering, an optimal time offset can be estimated, thereby accounting for and reducing any time drift causing temporal alignment offset of the imaging, as recognized by Rasheed et al (col 21 ln 14-31).
Regarding Claim 15, Su et al in view of Kroeger teach the method of claim 13 (as described above), further comprising steps identical to claim 3 (as described above).
Claims 11-12 are rejected under 35 U.S.C. 103 as being unpatentable over Su et al (Improved Cross-Ratio Invariant-Based Intrinsic Calibration of A Hyperspectral Line-Scan Camera) in view of Kroeger (US 2020/0005489) and Ren et al (US 2024/0104941).
Regarding Claim 11, Su et al in view of Kroeger teach the apparatus of claim 10 (as described above), including detecting motion of the sensor via rotation and translation of the camera pose (Su et al, 3.5 Nonlinear Optimization for Each Camera Pose ¶ 3-4).
Su et al in view of Kroeger do not teach wherein the relative motion of the sensor is represented as a matrix or a vector associated with a change in orientation of the sensor and a translation vector associated with a change in location of the sensor.
Ren et al is analogous art pertinent to the technological problem addressed in the current application and teaches wherein the relative motion of the sensor is represented as a matrix or a vector associated with a change in orientation of the sensor and a translation vector associated with a change in location of the sensor (coordinate positions of a captured image frame 300 is used to estimate rotation and translation vectors representing the pose (motion) of the sensor and used for sensor calibration based on the calibration points 310 to validation points 320 and deviation (epipolar data) based on the rotation-translation transform represented as vectors and matrix; Fig 1-3 and ¶ [0052]-[0053]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Su et al in view of Kroeger with Ren et al including wherein the relative motion of the sensor is represented as a matrix or a vector associated with a change in orientation of the sensor and a translation vector associated with a change in location of the sensor. By using vector and matrix representation for the image sensor points used for calibration, the calibration accuracy may be quickly and easily tracked and allowing for quick positional adjustments, thereby improving sensor calibration through calibration error reduction, as recognized by Ren et al (¶ [0025]-[0026]).
Regarding Claim 12, Su et al in view of Kroeger and Ren et al teach the apparatus of claim 110 (as described above), wherein the at least one processor is configured to: estimate the relative motion of the sensor based on the reduced goal function (the calibration errors computed based on the rotation-translation transform may be tracked and associated with the sensor 101 pose changing over time as a result of vehicle vibrations (estimate relative motion of sensor); Fig 3 and ¶ [0053]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Jia et al (US 2019/0147625) teach a system and method for extrinsic calibration of cameras based on diffractive optical analysis including the use of identified data pairs of pixel coordinates to determine extrinsic parameters of the camera.
Dhome et al (US 2014/0267608) teach a system and method for calibration of a computer-based vision system including analysis of extrinsic and intrinsic parameters using epipolar geometry.
Livyatan et al (US 2015/0145965) teach an auto-calibration of stereo camera of a vehicle based on image comparison of coordinating points between images using epipolar geometry, with the camera coordinate points analyzed based on both extrinsic and intrinsic parameters.
Ricolfe-Viala et al (Depth Dependent High Distortion Lens Calibration) teach a method and system for non-linear camera lens distortion based on comparison of points to correction points and adjusting the camera in response to the distortion.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATHLEEN M BROUGHTON whose telephone number is (571)270-7380. The examiner can normally be reached Monday-Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATHLEEN M BROUGHTON/Primary Examiner, Art Unit 2661
Prosecution Insights