Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This action is responsive to applicant’s amendments and remarks received on 12/29/2025.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-16 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, 6-7, 9, 12, 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Hitachi et al. (JP 6035620 B2), hereinafter Hitachi, in view of Xie et al. (WO 2020258936 A1), hereinafter Xie.
Regarding claim 1, Hitachi teaches An image correction method comprising: (Para. 1 see "The present invention relates to a stereo camera system for recognizing an environment by photographing a subject with two or more cameras to calculate a three-dimensional position of the subject, and a mobile object equipped with the stereo camera system." Para. 5 see "when a mobile object equipped with a stereo camera system is traveling, external parameters may change due to physical impact and vibration on the stereo camera system. For this reason, it is necessary to recalibrate the external parameters in the stereo camera system in operation, but it is expensive to prepare an object with a known shape as a marker and perform special imaging during operation. For this reason, it is required to automatically calibrate external parameters using images taken during operation, which is called self-calibration." Para. 17 see " FIG. 2 shows a state where the above-described in-vehicle stereo camera system is mounted on the dump truck 111. The main parts of the in vehicle stereo camera system 100 and the environment recognition system control unit 109 in FIG. 1 are built in the in-vehicle controller 110 shown in FIG. This invehicle controller has a general hardware configuration such as a CPU, a RAM, and a storage unit."). collecting a plurality of images captured at different time points by an imaging device installed in a vehicle; (Para. 11 see "using images captured by each of the plurality of imaging units, and a time series." Para. 17 see "FIG. 1 is a block diagram illustrating a configuration of an in-vehicle stereo camera system 100" and "an image acquisition unit 102." Para. 20 see "an R image and an L image captured simultaneously by the imaging unit 101 are input to the image acquisition unit In step S <b> 303, an image of each frame is provided to the image storage unit In S304, the image of each frame stored in the image storage unit 103."). determining a plurality of feature points in a first image captured at a first time point among the plurality of images; (Para. 28 see "FIG. 8 illustrates a step of extracting feature points on both images of the Nth frame simultaneously captured by the image capturing units." Para. 29 see "A plurality of feature points are extracted from the left image of both images of the Nth frame captured by the imaging units 101a and 101b."). detecting corresponding points respectively matching the plurality of feature points in a second image captured at a second time point among the plurality of images; (Para. 20 see "both the image and the previous frame image are provided to the corresponding point detection unit The corresponding point detection unit 104 detects feature corresponding points of both images of the frame for which posture estimation is required." Para. 34 see "FIG. 9 is a schematic diagram illustrating an example of a result of a step (left image optical flow calculation step S1053 in FIG. 5) of detecting corresponding points from the Nth frame to the (N + 1) th frame captured by the left camera in the first embodiment. To describe the method of detecting corresponding points, feature points extracted from the previous frame (Nth frame) image based on the Lucas-Kanade optical flow extraction method for two images continuously captured by the left camera 101a of the imaging unit The corresponding points are detected in the rear frame (N + 1th frame) image."). determining movement information of the plurality of feature points based on the plurality of feature points and a result of the detecting; (Para. 12 see "The corresponding point detection unit calculates a corresponding point position of the Nth frame image from a set of a plurality of frame images captured by each of the plurality of imaging units, and a positional relationship between the Nth frame and the (N + 1) th frame for each of the left and right images."). determining a correction parameter based on the movement information; (Para. 20 see "FIG. 3 is a flowchart for explaining a processing example of the entire in-vehicle stereo camera system 100" and "the image of each frame stored in the image storage unit 103 is provided to the posture change determination unit 105 to determine whether or not the camera relative posture has changed." and "both the image and the previous frame image are provided to the corresponding point detection unit The corresponding point detection unit 104 detects feature corresponding points of both images of the frame for which posture estimation is required." and "In S306, the image correction unit 107 corrects the positions of the corresponding points of both images detected by the corresponding point detection unit 104 distorted by lens distortion or the like in each image. In S307, the parameter estimation unit 108 uses the set of corrected feature corresponding points provided by the image correction unit 107," and "In step S308, the posture adjustment necessity determination unit 106 determines whether manual correction of the camera relative posture is necessary." Para. 42 see "The parameter estimation unit 108 operates the set of corresponding points after correction provided by the image correction unit 107."). and performing image correction based on the correction parameter. (Para. 17 see "FIG. 1 is a block diagram illustrating a configuration of an in-vehicle stereo camera system 100" and "An in-vehicle stereo camera system 100 for an autonomous traveling dump truck includes a plurality of imaging units 101 (101a, 101b), an image acquisition unit 102, an image storage unit 103, a corresponding point detection unit 104, a posture change determination 105, and a posture adjustment necessity determination unit. 106, an image correction unit 107, and a parameter estimation unit 108,." Para. 40 see "The image correction unit 107 corrects the position of the corresponding points of the two images detected by the corresponding point detection unit 104 detected by the corresponding point detection unit 104 distorted due to the lens distortion and the position from the image center.").
Hitachi does not teach wherein detecting corresponding points includes: determining whether a number of detected corresponding points in the second image is less than a predetermined number, and continuing to detect corresponding points if the number of the detected corresponding points in the second image is less than the predetermined number;.
However, Xie teaches wherein detecting corresponding points includes: determining whether a number of detected corresponding points in the second image is less than a predetermined number, and continuing to detect corresponding points if the number of the detected corresponding points in the second image is less than the predetermined number; (Para. 15 see " the method further includes: determining whether the number of feature points extracted from the current frame is less than an expected threshold for feature matching, and If it is less than the expected threshold, triggering the processing of supplementing the feature points of the current frame." Para. 16 see "it can be judged whether the number of extracted feature points in the current frame meets the expected threshold for feature matching, and the feature points extracted in the current frame are directly used if they meet, and the process of supplementing feature points in the current frame is triggered again if it does not meet the requirements." Para. 196-199 see "1. The feature points are extracted on the current frame image acquired by the device 2 with default parameters, and the number of feature points extracted can be twice the number of feature points actually acquired by the SLAM system itself. 2. Check the number of feature points extracted in step 1. If the number of feature points is less than a specific expected threshold, skip to step 3, otherwise skip to step 4. 3. Reduce the screening threshold of the feature points, and add points (or increase the number of feature points in the current frame). 4. Perform feature descriptor extraction on the extracted feature points and return the extraction result.").
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hitachi to incorporate the teachings of Xie to continue to detect corresponding points based on a predetermined threshold. Doing so would predictably improve the robustness of the method by ensuring a sufficient quantity of corresponding points are generated/detected to determine movement information.
Regarding claim 4, Hitachi in view of Xie teaches The image correction method of claim 1.
In addition, Hitachi teaches wherein the detecting of the corresponding points comprises detecting a corresponding point for each of the plurality of feature points until the predetermined number of corresponding points are detected in the second image. (Para. 47 see "As a determination method, when the number of feature corresponding points calculated by the corresponding point detection unit 104 is smaller than the lower limit of the number of corresponding points necessary for calibration" and "When the ratio of the feature corresponding points calculated by the corresponding point detection unit 104 to the number of points is 20% or less, and the deviation from the center of the image of the centroid of the feature corresponding points detected by the corresponding point detection unit is 80%.").
Regarding claim 6, Hitachi in view of Xie teaches The image correction method of claim 1.
In addition, Hitachi teaches wherein the determining of the movement information comprises determining, in response to the number of the detected corresponding points in the second image being equal to or greater than the predetermined number, the movement information based on coordinates of feature points respectively matching the detected corresponding points among the plurality of feature points and coordinates of the detected corresponding points. (Para. 12 see "The corresponding point detection unit calculates a corresponding point position of the Nth frame image from a set of a plurality of frame images captured by each of the plurality of imaging units, and a positional relationship between the Nth frame and the (N + 1) th frame for each of the left and right images." Para. 47 see "The posture adjustment necessity determination unit 106 determines that a relative posture change exceeding an allowable range has occurred when a certain condition is not satisfied, and issues an alarm requesting manual system correction. As a determination method, when the number of feature corresponding points calculated by the corresponding point detection unit 104 is smaller than the lower limit of the number of corresponding points necessary for calibration" and "When the ratio of the feature corresponding points calculated by the corresponding point detection unit 104 to the number of points is 20% or less, and the deviation from the center of the image of the centroid of the feature corresponding points detected by the corresponding point detection unit is 80%.").
Regarding claim 7, Hitachi in view of Xie teaches The image correction method of claim 6.
In addition, Hitachi teaches wherein the determining of the movement information comprises: determining a motion vector using pairs of the coordinates of the feature points respectively matching the plurality of detected corresponding points and the coordinates of the plurality of detected corresponding points; (Para. 12 see "The corresponding point detection unit calculates a corresponding point position of the Nth frame image from a set of a plurality of frame images captured by each of the plurality of imaging units, and a positional relationship between the Nth frame and the (N + 1) th frame for each of the left and right images." Para. 34 see "FIG. 9 is a schematic diagram illustrating an example of a result of a step (left image optical flow calculation step S1053 in FIG. 5) of detecting corresponding points from the Nth frame to the (N + 1)th frame captured by the left camera" and "To describe the method of detecting corresponding points, feature points extracted from the previous frame (Nth frame) image based on the Lucas-Kanade optical flow extraction method for two images continuously captured by the left camera 101a of the imaging unit The corresponding points are detected in the rear frame (N + 1th frame) image. A small window centered on the feature point extracted from the Nth frame image is cut out, a search is made where the small window exists in the (N + 1) th image, and the partial differentiation of the image is performed." Para. 38 see " In the feature point search, simultaneous equations expressed by the following equations are solved, and corresponding points between the Nth frame image and the (N + 1) th frame image captured by the right image 101b are detected."). and determining the movement information of the plurality of feature points based on the motion vector. (Para. 40 see "The image correction unit 107 corrects the position of the corresponding points of the two images detected by the corresponding point detection unit 104 detected by the corresponding point detection unit 104 distorted due to the lens distortion and the position from the image center. For example, the internal parameters can be represented by the focal length {f x, f y}, the image center position {c x, c y}, distortion parameters {k 1, k 2, k 3, p 1, p 2} it can. The width of the image w, when the height of the image to be is h, distorted coordinate {u d, v d} of a point on the image and the coordinates {u d, v d} of a point on the corrected image and the relationship can be expressed by [Equation 5].").
Claim 9 is rejected under the same analysis as claim 1 above.
Claim 12 is rejected under the same analysis as claim 4 above.
Claim 14 is rejected under the same analysis as claim 6 above.
Claim 15 is rejected under the same analysis as claim 7 above.
Claims 2, 10 are rejected under 35 U.S.C. 103 as being unpatentable over Hitachi et al. (JP 6035620 B2), hereinafter Hitachi, in view of Xie et al. (WO 2020258936 A1), hereinafter Xie, and Kanhere et al. (US 20100322476 A1), hereinafter Kanhere.
Regarding claim 2, Hitachi in view of Xie teaches The image correction method of claim 1.
Hitachi does not teach wherein the collecting of the plurality of images comprises transforming the plurality of images captured at different time points into pyramid images or partial images showing parts thereof.
However, Kanhere teaches wherein the collecting of the plurality of images comprises transforming the plurality of images captured at different time points into pyramid images or partial images showing parts thereof. (Para. 72 see "Block 34 of FIG. 1 represents the selecting and tracking feature points." and "A coarse to fine pyramidal strategy allows for large image motions, and features are automatically selected, tracked, and replaced.").
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hitachi and Xie to incorporate the teachings of Kanhere to transform images into pyramids. Doing so would predictably save on computing resources (hardware, energy, or time) by using downsampling images to estimate regions of an image where the corresponding feature points are likely to reside.
Claim 10 is rejected under the same analysis as claim 2 above.
Claims 3, 11 are rejected under 35 U.S.C. 103 as being unpatentable over Hitachi et al. (JP 6035620 B2), hereinafter Hitachi, in view of Xie et al. (WO 2020258936 A1), hereinafter Xie, and Kawabe et al. (US 20220189065 A1), hereinafter Kawabe.
Regarding claim 3, Hitachi in view of Xie teaches The image correction method of claim 1.
Hitachi does not teach wherein the determining of the plurality of feature points comprises determining the plurality of feature points using at least one of a crack and a pattern appearing on a road surface contained in the first image.
However, Kawabe teaches wherein the determining of the plurality of feature points comprises determining the plurality of feature points using at least one of a crack and a pattern appearing on a road surface contained in the first image. (Para. 43 see "The characteristic point extraction unit 202 extracts the characteristic point by performing image processing to the captured image input from the image acquisition unit 201. The term “characteristic point” as used herein refers to the intersection point of edges such as the corner of walls, corner of curbs, corner of broken lines or corner of crosswalks in the image; that is, a characteristic point can also be referred to as a corner characteristic point.").
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hitachi and Xie to incorporate the teachings of Kawabe to use a pattern on the road when determining feature points. Doing so would make the determination of points predictably more accurate and reliable by using features in the image that are substantially different in color and shape (with distinct edges) from ambient features in the image.
Claim 11 is rejected under the same analysis as claim 3 above.
Claims 5, 8, 13, 16 are rejected under 35 U.S.C. 103 as being unpatentable over Hitachi et al. (JP 6035620 B2), hereinafter Hitachi, in view of Xie et al. (WO 2020258936 A1), hereinafter Xie, and Devitt et al. (US 20210158081 A1), hereinafter Devitt.
Regarding claim 5, Hitachi in view of Xie teaches The image correction method of claim 1.
Hitachi does not teach wherein the detecting of the corresponding points comprises: collecting at least one of vehicle speed and steering information as vehicle information; and detecting a corresponding point for each of the plurality of feature points in the second image captured at the second time point based on the collected vehicle information.
However, Devitt teaches wherein the detecting of the corresponding points comprises: collecting at least one of vehicle speed and steering information as vehicle information; and detecting a corresponding point for each of the plurality of feature points in the second image captured at the second time point based on the collected vehicle information. (Para. 114 see "The previous correspondence map (e.g., final correspondence map, validated correspondence map, etc.) is preferably used as the initial correspondence map; alternatively, new correspondence vectors can be calculated from the corresponding pixel's prior correspondence vector (e.g., based on odometry, interim egomotion movement, etc.), correspondence vectors can be selectively populated to the initial correspondence map, any suitable subset of the previous correspondence map can be propagated (e.g., the set of correspondence vectors from the previous correspondence map that are not nullified; the set of correspondence vectors that meet an initial correspondence vector criterion such as based on the quality of the previous correspondence vector; etc.), or the prior correspondence map can be otherwise used. This can be particularly useful for optical flow and/or feature tracking in live video, where the previous correspondence map from a prior frame (e.g., immediately prior frame, n frames prior, etc.) can be used as the initial correspondence map for the current frame (for the same camera).").
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hitachi and Xie to incorporate the teachings of Devitt to use vehicle speed or steering information in detecting the corresponding points. Doing so would predictably save on computing resources (hardware, energy, or time) by using this information to estimate regions of an image where the corresponding feature points are likely to reside.
Regarding claim 8, Hitachi in view of Xie teaches The image correction method of claim 1.
Hitachi does not teach wherein the determining of the correction parameter comprises: deriving a homography based on the movement information; and determining the correction parameter using a component of the derived homography.
However, Devitt teaches wherein the determining of the correction parameter comprises: deriving a homography based on the movement information; and determining the correction parameter using a component of the derived homography. (Para. 142 see "for each frame (e.g., image), determining a rectifying homography and determining an epipole. In an illustrative example, determining a rectifying homography can include H=KR.sup.TK.sup.−1, where H is the rectifying homography, K is a projection matrix, and R.sup.T is a rotation matrix. In an illustrative example, determining the epipole can include {right arrow over (e)}=π(K{right arrow over (t.sub.R)}), where e is the epipole, π is a projection operator representing conversion from homogeneous to regular coordinates, and t.sub.R is the translation of the object between frame t and t−1 (e.g., determined from the correspondence map).").
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hitachi and Xie to incorporate the teachings of Devitt to use homography when determining the correction parameter. Doing so would predictably increase accuracy by accounting for perspective distortion and/or discarding incorrect matches of corresponding points.
Claim 13 is rejected under the same analysis as claim 5 above.
Claim 16 is rejected under the same analysis as claim 8 above.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Anai et al. (US 20090175497 A1) discloses an apparatus and method for measuring in three dimensions by applying an estimating process to points corresponding to feature points in a plurality of motion image frames.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEXANDER J VAUGHN whose telephone number is (571) 272-5253. The examiner can normally be reached M-F 8:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ANDREW MOYER can be reached on (571) 272-9523. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALEXANDER JOSEPH VAUGHN/Examiner, Art Unit 2675
/EDWARD PARK/Primary Examiner, Art Unit 2675