DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This action is responsive to applicant's amendment and remarks received on 05/21/2026.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1–6 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites "a sensor configured to detect a distance to an object in a real space around the vehicle," then recites "the distance to the object in the real space measured by the sensor." The claim refers to a measured distance where only a detected distance was introduced. It is unclear whether the two terms denote the same operation or whether a further measuring operation is required. The same inconsistency appears in claims 5 and 6. Claims 2–4 are rejected as depending from claim 1 and thereby incorporating the same indefiniteness.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 is/are rejected under 35 U.S.C. 103 as being unpatentable over Huebner (US 2012/0262580 A1) in view of Liu (US 2004/0001146 A1).
Regarding claim 1, Huebner discloses a video display device (Abstract; [0007]; FIG. 1) comprising:
a camera with a wide-angle lens configured to capture a two-dimensional wide-angle video around a vehicle (Huebner discloses fisheye cameras having a horizontal opening angle of more than 170°, and further that the cameras can include any image recording device that includes or can be combined with a wide-angle lens, such as a fisheye lens, positioned about the vehicle perimeter ([0021]; [0028]).);
a sensor configured to detect a distance to an object in a real space around the vehicle (Huebner discloses a sensor that gathers distance information of objects located in the vehicle's surroundings, including a PMD-sensor, an ultrasound sensor, and a radar sensor ([0026]; [0028]).);
a controller connected to the camera and the sensor (Huebner discloses image processing device 20 receiving image data from cameras 11–14 and including a second interface 22 that receives signals from distance sensors 23–26 ([0028]).);
a display device configured to display video based on signals output from the controller (Huebner discloses that processing device 20 combines the images such that a surround view is generated that can be displayed on display 30 ([0028]).);
wherein the controller is configured to calculate a position of the object in the wide-angle video captured by the camera based on the distance to the object in the real space measured by the sensor (Huebner discloses that the PMD-sensor provides distance information of objects located in an image plane of the sensor, and that the processing device can then render the objects in the surround view in a location representative of an actual location with respect to the vehicle ([0026]). Huebner further discloses that additional distance-measuring sensors may be used to fine tune the simulated predetermined shape, or correct positions and distortions of objects in the simulated predetermined shape to more accurately reflect spatial realities ([0037]).),
and while a position in the wide-angle video other than the calculated position of the object remains distorted after correction of the distortion (Huebner discloses that objects located in the vehicle's surroundings located where a portion of the simulated predetermined shape would exist are generated without distortion by the processing device, whereas points located outside or inside the simulated predetermined shape can be generated and displayed with a distortion ([0034]; see also [0025], objects may be displayed with a little distortion). Huebner thus discloses a display in which content at the reference surface is rendered undistorted while other content remains distorted, and further discloses adjusting that reference surface using sensor-measured object distance so that the object is accurately visualized ([0025]; [0026]; [0037]).).
Huebner does not expressly disclose correcting a distortion of the wide-angle video to cause the calculated position of the object in the wide-angle video to become horizontal on a screen of the display device.
In an analogous art of wide-angle image correction, Liu discloses a real-time wide-angle image correction system that generates a warp table from the pixel coordinates of a wide-angle image and applies the warp table to produce a corrected wide-angle image (Abstract; [0012]). Liu discloses that the correction is performed with reference to user-specified source curves that define common real-world horizontal features, such as the tops of people's heads and the edge of a table ([0077]; [0078]). Liu discloses a warping factor α which determines the degree to which pixels on the source curves are pulled toward the straight lines connecting their end points, with α = 1 pulling those pixels fully onto the straight lines ([0078]; [0079]). Liu further discloses that the SVU scaling functions induce vertical shear that is visually perceived as slanting and distorted horizontal lines, and that a horizontal distortion correction is therefore performed to correct the image for horizontal distortion ([0066]; [0067]). Critically, Liu discloses that this correction is spatially selective. The horizontal distortion correction module divides the preliminary warp table into sections and scales each section in the vertical direction according to a scaling factor specific to that section, the scaling factors being different between sections and usually different for at least one section ([0091]). Liu discloses a first section defined between the bottom source curve and the top source curve, which is scaled using the main scaling factor r(x), and second and third sections outside the source curves, which are scaled less, using a secondary scaling factor s that is a different value than the main scaling factor, the sections being joined by cubic splines ([0092]–[0094]; FIG. 9). Liu claims this arrangement expressly (Liu claims 12, 13, 29, 33, 34). Liu further discloses that the SVU scaling function is not a perfect uniform scaling everywhere, that only a uniform global scaling function could be, and that this lack of local uniform scaling causes objects in the image to appear stretched ([0087]; [0088]).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Huebner's surround view system to perform distortion correction using Liu's warp-table technique, with the reference source curve located at the object position Huebner computes from measured distance, so that the object position becomes horizontal on the display. The motivation is expressly supplied by Huebner: Huebner's wide-angle lenses generate distorted images of the surroundings ([0028]), Huebner seeks to correct positions and distortions of objects to more accurately reflect spatial realities using distance-measuring sensors ([0037]), and Huebner already renders objects at locations representative of their actual locations based on measured distance ([0026]). Liu supplies a known, real-time technique for correcting wide-angle distortion about a selectable reference, that reference being a specified source curve ([0078]). Selecting the sensor-derived object position as that reference, rather than requiring a user to specify it, is the simple substitution of one known reference-selection mechanism for another to obtain the predictable result of rendering the object of interest undistorted and horizontally aligned — the very outcome Huebner identifies as desirable.
Claim(s) 2–6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Huebner in view of Liu as applied to claim 1 above, and further in view of Gupta (US 2015/0049193 A1).
Regarding claim 2, Huebner in view of Liu discloses the video display device according to claim 1, but does not expressly disclose wherein the controller is configured to determines determine whether the object exists around the vehicle and in response to determining the object not existing around the vehicle, correct the distortion of the wide-angle video to cause a predetermined position in the wide-angle video to become horizontal on the screen of the display device.
Liu further discloses correcting distortion with reference to a predetermined position independent of any object: Liu discloses a symmetric SVU scaling function in which the bottom source curve is the mirror of the top source curve, the source curves pass through the image corners, and the curves have a minimum or maximum at the center vertical scanline ([0098]). Liu thereby discloses a default, geometrically predetermined reference that is straightened without reliance on any detected object.
In the analogous art of vehicular camera image processing, Gupta discloses a controller that determines whether an object or feature is detected at an expected pixel location and, when it is not, adjusts or shifts the image processing accordingly (Gupta, [0011]; [0150]–[0153]). Gupta discloses a predetermined horizontal reference condition: when the camera is properly aligned, the central vanishing point lies at a preordained pixel position and the vanishing line is "perfectly horizontal" in the image plane at a preordained vertical pixel height (Gupta, [0072]; [0148]). Gupta further discloses an express conditional branch at step 170D of FIG. 20, selecting between a vanishing-line reference and an object-derived reference according to whether the object cue is available (Gupta, [0125]; [0133]–[0134]).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to configure the controller of Huebner as modified by Liu to determine whether an object exists around the vehicle and, when no object is present, to perform the correction using Liu's predetermined symmetric source-curve reference (Liu, [0098]) or Gupta's preordained horizontal reference (Gupta, [0072]). The motivation is to maintain a stable dewarping reference when an object-based cue is unavailable, preserving a consistent and interpretable display under varying scene conditions — the function Gupta assigns to expected and preordained pixel locations (Gupta, [0148]; [0150]–[0153]).
Regarding claim 3, Huebner in view of Liu and Gupta discloses the video display device according to claim 2, wherein the position in the wide-angle video other than the calculated position of the object corresponds to a lane boundary line in the real space (Huebner discloses that road markings can be accurately projected on the bottom of the simulated predetermined shape (Huebner, [0034]); that lines of a parking lot located on the ground are depicted and positioned in the surround view (Huebner, [0035]; FIG. 4); and that the driver can observe the lane markings using the surround view to stay in a lane and change lanes (Huebner, [0036]). Huebner further discloses that points located outside or inside the simulated predetermined shape are displayed with a distortion (Huebner, [0034]). Gupta likewise discloses ground-plane parallel lines projected into the image plane as non-parallel lines (Gupta, [0070]–[0072]; FIGS. 2A–3B). In the combination, the position other than the calculated object position that remains distorted corresponds to a lane boundary line, since Huebner's displayed scene includes lane and parking-lot boundary markings distributed across the field of view, only a portion of which coincides with the correction reference. The recitation additionally characterizes the content of the scene imaged by the claimed apparatus rather than any structural element of it, and does not distinguish the claimed apparatus from that of the combination. See MPEP 2115.).
Regarding claim 4, Huebner in view of Liu and Gupta discloses the video display device according to claim 3, wherein the predetermined position is a position of an optical axis (Gupta's predetermined reference is anchored to the optical axis by definition. Gupta defines the central vanishing point as the point at which ground-plane parallel lines parallel to the camera optical axis appear to converge in the image plane (Gupta, [0065]). Gupta defines the principal point as the central vanishing point of the camera at its nominal installed position and orientation, and states that the principal point is an intrinsic camera parameter provided as part of the manufacturing data (Gupta, [0066]). Gupta sets forth the intrinsic parameter matrix K, in which axisX and axisY are the coordinates of the principal point (Gupta, [0114], eq. (2)). Gupta computes the vertical distance from the principal point to the vanishing line as the alignment measure (Gupta, [0112]; FIG. 19), and discloses that at nominal alignment the vanishing line is perfectly horizontal at a preordained vertical pixel height (Gupta, [0072]; [0148]). It would have been obvious to select the optical-axis position as the predetermined position of claim 2, because it is a fixed, manufacturer-supplied reference available without scene analysis (Gupta, [0066]), yielding the predictable result of a stable default alignment datum. The present specification identifies the position of the optical axis and the center of the image as suitable predetermined positions, PGPUB Specification, [0020], confirming that this selection is a choice among known alternatives.).
Regarding claim 5, Huebner in view of Liu and Gupta discloses the video display device according to claim 4, wherein the controller is configured to, in response to determining the object existing around the vehicle, calculate the position of the object in the wide-angle video captured by the camera based on the distance to the object in the real space measured by the sensor (Huebner discloses receiving distance information for objects located in the image plane of the PMD-sensor and rendering those objects at a location representative of their actual location (Huebner, [0026]) — i.e., calculating object position from measured distance upon the object being present. Gupta discloses that when an object is in the overlapping area, the system determines the pixel location of the detected object and thereby the expected pixel location (Gupta, [0011]), and that responsive to determining the location of a detected object such as a vehicle, mailbox, sign, pole, or curb, the system determines where that object should appear (Gupta, [0150]–[0151]). Configuring the controller to calculate the object position in response to determining that the object exists is the complementary branch of the conditional logic already required by claim 2 and would have been obvious, yielding the predictable result that no calculation is performed where no input to the calculation exists.).
Regarding claim 6, Huebner in view of Liu and Gupta discloses the video display device according to claim 5, wherein the controller is configured to calculate, as the position of the object in the wide-angle video captured by the camera, a coordinate point of the object on the two-dimensional wide-angle video based on a relationship between the distance to the object in the real space measured by the sensor and a field of view of the camera (Huebner discloses that the PMD-sensor provides distance information for objects located in an image plane of the sensor, which the processing device uses to render the objects at their representative locations (Huebner, [0026]), thereby resolving measured distance to an image-plane location. Huebner discloses the camera field of view as a horizontal opening angle of more than 170° (Huebner, [0021]). Gupta discloses the governing relationship between real-space position and image-plane coordinates. By using the intrinsic parameters of a camera, the pixel position of a feature point is resolved to a physical location (X, Y, Z), and conversely a feature point is associated with a nominal physical position based on those intrinsic parameters. Gupta, [0079]. The intrinsic parameters comprise the principal point, focal length, and lens map. Gupta, [0089]. Gupta sets forth the camera calibration equation mapping real-space coordinates to image coordinates through the intrinsic parameter matrix K, the rotation matrix R, and the translation matrix T (Gupta, [0114], eqs. (1)–(2)). Gupta provides a worked example computing expected pixel coordinates for an object of known physical location. Gupta, [0153]. Huebner discloses the measured distance to the object (Huebner, [0026]; [0028]). Liu likewise addresses the field of view of a wide-angle camera and its effect on the resulting image coordinates (Liu, [0002]; [0005]–[0006]). It would have been obvious to calculate, as the position of the object in the wide-angle video, a coordinate point on the two-dimensional wide-angle video based on the relationship between the measured distance and the camera field of view, in order to locate the object in the distorted image plane with the accuracy Huebner requires for rendering objects at representative locations (Huebner, [0026]), and correcting positions and distortions to reflect spatial realities (Huebner, [0037]).).
Response to Arguments
Applicant's arguments filed 05/21/2026 have been fully considered but they are not persuasive.
Applicant argues that Liu performs horizontal distortion correction "for the entire image," relying on the sentence at Liu [0068] describing the warp table as producing a distortion-free wide-angle image, and concludes that the combination would produce a distortion-free image rather than one in which other positions remain distorted. The characterization is contradicted by Liu's operative disclosure.
First, Liu's horizontal distortion correction is expressly sectional and non-uniform. Liu [0091] states that the module divides the preliminary pixel coordinates into sections and scales each according to a scaling factor specific to that section, that the factors may be different between sections, and that they are usually different for at least one section. Liu [0092]–[0094] and FIG. 9 describe three sections: a first between the bottom and top source curves scaled by the main factor r(x), and second and third sections outside the source curves scaled less, using a secondary factor that is "a different value than the main scaling factor." Liu claims this arrangement (Liu, claims 12, 13, 29, 33, 34). The sectional two-factor scheme is therefore not incidental to Liu's disclosure; it is the subject matter Liu claimed as its invention.
Second, the reference about which Liu straightens content is a selected position, not the whole image. Liu [0077]–[0078] disclose that the source curves are specified curves defining common real-world horizontal features, and that the warping factor determines how far pixels on those curves are pulled to the straight lines between their endpoints. Content not on the selected source curves is scaled by the secondary factor and is not so straightened.
Third, Liu disclaims the global result Applicant attributes to it. Liu [0087] states that the scaling function "is not a perfect uniform scaling everywhere," and that the only function that is a perfect uniform scaling everywhere is a uniform global scaling function. Liu [0088] states that this lack of local uniform scaling causes objects to appear stretched. Liu [0012] and [0072] describe the output as virtually free of distortion, and Liu [0099] reports that after correction very little stretching is observed — residual distortion in each case. Liu characterizes the operation as relaxing the uniformity of the vertical scaling (Liu, [0067]; [0090]).
As to FIGS. 10A–10B, on which Applicant relies: Liu's own description states that the image is less distorted after processing and that very little stretching is observed (Liu, [0099]). Those figures are presented to illustrate mitigation of swimming motion during panoramic viewing (Liu, [0097]–[0099]), and are not the figures by which Liu illustrates the sectional scaling scheme, which appears at FIG. 9 and the accompanying text.
The single sentence at Liu [0068] appears in Liu's operational overview and is a general characterization of purpose. It does not override Liu's detailed disclosure of how the correction is performed. A reference is evaluated for all that it teaches to a person of ordinary skill.
Applicant's "even if combinable" argument fails for the same reason: because Liu's correction is sectional and admittedly non-uniform, the combination does not and cannot produce the globally distortion-free image Applicant posits. Neither reference criticizes or discourages leaving residual distortion outside the corrected region; both affirmatively disclose it.
Applicant argues that Huebner [0037] relates to correcting object positions and distortions generally, not to selectively correcting at the object position while leaving other areas distorted.
Applicant does not address Huebner [0034], which states that objects located where a portion of the simulated predetermined shape would exist are generated "without distortion" by the processing device, while points located outside or inside that shape "can be generated and displayed with a distortion." See also Huebner, [0025]. Huebner therefore expressly discloses a display in which content at the reference surface is undistorted while other content remains distorted — the selective behavior Applicant contends is absent from the art.
Read with Huebner [0026] (rendering objects at representative locations using PMD-sensor distance data) and [0037] (additional distance sensors used to fine tune the simulated predetermined shape, or correct positions and distortions of objects), Huebner discloses adjusting the correction reference using sensor-measured object distance so that the object is accurately visualized while other points remain distorted. The limitation added to claim 1 is met by the primary reference, with Liu supplying the display-plane horizontal alignment.
Applicant traverses claim 2 solely on the asserted deficiencies of Huebner and Liu with respect to claim 1, and traverses claims 3–6 solely by reference to claim 1 and to unspecified "additionally recited features." As no deficiency in the treatment of claim 1 has been shown, and as no specific argument is directed to the additional limitations, these arguments are not persuasive. Claims 3–6 are rejected on the grounds newly set forth above.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RAJSHEED O BLACK-CHILDRESS whose telephone number is (571)270-7838. The examiner can normally be reached M to F, 10am to 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Quan-Zhen Wang can be reached at (571) 272-3114. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RAJSHEED O BLACK-CHILDRESS/Examiner, Art Unit 2685
Prosecution Insights