DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 6/4/2026 has been entered.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim 11 is rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Rathi et al. (Pub. No. US 20190143896).
Regarding claim 11, Rathi teaches a processor (processor 18) comprising: one or more processing units to: generate, based at least on ego-motion (vehicle dynamics) of an ego-object (vehicle) in an environment (scene surrounding the vehicle), a representation (determined ratio) of at least one of an estimated rotation (rolled or pitched (skewed)) or an estimated translation of a suspension (suspension system of all wheels shock absorbers) of the ego-object (vehicle) [Para. 25 “Such a change in vehicle dynamics may be due to uneven or changed load distribution at the vehicle, such as a heavy load added at one side or at the rear of the vehicle or at the top of the vehicle causing the vehicle to tilt towards one side or rearward, or due to uneven passenger loading or the like or uneven vehicle suspension performance or the like.”; para. 35 “In FIG. 8, objects 30 are shown symbolized by boxes passing the scene surrounding the vehicle when the vehicle is driving forward.”; and Para. 40 “As an additional or alternative way to detect that the vehicle is not fully leveled horizontally (rolled in x axis) and/or has a pitch angle (in y axis) or the like, the system may utilize the compression ratio of the shock absorber system or suspension system of all wheels shock absorbers of the vehicle. Nowadays vehicles usually have sensors to detect the compression rate and usually this signal is provided via the vehicle's CAN or vehicle bus network, which makes it possible that the vision system is able to process this information. By such processing of suspension information (for the individual wheels of the vehicle) the system may determine uneven suspension of the corner regions of the vehicle and may correct the virtual (assumed) ground plane for the virtual top view vision projection in accordance with the determined ratio that the vehicle is rolled or pitched (skewed)”];
generate, using a transformation (affine transforms or other suitable transforms) based at least on the representation (determined ration) of the estimated rotation (rolled or pitched (skewed)) or the estimated translation, an ego-motion compensated projection (virtual top view vision projection) of frames of image data representing (representative of ) two or more overlapping views (overlapping regions in the adjacent camera fields of view) of the environment (scene surrounding the vehicle) [Para. 26 “The system is operable to process image data representative of overlapping regions in the adjacent camera fields of view to detect corresponding features.”; Para. 28 “The system of the present invention is operable to generate affine transforms or other suitable transforms that minimize the error in feature mapping from different views.”; Para. 29; Para. 35 “In FIG. 8, objects 30 are shown symbolized by boxes passing the scene surrounding the vehicle when the vehicle is driving forward.”; Para. 40 and 41]; and
generate a surround view visualization (provide or generate the stitched composite image) based at least on the ego-motion compensated projection (virtual top view vision projection) [Para. 24; Para. 29; para. 34 “Thus, utilizing the present invention, the stitching of the captured images (to provide or generate the stitched composite image) may be adapted for vehicle load changes and level changes and the like in real time and during normal operation of the vehicle”; and Para. 40 “By such processing of suspension information (for the individual wheels of the vehicle) the system may determine uneven suspension of the corner regions of the vehicle and may correct the virtual (assumed) ground plane for the virtual top view vision projection in accordance with the determined ratio that the vehicle is rolled or pitched (skewed)”].
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 10-12, 15-16, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over WANG et al. (Pub. No. US 2015/0332446) in view of Weston et al. (Pub. No. US 20230410528).
Regarding claim 1, WANG teaches generating, based at least on ego-motion (vehicle dynamics) of an ego-object (Vehicle) in an environment (world coordinates), a representation of at least one of an estimated rotation (Rotation dynamics/matrix) or an estimated translation (transition dynamics/vector) of a body (vehicle body) of the ego-object relative to a calibration state (Stationary orientation) of the body during calibration of one or more extrinsic sensor parameters (extrinsic parameters) [Para. 8-11, 29, 30, and 32]; and generating, using a transformation (rotation matrices R and translation vectors t) based at least on the representation of the estimated rotation or the estimated translation, an ego-motion compensated (corrected) projection (top-down view image) of frames of image data representing two or more overlapping views of the environment (areas around the vehicle) [Para. 24, 25, and 27].
However, WANG doesn’t explicitly teach the vehicle body as being the suspended mass.
Weston teaches generating (can be determined), based at least on ego-motion (vehicle) of an ego-object in an environment (driving surface 1003), a representation (differences in measured angle and the calibrated angle 1020) of at least one of an estimated rotation (pitch rotation 910) or an estimated translation of a suspended mass (sprung mass) of the ego-object relative to a calibration state (calibration of the camera 1002) of the suspended mass during calibration of one or more extrinsic sensor parameters (angle 1020 between the optical centerline 1014 and the horizon line 1012) [Para. 32, 102, and 105-108].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Wang’s surround-view calibration-correction algorithm by incorporating Weston’s determination of Estimated rotation (pitch rotation 910) of the suspended mass (sprung mass) from ego-motion (suspension travel from vehicle loading), represented b differences in measured angle and the calibrated angle 1020 relative to the unloaded condition established during calibration of the camera 1002, and supplying that calibration relative rotation to Wang’s camera coordinate transformation. This medication improves Wang’s image alignment by making its correction responsive to suspension induced departure of the camera carrying suspended mass, thereby reducing stitching misalignment caused by vehicle loading.
Regarding claims 2, Wang teaches generating (determines) a surround view visualization (surround-view image) based at least on (based on) the ego-motion (vehicle dynamics) compensated (corrected) projection [Para. 27].
Regarding claims 3 and 12, Wang teaches wherein the generating of the ego-motion compensated projection comprises applying the transformation (integrating the rotation matrix and the translation vector) to one or more calibration parameters (relationship between a vehicle coordinate system and a camera coordinate system) associated with a camera that captured (generate) a corresponding at least one frame of the frames of image data [Para. 11 and 24].
Regarding claims 10, 15, and 20, Weston further teaches wherein the method is performed by a system incorporating one or more virtual machines (VMs) [Para. 124 and 143].
Regarding claim 16, Wang teaches one or more processing units to estimate at least one of rotation information (pitch, roll) or translation information (height variation) corresponding to a vehicle body of an ego-object (vehicle) in an environment (areas around the vehicle 10) based on ego-motion (vehicle dynamics) of the ego-object [Para. 11 “The method includes reading measurement values from one or more sensors on the vehicle that identify a change in vehicle dynamics and defining the plurality of cameras and a vehicle body as a single reference coordinate system”; Para. 30; Para. 32 “Providing the matching feature points (u, v) for the same point from two cameras and solving the dynamic equations with the unknowns gives an estimate of the pitch α, roll β, and/or height variation of the vehicle 50 based on the distance between the points.”; Para. 25 “The cameras 12-18 generate images of certain limited areas around the vehicle 10 that partially overlap.” and para. 26]; and to generate (processes the image data to stitch the images together) an ego-motion compensated (corrected based on those changes to the vehicle dynamics) projection (single top-down view image) of frames of image data (frames of image data) representing two or more overlapping views (images from adjacent cameras) of the environment (areas around the vehicle 10) using a transformation (rotation matrices R and translation vectors t) based at least on the estimated rotation information (pitch and roll) or the estimated translation information (height variation) [para. 24 “The cameras 12-18 generate frames of image data at a certain data frame rate that can be stored for subsequent image processing in a video processing module (VPM) 20”; Para. 25; Para. 27 “The present invention proposes a system and method for integrating information available from sensors on the vehicle about the vehicle dynamics into the algorithm in the VPM 20 that determines the surround-view image using the cameras 12-18 so the image can be corrected based on those changes to the vehicle dynamics.”; para. 11; and para. 32 “Providing the matching feature points (u, v) for the same point from two cameras and solving the dynamic equations with the unknowns gives an estimate of the pitch α, roll β, and/or height variation of the vehicle 50 based on the distance between the points.”].
However, WANG doesn’t explicitly teach the vehicle body as being the suspended mass.
Weston teaches generating (can be determined), based at least on ego-motion (vehicle) of an ego-object in an environment (driving surface 1003), a representation (differences in measured angle and the calibrated angle 1020) of at least one of an estimated rotation (pitch rotation 910) or an estimated translation of a suspended mass (sprung mass) of the ego-object relative to a calibration state (calibration of the camera 1002) of the suspended mass during calibration of one or more extrinsic sensor parameters (angle 1020 between the optical centerline 1014 and the horizon line 1012) [Para. 32, 102, and 105-108].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Wang’s surround-view calibration-correction algorithm by incorporating Weston’s determination of Estimated rotation (pitch rotation 910) of the suspended mass (sprung mass) from ego-motion (suspension travel from vehicle loading), represented b differences in measured angle and the calibrated angle 1020 relative to the unloaded condition established during calibration of the camera 1002, and supplying that calibration relative rotation to Wang’s camera coordinate transformation. This medication improves Wang’s image alignment by making its correction responsive to suspension induced departure of the camera carrying suspended mass, thereby reducing stitching misalignment caused by vehicle loading.
Regarding claim 11, Wang teaches a processor comprising: one or more processing units to: generate (gives), based at least on ego-motion (pitch, roll and height variation) of an ego-object (vehicle) in an environment (areas around the vehicle 10), a representation (an estimate) of at least one of an estimated rotation (pitch and roll) or an estimated translation (height variation) [Para. 38, 25, 26, and 32].
However, Wang doesn’t explicitly teach of a suspension of the ego-object.
Weston teaches generate (determine), based at least on ego-motion of an ego-object (relative movement of an object rigidly coupled to and/or disposed on the strung mass of the vehicle 100) in an environment (driving surface 302), a representation (magnitude) of at least one of an estimated (estimate) rotation or an estimated translation (deflection) of a suspension (suspension component 104A, 104B, …) of the ego-object (vehicle 100) [Para. 57, 51, 53, 55, and 102].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Wang’s surround view calibration algorithm by incorporating Weston’s based estimation of suspension translation (deflection) from movement of a vehicle fixed feature relative to stationary surroundings, and supplying that estimate to the height related input of Wang’s calibration correction transformation. This medication improves Wang’s correction of load induced camera height changes, thereby improving alignment of overlapping camera images in the resulting surround view output.
Wang teaches generate (processes the image data to stitch the image together), using (employ) a transformation (rotation matrices R and translation vectors t) based at least on the representation (estimate) of the estimated rotation (pitch and roll) or the estimated translation (height variation), an ego-motion compensated (corrected based on those changes to the vehicle dynamics) projection (single top down view image) of frames of image data (frames of image data) representing two or more overlapping views (images from adjacent cameras) of the environment (areas around the vehicle 50) [Para. 24, 25, 11, 32, and 27], and
Wang also teaches generate (processes the image data to stitch the images together) a surround view visualization (surround-view image) based at least on the ego-motion compensated (corrected based on those changes to the vehicle dynamics) projection (single top-down view image) [Para. 25, and 27].
Claims 4 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over WANG et al. (Pub. No. US 2015/0332446) in view of Weston et al. (Pub. No. US 20230410528) further in view of Lu et al. (Pub. No. 2007/0067085).
Regarding claims 4 and 13, Wang in view of Weston does not explicitly teach the claim limitations.
However, Lu teaches wherein the generating (may be determined) of the representation (relative roll angle) of the estimated rotation (relative roll angle) of the suspended mass of the ego-object (vehicle body) is based at least on estimated stiffness (suspension stiffness) of the suspended mass in one or more rotational directions and detected acceleration (lateral acceleration) of the suspended mass in the one or more rotational directions (roll dynamics) [Para. 123, 125-128, and 135-138].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Wang’s suspension roll/pitch model, modified by Weston, which determines the vehicle body’s relative roll or pitch from suspension stiffness (suspension stiffness) and measured body acceleration (lateral acceleration). This modification improves Wang by making it pitch and roll compensation responsive to current suspension stiffness.
Claims 5 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over WANG et al. (Pub. No. US 2015/0332446) in view of Weston et al. (Pub. No. US 20230410528) further in view of Irwin et al. (Pub. No. US 2017/0080770).
Regarding claims 5 and 14, Wang in view of Weston does not explicitly teach the claim limitations.
However, Irwin teaches wherein the generating of the representation of the estimated rotation of the suspended mass of the ego-object comprises estimating rotation with respect to a ground surface using one or more suspension level sensors to measure displacement between the suspended mass and the ground surface [Para. 28, 30, and 39].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Wang’s surround view calibration correction method, modified by Weston, by incorporating Irwin’s suspension level sponsors at multiple body locations to determine estimated rotation from displacement relative to the ground surface and provide that rotation to Wang’s rotation dynamics correction. This modification improves Wang by grounding the correction in body, thereby improving roll compensation during cornering and unequal left to right.
Claims 6 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over WANG et al. (Pub. No. US 2015/0332446) in view of Weston et al. (Pub. No. US 20230410528) further in view of Doering et al. (Pub. No. US 2010/0017070).
Regarding claims 6 and 17, Wang in view of Weston does not explicitly teach the claim limitations.
However, Doering teaches wherein the generating of the representation of the estimated rotation (roll angle and roll rate of the vehicle) of the suspended mass (body of the vehicle) of the ego-object (road grade) comprises estimating rotation with respect to a ground surface based at least on applying low pass filtering (low frequency band pass filter 274) to a signal (signal from the common vehicle inclination sensor) representing a detected up-vector of the suspended mass to estimate orientation of the ground surface and applying high pass filtering to the signal to estimate orientation of the sprung mass (disposition of the vehicle 14) [Para. 13, 18, 45-47, and 48].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Wang’s vehicle dynamics estimator, modified by Weston, by incorporating Doering’s common body mounted vehicle inclination sensor and routing its signal through a low frequency filter to estimate road grade and through a high frequency filter to determine dynamic roll and then using the separated components to estimate vehicle body rotation.
Claims 7 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over WANG et al. (Pub. No. US 2015/0332446) in view of Weston et al. (Pub. No. US 20230410528) further in view of Zhang et al. (Pub. No. US 2019/0277632).
Regarding claims 7 and 18, Wang in view of Weston does not explicitly teach the claim limitations.
However, Zhang teaches the generating of the representation of the estimated rotation of the suspended mass of the ego-object comprises applying structure-from-motion to triangulate and track positions of observed objects on a ground surface [Para. 75, 76, 143, 144, and 145].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Wang’s vehicle dynamics estimator by incorporating Zhang’s structure from motion process to obtain estimated rotation by triangulation and tracked feature correspondence on a ground surface and to provide the resulting 3D motion transformation to Wang’s camera calibration correction; The modification improves Wang, thereby providing an image based rotation estimate when suspension are unavailable or unreliable.
Claims 8 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over WANG et al. (Pub. No. US 2015/0332446) in view of Weston et al. (Pub. No. US 20230410528) further in view of Loy et al. (Pub. No. US 2018/0150976).
Regarding claims 8 and 19, Wang in view of Weston does not explicitly teach the claim limitations.
However, Loy teaches wherein the generating of the representation of the estimated rotation of the suspended mass of the ego-object comprises using a deep neural network to predict the representation of the estimated rotation based at least on an input representation of a ground surface [Para. 4, 17, 35, and 37].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Wang’s vehicle dynamics estimator by incorporating Loy structure from motion process to obtain estimated rotation by triangulation and tracked feature correspondence on a ground surface and to provide the resulting 3D motion transformation to Wang’s camera calibration correction; The modification improves Wang, thereby providing an image based rotation estimate when suspension are unavailable or unreliable.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over WANG et al. (Pub. No. US 2015/0332446) in view of Weston et al. (Pub. No. US 20230410528) further in view of Ryu et al. (Pub. No. US 2007/0239320).
Regarding claim 9, Wang in view of Weston does not explicitly teach the claim limitations.
However, Ryu teaches wherein the generating (estimating) of the representation (roll angle) of the estimated rotation (roll angle) of the suspended mass of the ego-object comprises selecting (calculates the roll rate one way if none of the vehicle wheels are off of the ground and calculates it another way if any of the wheels are off of the ground) a rotation estimation technique (roll rate estimator) from a plurality of supported rotation estimation techniques [Para. 7].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Wang’s vehicle dynamics estimator by incorporating Ryu’s rotation estimation technique which uses one roll-rate calculation when no wheel is off the ground and another roll-rate calculation when a wheel is off the ground. This medication improves Wang by selecting a rotation estimation calculation appropriate to the detected wheel contact condition, thereby reducing inaccurate roll estimates during wheel lift.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Rathi et al. (Pub. No. US 20190143896) in view of Ryu et al. (Pub. No. US 2007/0239320).
Regarding claim 16, Rathi teaches system comprising: one or more processing units (processor 18) to estimate (determine) at least one of rotation information (determined ratio that the vehicle is rolled or pitched (skewed)) or translation information corresponding to an ego-object (vehicle) in an environment (scene surrounding the vehicle) based on ego-motion (skewing or rolling of the vehicle) of the ego-object (object) [Para. 35 “in FIG. 8, objects 30 are shown symbolized by boxes passing the scene surrounding the vehicle when the vehicle is driving forward.”; Para. 37 “Responsive to determination of such skewing or rolling of the vehicle, the system may alter the virtual ground plane in a way that a virtual top view as like in FIG. 10 results.”; and para. 40 “As an additional or alternative way to detect that the vehicle is not fully leveled horizontally (rolled in x axis) and/or has a pitch angle (in y axis) or the like, the system may utilize the compression ratio of the shock absorber system or suspension system of all wheels shock absorbers of the vehicle. Nowadays vehicles usually have sensors to detect the compression rate and usually this signal is provided via the vehicle's CAN or vehicle bus network, which makes it possible that the vision system is able to process this information. By such processing of suspension information (for the individual wheels of the vehicle) the system may determine uneven suspension of the corner regions of the vehicle and may correct the virtual (assumed) ground plane for the virtual top view vision projection in accordance with the determined ratio that the vehicle is rolled or pitched (skewed).”]; to generate an ego-motion (skewing or rolling of the vehicle) compensated (correct) projection (virtual top view vision projection) of frames of image data representing (representative of) two or more overlapping views (overlapping regions in the adjacent camera field of view) of the environment (scene surrounding the vehicle) using a transformation (affine transforms or other suitable transforms) based at least on the estimated rotation information (determined ratio that the vehicle is rolled or pitched (skewed)) or the estimated translation information [Para. 28, 29, 35, 37, 40 and 41].
However, Rathi doesn’t explicitly teach sprung mass.
Ryu teaches estimating at least one of rotation information (roll angle) or translation information corresponding to a sprung mass (vehicle body 16 (sprung mass)) of an ego-object (vehicle 10) in an environment (ground) based on ego-motion (measured suspension deflection rates) of the ego-object [Para. 13 “Assuming suspension deflection rates are measured at springs or shock absorbers (dampers) of the vehicle, the vehicle roll rate is first calculated using measured suspension deflection rates”; Para. 14; Para. 19; para. 25 “Where I.sub.xx is the roll moment of inertia of the vehicle body 16 (sprung mass) with respect to the center of gravity, M.sub.s is the mass of the vehicle body 16,”; Para. 25; and Para. 26].
It would have been obvious to one of ordinary to one of ordinary skill in the art before the effective filing date to modify Rathi’s suspension information processing by incorporating Ryu’s model-based estimation of rotation information (roll angle) corresponding to the sprung mass (vehicle 16 (sprung mass)) of an ego-object (vehicle 10) in an environment (ground), based on ego-motion (measured suspension deflection rates) and lateral-acceleration input, and supplying that estimated rotation to Rathi’s virtual ground plane correction for the projection (virtual top view vision projection). This modification improves Rathi by providing a noise-resistant sprung-mass rotation estimate for its existing suspension responsive projection correction, thereby predictably reducing erroneous ground plane adjustment and resulting image stitching misalignment.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOLOMON G BEZUAYEHU whose telephone number is (571)270-7452. The examiner can normally be reached on Monday-Friday 10 AM-7 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’Neal Mistry can be reached on 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-0101 (IN USA OR CANADA) or 571-272-1000.
/SOLOMON G BEZUAYEHU/ Primary Examiner, Art Unit 2666
Prosecution Insights