7DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s amendments filed on 11 June 2026 have been entered. Claims 18, 19, 24 and 27 have been amended. Claim 26 has been cancelled. Claims 1-25 and 27-30 are still pending in this application, with claims 1, 12 and 24 being independent.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-6, 12-17, 23-25 and 27-30 is/are rejected under 35 U.S.C. 103 as being unpatentable over REN et al. (US 12346995 B2), referred herein as REN in view of Bichu et al. (US 20210082086 A1), referred herein as Bichu.
Regarding Claim 1, REN in view of Bichu teaches a method for adjusting a stitching seam for surround view stitching, comprising (REN [0010] relate to surround view or environment visualization, dynamic seam placement based on object saliency, dynamic seam placement based ego-object state, an adaptive 3D bowl that models the surrounding environment; [0016] The present systems and methods for surround view or environment visualization; [0162] The methods may also be embodied as computer-usable instructions stored on computer storage media):
receiving a first fisheye image from a first fisheye-lens camera and a second fisheye image from a second fisheye-lens camera (REN [0179] In FIG. 13, the input images 1310 are four fisheye images captured by fisheye cameras located at the front, left, rear and right sides of a vehicle body);
detecting a target object in the first fisheye image and the second fisheye image (REN [0065] At a high level, objects (e.g., salient objects) may be detected from sensor data (e.g., fisheye images) representing an environment surrounding an ego-object such as a vehicle; [0067] to identify detected objects and/or salient regions, object detection may be performed on multiple images of an environment to create multiple object and/or saliency masks);
projecting the first fisheye image, the second fisheye image, and the target object to a stitching image (REN [0011] The images may be aligned to create an aligned composite image or surface (e.g., a panorama, a 360° image, bowl shaped surface) with overlapping regions of image data, and a representation of the detected objects and/or salient regions (e.g., a saliency mask) may be generated and projected onto the aligned composite image or surface. Seams may be positioned in the overlapping regions to avoid or minimize crossing salient pixels represented in the projected masks, and the image data may be blended at the seams to create a stitched image or surface (e.g., a stitched panorama, stitched 360° image, stitched textured surface));
wherein the stitching image comprises the stitching seam and the target object appears on a seam region of the stitching image (REN [0166] The method 1000, at block B1008 includes updating the candidate position for the seam to an updated position based at least on an intersection of the seam at the candidate position with pixels of the detected objects in one or more of the two or more projected masks);
calculating an adjustment direction of the stitching seam according to a distance between the target object and the first REN[0165] The method 1000, at block B1006 includes determining a candidate position for a seam in an overlapping region of the two or more aligned image frame; [0206] The method 2100, at block B2102, includes determining a distance from an ego-object to a detected object in an environment. For example, with respect to FIG. 16, the 3D object detector 1620 may perform 3D object detection on sensor data captured by the ego-object (e.g., the input images 1610, corresponding RADAR or LiDAR data), and the radial distance mapper 1630 may compute distance(s) to the detected objects (e.g., the closest detected object in a direction corresponding to each angular increment); [0191] Based on the distances and directions between the ego-object 1640 and the detected objects 1645, the 3D bowl parameter controller 1650 may adapt the shape of a 3D bowl modeling the surrounding environment based on the distances and directions to the detected objects 1645); and
REN teaches the distances between the detected object and the ego-object, but does not explicitly teach between the first and second fisheye cameras. However, Bichu teaches distance between object and fisheye cameras (Bichu [0189] …in order to correctly align every point in the two images including points that capture objects at different distances from the camera/lens; [0169] projecting an image captured with a left fisheye camera/lens and the right sphere at 1530 of FIG. 15 obtained from projecting an image captured with a right fisheye camera/lens).
adjusting the stitching seam according to the adjustment direction (REN [0166] The method 1000, at block B1008 includes updating the candidate position for the seam to an updated position based at least on an intersection of the seam at the candidate position with pixels of the detected objects in one or more of the two or more projected masks).
Bichu discloses an image stitching technique including depth or disparity estimation, alignment, and blending processes. Bichu is analogous to the present patent application.
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified REN to incorporate the teachings of Bichu, and apply the distances between capture objects and fisheye cameras into the systems and methods for image stitching with dynamic seam placement based on ego-vehicle state for surround view visualization.
Doing so would provide an improved and optimized image calibration and rendering techniques that may both improve output image quality for viewing and reduce the image processing time, which may facilitate the real-time output of tiled composite panoramic images.
Regarding Claim 2, REN in view of Bichu teaches the method for adjusting the stitching seam as claimed in claim 1, and further teaches further comprising:
determining whether the stitching seam has reached a limit value (REN [0071] a state machine implementing a decision tree is used to determine whether to use a default seam placement or dynamic seam placement that avoids salient objects or regions, and to enable and disable dynamic seam placement based on speed of ego-motion, direction of ego-motion, proximity to salient objects, active viewport direction, driver gaze, and/or other factors);
wherein the limit value is associated with extrinsic parameters of the first fisheye-lens camera and the second fisheye-lens camera (REN [0118] each camera may have various intrinsic or extrinsic values that can impact the appearance of a captured image, where those values may relate to field of view, optical center, focal length, or camera pose, among other such options; [0133] a seam may be placed horizontally to present a better (less disrupted) forward facing view to the driver. However, when driving at low speeds and/or whenever an object (e.g., another vehicle) passes by closely (e.g., within a threshold distance), the driver may need to pay closer attention to the object passing by). The threshold distance is calculated from extrinsic parameters of a camera.
Regarding Claim 3, REN in view of Bichu teaches the method for adjusting the stitching seam as claimed in claim 2, and further teaches further comprising:
determining that the stitching seam has reached the limit value; not adjusting the stitching seam (REN [0142] If the distance to the closest object is more than the threshold proximity, the dynamic seam toggling state machine 601 disables dynamic seam placement at block 625 and uses a default seam placement instead (e.g., horizontal or other default values, such as one that minimizes seam length visible in a particular viewport)); and
continuously receiving a third fisheye image from the first fisheye-lens camera and a fourth fisheye image from the second fisheye-lens camera (REN [0142] Returning to block 610, when ego-speed is within some medium speed range (e.g., from 5-16 km/hr), the dynamic seam toggling state machine 601 disables dynamic seam placement in favor of a default seam (e.g., a horizontal seam); 0143] Returning to block 615, distances to surrounding objects may be obtained or determined in various ways. In some embodiments, 3D object detection is performed (e.g., by processing sensor data) and/or a representation of detected 3D objects (e.g., 3D cuboids in rig coordinates) is accessed. For example, distance to objects may be computed using depth or stereo camera arrays).
Regarding Claim 4, REN in view of Bichu teaches the method for adjusting the stitching seam as claimed in claim 2, and further teaches further comprising:
determining that the stitching seam has not reached the limit value (REN [0142] If the distance to the closest object is less than some threshold proximity (e.g., less than 3 m), the dynamic seam toggling state machine 601 enables dynamic seam placement at block 620.); and
adjusting the stitching seam according to the adjustment direction (REN [0167] detected objects that are less than some length in pixels such as 500 pixels), a vertical or horizontal seam is selected when the entire mask projection being evaluated (the overlapping region) is occupied by a detected object (to avoid the object), and/or a seam is selected that crosses the least number of (remaining) object pixels).
Regarding Claim 5, REN in view of Bichu teaches the method for adjusting the stitching seam as claimed in claim 1, and further teaches further comprising:
retrieving a representative point of the target object according to a type of the target object (REN [0074] adapt the shape, orientation, and/or dimensions of a 3D bowl (e.g., a mesh) modeling the surrounding environment based on the distance to nearby detected objects and project image data onto the adapted 3D bowl. The present techniques may be utilized to visualize an environment around an ego-object, such as a vehicle, robot, and/or other type of object);
projecting the representative point to the stitching image (REN [0075] sensor data such as a LiDAR point cloud is projected onto a top-down 2D occupancy grid that represents locations of detected objects or a 3D occupancy grid that represents locations and projected or assumed heights of detected objects (e.g., assuming vehicles have a height of 2 or 3 meters above the ground surface).); and
calculating the adjustment direction of the stitching seam according to a distance between the representative point in the stitching image and the first fisheye-lens camera and a distance between the representative point in the stitching image and the second fisheye-lens camera (REN [0075] The 3D object detections and/or the 2D/3D occupancy grid may be used to compute distances to detected objects, and the distances may be populated in a radial distance map that represents distance to (e.g., a representative point(s) on) the closest detected object as a function of angle (e.g., representing a rotation around an axis of the vehicle coordinate system, such as yaw)).
Regarding Claim 6, REN in view of Bichu teaches the method for adjusting the stitching seam as claimed in claim 5, and further teaches wherein the step of calculating the adjustment direction of the stitching seam according to the distance between the representative point in the stitching image and the first fisheye-lens camera and the distance between the representative point in the stitching image and the second fisheye-lens camera, comprises:
calculating a first distance between the representative point and the first fisheye-lens camera; calculating a second distance between the representative point and the second fisheye-lens camera (Bichu [0189] …in order to correctly align every point in the two images including points that capture objects at different distances from the camera/lens; [0169] projecting an image captured with a left fisheye camera/lens and the right sphere at 1530 of FIG. 15 obtained from projecting an image captured with a right fisheye camera/lens); and
adjusting a slope of the stitching seam according to a relationship between the first distance and the second distance (Bichu [0188] misalignment in the lenses and cameras is determined by performing a 3D calibration on the cameras/lenses such that what remains is misalignment due to parallax. In particular, image points that correspond to scene points at different distances from the cameras/lenses must be adjusted by different shifts (e.g., shifting individual pixels in a first image with respect to a second image to align points at different distances by different amounts) in order to correctly align two images that are misaligned due to parallax. Accordingly, in order to correctly align points at different distances from a camera/lens, the depth of each point is detected using a depth estimation process).
Regarding Claims 12-17, REN in view of Bichu teaches a surround view stitching seam adjustment system, comprising (REN [0010] relate to surround view or environment visualization, dynamic seam placement based on object saliency, dynamic seam placement based ego-object state, an adaptive 3D bowl that models the surrounding environment; [0016] The present systems and methods for surround view or environment visualization; [0162] The methods may also be embodied as computer-usable instructions stored on computer storage media).
The metes and bounds of the claim substantially correspond to the claimed limitations set forth in claims 1-6; thus they are rejected on similar grounds and rationale as their corresponding limitations.
Regarding Claim 23, REN in view of Bichu teaches the surround view stitching seam adjustment system as claimed in claim 12, and further teaches wherein the first and second fisheye images are obtained from a simulation platform (REN [0235] FIG. 28 shows an example under vehicle reconstruction 2830 using simulated fisheye images 2810a-d).
Regarding Claims 24, 28 and 29, REN in view of Bichu teaches a method for adjusting a stitching seam for surround view stitching, comprising (REN [0010] relate to surround view or environment visualization, dynamic seam placement based on object saliency, dynamic seam placement based ego-object state, an adaptive 3D bowl that models the surrounding environment; [0016] The present systems and methods for surround view or environment visualization; [0162] The methods may also be embodied as computer-usable instructions stored on computer storage media).
The metes and bounds of the claim substantially correspond to the claimed limitations set forth in claims 1-3; thus they are rejected on similar grounds and rationale as their corresponding limitations.
Regarding Claim 25, REN in view of Bichu teaches the method of claim 24, and further teaches wherein the target object appears in a first-level warning frame or a second-level warning frame of the stitching image (REN [0288] Cameras with a field of view that include portions of the environment to the rear of the vehicle 3500 (e.g., rear-view cameras) may be used for park assistance, surround view, rear collision warnings, and creating and updating the occupancy grid).
Regarding Claim 27, REN in view of Bichu teaches the method of claim 24, and further teaches wherein the step of adjusting the adjustment direction further comprising the step of adjusting the stitching seam according to the adjustment direction comprises the steps of: increasing the slope of the stitching seam by a predetermined increment if the second distance is less than the first distance so that the weight of the second fisheye image in the stitching image is increased; and decreasing the slope of the stitching seam by the predetermined increment if the second distance is larger than the first distance so that the weight of the first fisheye image in the stitching image is increased (REN [0190] the representation of the distances and directions may take the form of a 2D array with a radial component (e.g., identifying an angular increment, such as 1 degree) and one or more distance components (representing the distance to the closest detected object in the direction represented by a corresponding angular increment, representing the distance to a particular corner or center of the closest detected object in that direction); [0196] By contrast, in some embodiments, an initial 3D bowl 1720 (e.g., a circular 3D bowl) is deformed using the 2D top-down occupancy grid 1710 to generate a deformed 3D bowl 1730. More specifically, for each angular increment (e.g., 1°), the distance to the closest detected object in that direction is determined using the 2D top-down occupancy grid 1710 (e.g., by computing and retrieving a corresponding value from a radial distance map that stores distance as a function of angle), and a radius is set for the deformed 3D bowl 1730 in a corresponding direction to equal the distance to the closest detected object in that direction). angular increment teaches the slope decreasing.
Regarding Claim 30, REN in view of Bichu teaches the method of claim 28, and further teaches wherein the limit value is associated with an imaging-range constrained by a physically installed position, rotation angle, and the field of view (FOV) of the first fisheye-lens camera or of the second fisheye-lens camera (REN [0190] In FIG. 16, the visualization 1635 illustrates one or more values that may be stored in an example radial distance map. The visualization 1635 depicts an ego-object 1640 (corresponding to the ego-object 1627), bounding boxes of detected objects 1645 (corresponding to a set of the detected objects 1625 within a threshold range of the ego-object 1627), and distances between the ego-object 1640 and the corners of the bounding boxes of the detected objects 1645).
Allowable Subject Matter
Claims 7-11 and 18-22 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Claims 7-11 and 18-22, REN in view of Bichu teaches the method/system of claim 1/12, but does not teach the limitation herein. Therefore, claims 7-11 and 18-22 in the context of claim 1 or 12 as a whole would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Response to Arguments
Applicant’s arguments, see page 12, filed on, with respect to claims 18 and 19 have been fully considered and are persuasive. The c of 16 March 2026 has been withdrawn.
Applicant’s arguments, see page 12, filed on 11 June 2026, with respect to claim 26 has been fully considered and are persuasive. The claim rejections under 112(d) rejection of 16 March 2026 has been withdrawn.
Applicant’s arguments, see page 12, filed on 11 June 2026, with respect to claims 24 and 27 have been fully considered and are persuasive. The claim rejections under 112(b) of 16 March 2026 has been withdrawn.
Applicant's arguments filed on 02 June 2026, with respect to the 103 rejection have been fully considered but they are not persuasive.
On page 16, Applicant's Remarks, with respect to claim 1, the applicant argues REN only relevantly discloses that REN can calculate the distance between the center point and the detected object, and can adjust the shape of the 3D bowl simulating the surrounding environment based on the distance between the center point and the detected object. Examiner respectfully disagrees with this argument. REN first disclosed the candidate seam position placement is based on speed of ego-motion, direction of ego-motion, proximity to salient objects, active viewport direction, driver gaze, and/or other factors (see [0072] [0165]). REN further disclosed the position is determined by a radial distance from an ego-object and real object (see [0206]), wherein the ego-object is an input image captured by a 3D object sensor. Therefore, the radial distance is calculated from the distance from the sensor to the real object. In 3D environment, a point, such as camera and/or real object is determined by radial distance, polar angle and azimuthal angle. Therefore, the candidate seam position can be calculated based on the known position of camera and real object. The adjusting direction from the default seam can subsequentially be calculated from the candidate position. Regarding the first argument, it is respectfully noted that REN teaches “calculating an adjustment direction of the stitching seam according to a distance between the target object and the first fisheye-lens camera and a distance between the target object and the second fisheye-lens camera in the stitching image”, as claimed.
On page 16, Applicant's Remarks, with respect to claim 1, the applicant argues Since REN does not disclose the feature "calculating an adjustment direction of the stitching seam, REN will not certainly not disclose the feature "adjusting the stitching seam.” Examiner respectfully disagrees with this argument. REN not only teaches the limitation of “calculating an adjustment direction” for the reason described above, but also explicitly disclosed steps of updating the candidate position (see [0166]). Regarding the second argument, it is respectfully noted that REN teaches “adjusting the stitching seam according to the adjustment direction”, as claimed.
On page 18 of Applicant’s Remarks, the Applicant argues that the independent claims 18 and 24 is not taught by the prior art for reasons similar to those discussed in regard to claim 1. Examiner respectfully disagrees with the argument for the reasons discussed above.
On page 18 of Applicant’s Remarks, the Applicant argues the dependent claims are not taught by the prior art, insomuch as they depend from claims that are not taught by the prior art. Examiner respectfully disagrees with the arguments, for the reasons discussed above.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Samantha (Yuehan) Wang whose telephone number is (571)270-5011. The examiner can normally be reached Monday-Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571)272-7440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Samantha (YUEHAN) WANG/
Primary Examiner
Art Unit 2617
Prosecution Insights