Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is the first office action on the merits and is responsive to the papers filed 08/05/2024. Claims 1-20 are currently pending and examined below.
Information Disclosure Statement
The information disclosure statement submitted by Applicant is in compliance with the provision of 37 CFR 1.97, 1.98 and MPEP § 609. It has been placed in the application file and the information referred to therein has been considered as to the merits.
Drawings
The drawings are objected to because in Fig. 13, step 1304, “STET” should be “the”.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 12085678 (Pat’ 678) in view of Wheeler et al. (US 20190122386 A1), Masahiro Harada (US 20180196127 A1), Rohatgi et al (US 20210048516 A1), Hickerson et al. (US 2008/0236568), Cai et al. (US 20210325520 A1, “Cai”), Payton et al. (US 2016/0368148 A1) and Muramatsu et al. (US 20190041523 A1, “Muramatsu”)..
Regarding Claim 1, a comparison of limitations is shown with reference to claims 1 and 10 of Pat’ 678. The limitations highlighted in claim 1 are taught by highlight limitations of claims 1 and 10 of Pat’ 678.
Instant, Application No. 18/794,511
U.S. Patent No. 12085678
Claim 1
Claims 1 and 10
A method comprising:
A method comprising:
generating a point cloud of a region based on data from a light detection and ranging (lidar) device, wherein the point cloud includes points representing at least a portion of a calibration target;
Claim 1: generating a point cloud of a region based on data from a light detection and ranging (lidar) device, wherein the point cloud includes points representing at least a portion of a calibration target;
determining a presumed location of the calibration target,
Claim 1: determining a presumed location of the calibration target;
wherein the presumed location comprises a geographic location of the calibration target;
Claim 10: The method of claim 1, wherein the presumed location comprises a geographic location of the calibration target, and wherein the geographic location is stored within a memory associated with the lidar device.
identifying, within the point cloud, a location of a first edge of the calibration target;
Claim 1: identifying, within the point cloud, a location of a first edge of the calibration target;
performing a comparison between the identified location of the first edge of the calibration target and a hypothetical location of the first edge of the calibration target within the point cloud if the calibration target were positioned at the presumed location;
Claim 1: performing a comparison between the identified location of the first edge of the calibration target and a hypothetical location of the first edge of the calibration target within the point cloud if the calibration target were positioned at the presumed location;
and revising the presumed location of the calibration target based on at least the comparison.
Claim 1: revising the presumed location of the calibration target based on at least the comparison;
Claim 1: and transmitting the revised presumed location of the calibration target to a fleet management server.
Regarding claim 1, claim 1 of the ’678 patent recites the corresponding lidar calibration-target localization method, and claim 10, which depends on claim 1, further recites that the presumed location comprises a geographic location of the calibration target. Thus, claims 1 and 10 of the ’678 patent render instant claim 1 not patentably distinct.
Regarding claims 2-9, respective claims 2-9 of the ’678 patent recite the corresponding additional limitations of instant claims 2-9. To the extent the reference claims do not explicitly recite the geographic-location limitation inherited from instant claim 1, Wheeler in view of Harada teaches determining and using a geographic location of the calibration target as discussed in the §103 rejection above. It would have been obvious to employ such geographic-location information to provide an initial target position and facilitate localization.
Regarding claim 10, claim 10 of the ’678 patent depends on claim 1 and explicitly recites that the presumed location comprises a geographic location of the calibration target stored within a memory associated with the lidar device. Thus, claims 1 and 10 of the ’678 patent render instant claim 10 not patentably distinct.
Regarding claim 11, claims 1 and 10 of the ’678 patent do not explicitly recite that the presumed location is based on a previously performed localization. Wheeler in view of Harada and Payton teaches this limitation, wherein Payton teaches using a previously recognized pose as an initial guess for subsequent localization. It would have been obvious to use the previous localization to reduce processing required for subsequent target localization.
Regarding claims 12 and 13, respective claims 12 and 13 of the ’678 patent recite the corresponding additional limitations of instant claims 12 and 13. Wheeler in view of Harada teaches the geographic-location limitation inherited from instant claim 1. It would have been obvious to employ the geographic location to provide an initial target position and facilitate localization.
Regarding claim 14, claim 1 of the ’678 patent recites transmitting the revised presumed location to a fleet management server, and claim 10, which depends on claim 1, further recites that the presumed location comprises a geographic location. Thus, claims 1 and 10 of the ’678 patent render instant claim 14 not patentably distinct.
Regarding claims 15 and 16, claims 14 and 15 of the ’678 patent recite the corresponding point-classification and edge-detection limitations of instant claims 15 and 16. Wheeler, as discussed above, in view of Harada teaches the geographic-location limitation inherited from instant claim 1. Therefore, instant claims 15 and 16 are not patentably distinct from claims 14 and 15 of the ’678 patent in view of Wheeler and Harada.
Regarding claim 17, claim 16 of the ’678 patent recites the corresponding limitations directed to moving the lidar device, acquiring an additional point cloud, and determining whether the calibration target is identifiable therein. Wheeler in view of Harada and Rohatgi teaches the remaining limitations as discussed in the §103 rejection above. Accordingly, instant claim 17 is not patentably distinct.
Regarding claim 18, claim 17 of the ’678 patent recites determining whether the point cloud includes at least a portion of the calibration target. Wheeler in view of Harada teaches the remaining limitations, including the geographic presumed location. Accordingly, instant claim 18 is not patentably distinct.
Regarding claim 19, claim 18 of the ’678 patent recites the corresponding non-transitory computer-readable medium for performing the calibration-target localization method. Wheeler, as the primary reference, in view of Harada teaches the geographic-location limitation not explicitly recited by the reference patent claim. It would have been obvious to employ the known geographic target location to facilitate target localization.
Regarding claim 20, claim 18 of the ’678 patent recites the corresponding non-transitory computer-readable medium. Wheeler in view of Harada further teaches that the presumed geographic location is stored within memory associated with the lidar device. It would have been obvious to store the presumed location for subsequent localization and calibration operations.
Accordingly, claims 1-20 are not patentably distinct from the claims of U.S. Patent No. 12,085,678 alone or in combination with the references identified above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 6, 8-10, 13, 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Wheeler et al. (US 20190122386 A1, “Wheeler”) in view of Masahiro Harada (US 20180196127 A1, “Harada”).
Regarding claim 1, Wheeler teaches a method (Claim 9) comprising:
generating a point cloud of a region based on data from a light detection and ranging (lidar) device, wherein the point cloud includes points representing at least a portion of a calibration target (Wheeler [0087]- [0089] teaches placing a checkerboard calibration pattern in front of the lidar and camera and capturing a lidar scan of the scene showing the checkerboard pattern. The lidar data are used for calibration. Wheeler further selects lidar points corresponding to the checkerboard, determines its 3-D plane geometry and corner coordinates, and classifies points near the plane/corners as belonging to the checkerboard. Fig. 14, steps 1430- 1470; [0106]- [0107]. See also, fig. 21);
determining a presumed location of the calibration target (In Wheeler's second-pass calibration, the rough lidar-to-camera transform is used to estimate where the checkerboard corners are in lidar coordinates. Wheeler then keeps lidar points within a small radius of that estimated location. [0103]- [0105], Fig. 13. See also, fig. 21),
identifying, within the point cloud, a location of a first edge of the calibration target (Wheeler, (Fig. 15, [0109]) states that the system identifies points at the checkerboard boundary as the first and last columns of scan-line segments. The system processes specific sides of that boundary and can process any two adjacent sides of the checkerboard. The identified checkerboard side/boundary corresponds to the claimed first edge of the calibration target. See also, Fig. 21 and Fig. 22, [0169]).
Wheeler fails to explicitly teach wherein the presumed location comprises a geographic location of the calibration target.
However, Harada teaches obtaining a geographic location of the calibration reference from a GPS of the reference vehicle. Harada [0032] discloses that the vehicle information includes the GPS location of the nearby vehicle, together with its model and yaw. Harada [0034] further explains that the vehicle information identifies the reference vehicle's position and orientation, including x, y, and z coordinates. More specifically, in Fig. 5, [0045]- [0046], Harada obtains the GPS locations of both vehicles, uses the GPS location of the reference vehicle to identify its x, y, z coordinates, and fits the model of the reference vehicle into the coordinate system according to the GPS coordinates and yaw information. Harada claim 4 recites obtaining “a location from a global positioning system (GPS) of the second vehicle.”.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Wheeler's calibration target localization technique to use Harada's GPS based target location and model based reference point cloud technique to establish the presumed location and expected geometry of Wheeler's calibration target because Wheeler recognizes that, when the checkerboard is farther from the lidar and surrounded by other objects, extracting the checkerboard is difficult “without any prior knowledge of where it is.” Wheeler [0095]. Harada provides such prior location and geometric information by using the reference object's GPS location, yaw, and known model to generate an accurate reference point cloud for comparison with the lidar-observed point cloud. Harada [0032]- [0038]. The modification would predictably facilitate identification and localization of Wheeler's calibration target, particularly at longer ranges, by providing an expected target position and geometry against which the measured target boundary can be compared and refined.
Wheeler, in view of Harada, teaches performing a comparison between the identified location of the first edge of the calibration target and a hypothetical location of the first edge of the calibration target within the point cloud if the calibration target were positioned at the presumed location (Wheeler teaches identifying boundary points of the checkerboard from the lidar point cloud and fitting the known checkerboard geometry from the identified boundary points ([0109]- [0114]; Fig. 15). Wheeler further varies the location of the checkerboard pattern, projects the lidar points onto the checkerboard pattern at the candidate location and determines an alignment score representing correspondence between the measured lidar data and the checkerboard geometry at that location ([0115]- [0121]; Fig. 16). Harada teaches using the known model, GPS location, and yaw of a calibration reference to generate a reference three-dimensional point cloud representing the expected position and geometry of the calibration reference and comparing the reference point cloud with the point cloud perceived by the lidar to determine positional differences ([0032]- [0038]; claims 4-5). Thus, in the combination, positioning Wheeler's known calibration-target geometry according to Harada's presumed geographic location and orientation establishes the hypothetical location of Wheeler's first target edge, which is compared with the corresponding edge actually identified in the lidar point cloud using Wheeler's edge matching comparison);
and revising the presumed location of the calibration target based on at least the comparison (Wheeler recognizes that the initially fitted checkerboard corners can contain error and states that the system uses lidar intensity information to “refine the location of checkerboard corners.” [0115]. Wheeler also teaches that the system begins with T1 and determines T2 by iteratively modifying the transform so that the distance between corresponding detected edges is reduced. [0171], Fig. 21.). Thus, after modifying Wheeler to use Harada's GPS-based reference representation to establish the presumed target position and expected target edge, it would have been predictable to use the measured versus expected edge discrepancy in Wheeler's existing refinement procedure to revise the presumed target location.
Regarding claim 2, Wheeler, in view of Harada, teaches the method of claim 1, wherein determining the presumed location of the calibration target comprises:
determining, in a coordinate system of the lidar device, a presumed three-dimensional position of a center of the calibration target (Wheeler uses an approximate lidar-to-camera transform to convert three-dimensional checkerboard locations into lidar coordinates, and selects lidar points within a threshold radius of an estimated checkerboard center (Wheeler, [0103]- [0105]; Fig. 13, steps 1340- 1360).);
and determining a presumed angular orientation of a planar surface of the calibration target relative to one or more scanning axes of the lidar (Wheeler fits a plane through the lidar points corresponding to the checkerboard, determines the checkerboard plane geometry in three dimensions, and fits the normal of the checkerboard plane (Wheeler, [0106]- [0108]; Figs. 14- 15). Wheeler further describes the checkerboard placement by its depth and orientation in a lidar/3-D coordinate system and determines the coordinates of the checkerboard from that placement ([0158]).
Regarding claim 6, Wheeler, in view of Harada, teaches the method of claim 1, further comprising: identifying, within the point cloud, a location of a second edge of the calibration target (Wheeler identifies checkerboard boundary points and processes two adjacent sides of the checkerboard; Wheeler further states that the procedure can be performed for any two adjacent sides of the checkerboard ([0109]- [0110]; Fig. 15).);
performing a further comparison between the identified location of the second edge of the calibration target and a hypothetical location of the second edge of the calibration target if the calibration target were positioned at the presumed location (Wheeler further teaches performing a further comparison involving the second edge, wherein its edgel-based calibration identifies corresponding pairs of edges and performs alignment based on the aggregate distance between one or more pairs of matching edges ([0169]-[0171]; Fig. 21). As discussed regarding claim 1, Harada's GPS location, yaw, and known-model information provide the reference position and geometry of the calibration target; thus, the same reference geometry establishes the expected locations of both Wheeler checkerboard edges when the target is positioned at the presumed location.); and
revising the presumed location of the calibration target based on at least the further comparison (Wheeler further teaches revising the presumed location based on the further comparison, wherein Wheeler varies the location of the checkerboard pattern and measures alignment with the underlying lidar data to refine the location of the checkerboard corners ([0115]-[0121]; Fig. 16)).
Regarding claim 8, Wheeler, in view of Harada, teaches the method of claim 1, wherein identifying the location of the first edge of the calibration target comprises distinguishing a surface of the calibration target from a background (Wheeler explains that at the checkerboard boundary, as the lidar scans from left to right, the laser transitions from far-away background points to nearby checkerboard points, thereby producing a distance/depth discontinuity at the checkerboard boundary ([0109]).), and wherein the surface of the calibration target is distinguishable from the background based on a threshold difference in reflectivity or distance between the surface of the calibration target and the background (Wheeler further identifies lidar edge points having a greater than a threshold change in depth and determines edges from those points ([0176]-[0178]; Fig. 22). Thus, Wheeler distinguishes the target surface from the background at the target boundary based on a threshold difference in distance.).
Regarding claim 9, Wheeler, in view of Harada, teaches the method of claim 1, wherein the calibration target comprises one or more fiducials located thereon (Wheeler's checkerboard pattern calibration target includes identifiable checkerboard corners that are used as reference points for calibration. Wheeler detects the two-dimensional checkerboard corners from camera images with subpixel accuracy and uses the corners as point correspondences for determining calibration geometry ([0090]- [0092])), and wherein the method further comprises:
capturing, using a camera associated with the lidar device, an image of the calibration target (Wheeler teaches a sensor arrangement including left and right cameras and a lidar viewing the checkerboard calibration target. Wheeler selects a frame and detects 2-D points representing checkerboard corners from the camera images ([0105]; Fig. 13).),
wherein revising the presumed location of the calibration target is further based on: positions of the one or more fiducials in the captured image (Wheeler detects the positions of the checkerboard corners in the camera images, triangulates the corresponding corner positions to obtain their three-dimensional locations in camera coordinates, and converts those corner positions into lidar coordinates. Wheeler then uses the converted locations to identify lidar points within a threshold radius of an estimated checkerboard center and fit the checkerboard plane ([0105]; Fig. 13). Wheeler subsequently refines the checkerboard geometry and obtains the final locations of the checkerboard corners in lidar coordinates ([0115]- [0122]; Fig. 16));
and a position and orientation of the camera relative to the lidar device (Wheeler applies an approximate lidar-to-camera transform when converting the camera derived checkerboard corner locations into lidar coordinates ([0105]). Wheeler further describes Tlidar2camera as a six-degree-of-freedom transform between the lidar and camera coordinate systems, thereby defining both the relative translational position and rotational orientation of the camera and lidar ([0170]).
Regarding claim 10, Wheeler, in view of Harada, teaches the method of claim 1, wherein the presumed location is stored within a memory associated with the lidar device (Wheeler determines the three-dimensional locations of the checkerboard corners and adds the 3-D corner locations to set H “for future reference” ([0133]; Fig. 17), thereby storing location information of the calibration target for subsequent use. Wheeler further teaches that the sensor-calibration modules may be stored and executed in the vehicle computing system associated with the vehicle sensors, including the lidar ([0086]; Fig. 9). Wheeler's disclosed computer system includes main memory 2404 and storage containing executable calibration instructions ([0192]- [0193]). Thus, Wheeler stores the calibration target location information for future reference in memory of the computing system associated with the lidar device.).
Regarding claim 13, Wheeler, in view of Harada, teaches the method of claim 1, further comprising storing, within a memory of the lidar device or within a memory of a fleet management server, the point cloud prior to identifying the location of the first edge of the calibration target.
Wheeler’s online HD map system 110 is implemented as a cloud-based/distributed computing system that interacts with a plurality of vehicles and receives sensor data collected by sensors of hundreds or thousands of vehicles ([0044]- [0047]), thereby corresponding to a fleet management server. Wheeler further teaches that sensor-calibration modules, including edgel-based calibration module 950, may be stored and executed in the online HD map system ([0086]). The edgel-based calibration module receives the lidar scan and thereafter determines edges from the 3-D lidar points ([0167]- [0169]; Fig. 21). Wheeler’s disclosed computing system includes main memory 2404 and storage for information used during processing ([0192]- [0193]). It would therefore have been obvious to store the received lidar point-cloud data in the memory of Wheeler’s online fleet system before processing those data to identify the calibration target edge.
Regarding claim 18, Wheeler, in view of Harada, teaches the method of claim 1, further comprising determining whether the point cloud includes the at least a portion of a calibration target (Wheeler determines whether the checkerboard can be located in the captured sensor data and, if the checkerboard cannot be located, skips the frame ([0101]). More specifically, Wheeler reads lidar points near the estimated checkerboard location, fits a dominant plane through those points, and accepts the frame only when the number of inliers exceeds a threshold; otherwise, another frame is selected ([0103]- [0105]; Fig. 13). Wheeler also states that the HD map system “detects the presence of the checkerboard pattern in sensor data” and determines its corner coordinates in the processed images and lidar scans ([0152]). Thus, Wheeler determines from the lidar point-cloud data whether points corresponding to at least a portion of the calibration target are present.).
Regarding claim 19, Wheeler teaches a non-transitory, computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a processor, result in a method being performed ([0198]- [0199]), the method comprising:
generating a point cloud of a region based on data from a light detection and ranging (lidar) device, wherein the point cloud includes points representing at least a portion of a calibration target (Wheeler [0087]- [0089] teaches placing a checkerboard calibration pattern in front of the lidar and camera and capturing a lidar scan of the scene showing the checkerboard pattern. The lidar data are used for calibration. Wheeler further selects lidar points corresponding to the checkerboard, determines its 3-D plane geometry and corner coordinates, and classifies points near the plane/corners as belonging to the checkerboard. Fig. 14, steps 1430- 1470; [0106]- [0107]. See also, fig. 21);
determining a presumed location of the calibration target (In Wheeler's second-pass calibration, the rough lidar-to-camera transform is used to estimate where the checkerboard corners are in lidar coordinates. Wheeler then keeps lidar points within a small radius of that estimated location. [0103]- [0105], Fig. 13. See also, fig. 21),
identifying, within the point cloud, a location of a first edge of the calibration target (Wheeler (Fig. 15, [0109]) states that the system identifies points at the checkerboard boundary as the first and last columns of scan-line segments. The system processes specific sides of that boundary and can process any two adjacent sides of the checkerboard. The identified checkerboard side/boundary corresponds to the claimed first edge of the calibration target. See also, Fig. 21 and Fig. 22, [0169]).
Wheeler fails to explicitly teach wherein the presumed location comprises a geographic location of the calibration target.
However, Harada teaches obtaining a geographic location of the calibration reference from a GPS of the reference vehicle. In particular, Harada [0032] discloses that the vehicle information includes the GPS location of the nearby vehicle, together with its model and yaw. Harada [0034] further explains that the vehicle information identifies the reference vehicle's position and orientation, including x, y, and z coordinates. More specifically, in Fig. 5, [0045]- [0046], Harada obtains the GPS locations of both vehicles, uses the GPS location of the reference vehicle to identify its x, y, z coordinates, and fits the model of the reference vehicle into the coordinate system according to the GPS coordinates and yaw information. Harada claim 4 recites obtaining “a location from a global positioning system (GPS) of the second vehicle.”.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Wheeler's calibration target localization technique to use Harada's GPS based target location and model based reference point cloud technique to establish the presumed location and expected geometry of Wheeler's calibration target because Wheeler recognizes that, when the checkerboard is farther from the lidar and surrounded by other objects, extracting the checkerboard is difficult “without any prior knowledge of where it is.” Wheeler [0095]. Harada provides such prior location and geometric information by using the reference object's GPS location, yaw, and known model to generate an accurate reference point cloud for comparison with the lidar-observed point cloud. Harada [0032]- [0038]. The modification would predictably facilitate identification and localization of Wheeler's calibration target, particularly at longer ranges, by providing an expected target position and geometry against which the measured target boundary can be compared and refined.
Wheeler, in view of Harada, teaches performing a comparison between the identified location of the first edge of the calibration target and a hypothetical location of the first edge of the calibration target within the point cloud if the calibration target were positioned at the presumed location (Wheeler teaches identifying boundary points of the checkerboard from the lidar point cloud and fitting the known checkerboard geometry from the identified boundary points ([0109]- [0114]; Fig. 15). Wheeler further varies the location of the checkerboard pattern, projects the lidar points onto the checkerboard pattern at the candidate location and determines an alignment score representing correspondence between the measured lidar data and the checkerboard geometry at that location ([0115]- [0121]; Fig. 16). Harada teaches using the known model, GPS location, and yaw of a calibration reference to generate a reference three-dimensional point cloud representing the expected position and geometry of the calibration reference and comparing the reference point cloud with the point cloud perceived by the lidar to determine positional differences ([0032]- [0038]; claims 4-5). Thus, in the combination, positioning Wheeler's known calibration-target geometry according to Harada's presumed geographic location and orientation establishes the hypothetical location of Wheeler's first target edge, which is compared with the corresponding edge actually identified in the lidar point cloud using Wheeler's edge matching comparison.);
and revising the presumed location of the calibration target based on at least the comparison (Wheeler recognizes that the initially fitted checkerboard corners can contain error and states that the system uses lidar intensity information to “refine the location of checkerboard corners.” [0115]. Wheeler also teaches that the system begins with T1 and determines T2 by iteratively modifying the transform so that the distance between corresponding detected edges is reduced. [0171], Fig. 21.). Thus, after modifying Wheeler to use Harada's GPS-based reference representation to establish the presumed target position and expected target edge, it would have been predictable to use the measured versus expected edge discrepancy in Wheeler's existing refinement procedure to revise the presumed target location.
Regarding claim 20, Wheeler, in view of Harada, teaches the non-transitory, computer-readable medium of claim 19, wherein the presumed location is stored within a memory associated with the lidar device (Wheeler teaches that the sensor calibration module determines the three-dimensional locations of the checkerboard corners and adds those locations to set H for future reference ([0133]; Fig. 17). Wheeler further teaches that its calibration modules may reside in the vehicle computing system associated with the lidar ([0086]) and that the computer system includes main memory and machine-readable storage for the calibration instructions and data ([0192]–[0193]). Thus, Wheeler teaches storing calibration-target location information for subsequent use in memory of the computing system associated with the lidar device.).
Claims 3, 7, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Wheeler in view of Harada and Rohatgi et al (US 20210048516 A1, “Rohatgi”).
Regarding claim 3, Wheeler, in view of Harada, fails to explicitly teach the method of claim 2, wherein revising the presumed location of the calibration target comprises: determining, in the coordinate system of the lidar device, a revised three-dimensional position of the center of the calibration target based on the comparison and a shape and size of the calibration target; and determining a revised angular orientation of the planar surface of the calibration target relative to the one or more scanning axes using points within the point cloud that are determined to have been reflected from the calibration target.
However, Rohatgi teaches identifying lidar points corresponding to a fiducial target to form a target cloud and fitting that target cloud to a geometric representation of a fiducial target of known size and shape. Rohatgi performs the fitting using rigid body operations in six degrees of freedom, including translations in the x, y, z directions and roll, pitch, and yaw rotations, with the operations optimized to minimize the distances between the measured target-cloud points and the known fiducial target geometry ([0039], [0041], [0048]- [0051]).
Thus, determining, in the coordinate system of the lidar device, a revised three-dimensional position of the center of the calibration target based on the comparison and a shape and size of the calibration target is taught by Rohatgi's translational fitting of the lidar-derived target cloud to the known size and shape target model; the fitted translation establishes the revised three-dimensional target position. Likewise, determining a revised angular orientation of the planar surface relative to the scanning axes using points determined to have been reflected from the target is taught by Rohatgi's use of the lidar target cloud points and rotational components roll, pitch, and yaw of the six degree of freedom rigid body fit. Rohatgi selects from the lidar point cloud the points corresponding to the fiducial target before performing the fit.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the calibration target localization method of Wheeler to employ Rohatgi's six-degree-of-freedom fitting of lidar-derived target points to a known target geometry because Rohatgi teaches that the fitting determines the translational and rotational pose of the known target while minimizing differences between the measured target cloud points and the known target geometry. Applying that fitting to Wheeler's calibration target points would predictably provide a refined three-dimensional target position and angular orientation based on the measured lidar points and the known shape and size of the calibration target.
Regarding claim 7, Wheeler, in view of Harada, fails to explicitly teach but Rohatgi teaches the method of claim 1, further comprising:
moving the lidar device relative to the calibration target (Rohatgi teaches a fiducial target held in position on a stand and a lidar sensor mounted on a rotatable platform. Rohatgi teaches rotating the platform and lidar sensor from a first position and/or angle to a second position and/or angle relative to the fiducial target ([0043]- [0046]; Figs. 2–3));
generating, using the lidar device, an additional point cloud of an additional region that includes at least a portion of the calibration target (Rohatgi teaches obtaining lidar sensor data at the first position, then rotating the lidar to the second position and obtaining one or more additional sets of lidar sensor data. Rohatgi further teaches that the lidar sensor data are converted into or represented by a 3-D point cloud and that the point cloud includes points corresponding to the fiducial target ([0043]- [0047])).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the Wheeler calibration target localization method to move the lidar relative to the calibration target and acquire an additional lidar point cloud, as taught by Rohatgi, because Rohatgi recognizes that the lidar field of view may be sufficiently large that not all lidar beams initially intersect the calibration target and therefore teaches rotating the lidar through multiple positions so that additional lidar data corresponding to the target are obtained. Applying Wheeler's existing edge identification and target location refinement process to the additional target observation would predictably provide an additional independent measurement of the calibration target edge with which to further refine the target location.
Wheeler, in view of Harada and Rohatgi, teaches identifying, within the additional point cloud, an additional location of the first edge of the calibration target (Wheeler teaches identifying points at a calibration-target boundary as first/last columns of lidar scan-line segments, splitting the boundary into sides, and fitting the checkerboard geometry from those boundary points ([0107]- [0114]; Fig. 15). Wheeler additionally teaches determining edges directly from a lidar scan based on depth and intensity discontinuities ([0167]- [0169]). Accordingly, applying Wheeler's disclosed edge identification process to Rohatgi's additional lidar point cloud identifies an additional location of the same calibration target edge from the second lidar observation.);
performing an additional comparison between the identified additional location of the first edge of the calibration target and a hypothetical location of the first edge of the calibration target within the additional point cloud if the calibration target were positioned at the revised presumed location (as discussed regarding claim 1, Wheeler identifies the measured target edge and performs alignment of measured lidar geometry with expected target geometry, while Harada supplies the known target/reference geometry positioned according to the presumed geographic location. After the first Wheeler-Harada comparison produces the revised presumed location, that revised location provides the reference position for determining where the corresponding target edge is expected to occur in Rohatgi's additional point cloud. Wheeler's edge/alignment technique is then applied to compare the newly measured edge with that expected edge location.);
and further revising the presumed location of the calibration target based on the additional comparison (Wheeler teaches refining the calibration-target location by varying the checkerboard position and measuring alignment with the lidar data, retaining the better alignment to obtain the refined/final checkerboard location ([0115]- [0121]; Fig. 16). Wheeler also teaches iterative edge-based refinement in which alignment is modified to reduce the difference between corresponding edges. Thus, repeating Wheeler's disclosed refinement using Rohatgi's additional lidar observation results in further revising the presumed target location based on the additional comparison.).
Regarding claim 17, Wheeler, in view of Harada, fails to explicitly teach but Rohatgi teaches the method of claim 1, further comprising:
moving the lidar device relative to the calibration target (Rohatgi teaches a fiducial target 210 held in position on a stand while lidar sensor 215 is mounted on rotatable platform 220. Rohatgi explains that, because not all lidar lasers may initially strike the fiducial target, the lidar sensor is rotated a number of times so that the lasers obtain lidar data corresponding to the target ([0039], [0042]- [0044]; Figs. 2-3). More specifically, after obtaining data at a first position/angle, the platform and lidar are rotated to a second position/angle, where additional lidar data are obtained. Thus, Rohatgi moves the lidar relative to the stationary calibration target.);
capturing an additional point cloud of an additional region that includes at least a portion of the calibration target (after the lidar is rotated from the first position/angle to the second position/angle, Rohatgi obtains “one or more additional sets of lidar sensor data” and teaches that the acquired lidar data are converted into or represented by a 3-D point cloud. The acquisition continues over different positions/angles until lidar data corresponding to the fiducial target are obtained. [0044]- [0047]. Rohatgi further states that a point cloud may be generated from the lidar data and that the point cloud includes at least some points corresponding to the fiducial target. Accordingly, the lidar's changed position/angle provides an additional observation region and an additional point-cloud acquisition containing at least a portion of the target.); and
determining whether the calibration target is identifiable within the additional point cloud (Rohatgi teaches that the fiducial target is configured so that it can be “identified based on lidar sensor data” and detected by the lidar sensor. [0039]. Rohatgi then processes the point cloud by selecting the points corresponding to the fiducial target to generate target cloud 410; after the target cloud is identified, it is fitted to the known geometric representation of the fiducial target. [0041], [0047]- [0048]; Fig. 10. Thus, Rohatgi determines from the additionally acquired lidar point-cloud data whether points corresponding to the calibration target can be identified.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the Wheeler method to move the lidar relative to the calibration target, capture an additional point cloud, and determine whether the target is identifiable therein, as taught by Rohatgi, because Rohatgi recognizes that, due to the lidar's field of view, not all lidar beams may initially intersect the calibration target. Rohatgi therefore teaches rotating the lidar to additional positions/angles until lidar data corresponding to the target are obtained. Applying this technique to Wheeler would predictably provide additional target observations and permit confirmation that the calibration target is detectable from the changed lidar position, thereby improving target coverage and reliability of the calibration process.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Wheeler in view of Harada, Rohatgi and Hickerson et al. (US 2008/0236568, “Hickerson”).
Regarding claim 4, Wheeler, in view of Harada and Rohatgi, fails to explicitly teach the method of claim 3, wherein the revised angular orientation of the planar surface of the calibration target relative to the one or more scanning axes using points within the point cloud that are deemed to have been reflected from the calibration target is determined via regression using singular value decomposition.
However, Hickerson [0025] teaches a method of calibrating orientation and position of an imager with respect to a mirror using a laser and retro-reflector. The location of spots created by light reflected back to the imager are used as calibration point, and SVD algorithm may be used to improve the accuracy of the orientation estimate.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to apply the SVD algorithm taught by Hickerson to the orientation estimate obtained from the point cloud that are deemed to have been reflected from the calibration target in the method of Wheeler.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Wheeler in view of Harada and Cai et al. (US 20210325520 A1, “Cai”).
Regarding claim 5, Wheeler, in view of Harada, fails to explicitly teach the method of claim 1, wherein the calibration target is rectangular, wherein the calibration target has a planar surface, and wherein the planar surface has approximately uniform Lambertian reflectivity across the planar surface.
Wheeler uses a checkerboard calibration target having four outer corners and fits a plane through lidar points corresponding to the checkerboard to determine the checkerboard plane geometry in three dimensions ([0106]- [0109]; Fig. 14; See also, Figs. 10 and 18). Wheeler further defines checkerboard coordinates having an x-axis extending along a short side and a y-axis extending along a long side, consistent with a rectangular checkerboard target ([0118]).
Wheeler does not explicitly teach that the planar surface has approximately uniform Lambertian reflectivity across the planar surface. Cai teaches calibration of a LiDAR device using highly Lambertian reflective surfaces. Cai [0079] teaches that the calibration target surface may be “highly Lambertian,” functioning as a nearly ideal diffuse reflector, and that the reflectance value is substantially uniform across the entire surface. Cai further teaches using Zenith Polymer® diffusers having nearly ideal Lambertian properties and a common nominal diffuse reflectance, subject to only a ±3% manufacturing tolerance. See also, [0077]- [0078]
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the planar calibration target of Wheeler to provide the target surface with the substantially uniform Lambertian reflective characteristics taught by Cai because Cai teaches such surfaces for LiDAR calibration as nearly ideal diffuse reflectors having substantially uniform reflectance across the target surface. Using such a surface in Wheeler would have predictably provided more uniform lidar returns across the planar calibration target and reduced variation in measured reflectance caused by the target surface itself, thereby improving the consistency and reliability of lidar calibration measurements.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Wheeler in view of Harada and Payton et al. (US 2016/0368148 A1, “Payton”).
Regarding claim 11, Wheeler, in view of Harada, fails to explicitly teach the method of claim 1, wherein the presumed location is based on a localization that was previously performed by the lidar device or by a different lidar device.
However, Payton [0033] teaches determining the pose of a known object from a 3-D point cloud and a known model using an iterative closest point (ICP) pose-registration process. Payton further teaches using information obtained from a previous localization result to initialize a subsequent localization, specifically by using a “previously recognized pose as an initial guess.”.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the Wheeler method to use a location determined during a previous localization as the presumed location for a subsequent localization, as taught by Payton, because Payton teaches using a previously recognized pose as an initial guess to speed the subsequent point-cloud pose registration process. Payton further teaches reusing previous information when the scene is substantially unchanged. Applying this technique to Wheeler would predictably reduce the amount of searching and processing required to relocate the calibration target during a subsequent lidar localization, since the previously determined target location provides a closer starting estimate than beginning the localization without prior target location information.
Claims 12, 14 are rejected under 35 U.S.C. 103 as being unpatentable over Wheeler in view of Harada and Muramatsu et al. (US 20190041523 A1, “Muramatsu”).
Regarding claim 12, Wheeler, in view of Harada, fails to explicitly teach the method of claim 1, wherein the presumed location is transmitted to a computing device associated with the lidar device from a fleet management server or transmitted to the computing device associated with the lidar device from a computing device associated with a different lidar device.
However, Muramatsu teaches an advanced-map system including vehicle-mounted device 1, LIDAR 2, and centralized server device 4 storing advanced map database 43. Vehicle-mounted device 1 controls LIDAR 2 and communicates with server device 4. Server device 4 centrally stores landmark information including position information for landmarks detectable by the LIDAR and provides applicable landmark information to the vehicle-mounted device. Thus, server device 4 performs the claimed fleet-management-server function of centrally maintaining and distributing target-location information for use by vehicle-mounted lidar systems.
More particularly, in Fig. 5 and [0062]- [0064], the vehicle transmits its position to server device 4, the server identifies a target landmark near the vehicle and retrieves the corresponding landmark position information, and the server transmits response information D2 including the landmark position information to vehicle-mounted device 1. Vehicle-mounted device 1 receives the information and uses it in controlling LIDAR 2 and detecting the target landmark. Accordingly, Muramatsu teaches transmission of presumed target-location information from a centralized fleet-management server to a computing device associated with the lidar device.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Wheeler's localization method to obtain the presumed geographic location from Muramatsu's centralized server because Wheeler recognizes that identification of a distant calibration target is more difficult without prior knowledge of the target location. Muramatsu's server-provided landmark location predictably supplies such prior location information, reducing the region that the lidar system must search and facilitating reliable target detection.
Regarding claim 14, Wheeler, in view of Harada, fails to explicitly teach the method of claim 1, further comprising transmitting the revised presumed location of the calibration target to a fleet management server.
However, Muramatsu teaches centralized server device 4 storing advanced map database 43 containing landmark position information. After receiving the stored landmark position from the server, vehicle-mounted device 1 processes the LIDAR output, extracts point group information corresponding to the target landmark, determines the relative position of the target landmark, and estimates an absolute position of the landmark ([0065]). Muramatsu then teaches that vehicle-mounted device 1 sends server device 4 the newly estimated target landmark position as update information for the advanced map database ([0065], step S106). The server receives the updated position and updates the centrally maintained database ([0066], step S204). Thus, Muramatsu teaches transmitting a revised lidar derived target location from the vehicle computing device to the centralized server.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Wheeler's method to transmit the revised calibration target location to the centralized server, as taught by Muramatsu, because doing so permits the server to maintain an updated target position derived from lidar measurements and to make that updated position available for subsequent localization operations.
Allowable Subject Matter
Claims 15 and 16 are objected to as being dependent upon a rejected base claim but contain allowable subject matter with respect to the prior art rejections under 35 U.S.C. § 103. Claims 15 and 16, however, remain subject to the nonstatutory double-patenting rejection set forth above. If the nonstatutory double patenting rejection is overcome, claims 15 and 16 would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 15, the prior art of record, either individually or in combination, fails to teach or suggest, together with all other limitations of the claim, assigning each point within the point cloud to the recited first, second, and third categories based on the claimed angle of incidence, distance, and reflectivity criteria for disregarding points and selecting points for edge detection.
Regarding claim 16, the prior art of record, either individually or in combination, further fails to teach or suggest, together with all other limitations of the claim, performing edge detection between points assigned to the claimed second category and points assigned to the claimed third category.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Yu et al. (US 20210190922 A1), teaches automatic autonomous vehicle and robot lidar-camera extrinsic calibration
Koichi Tsugimura (US 20150281519 A1), teaches Image processing apparatus configured to execute correction on scanned image
Sang jun Park (US 9704404 B2), teaches Lane Detection Apparatus and Operating Method for The Same
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEMPSON NOEL whose telephone number is (571) 272-3376. The examiner can normally be reached on Monday-Friday 9:30-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Yuqing Xiao can be reached on (571) 270-3603. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JEMPSON NOEL/Examiner, Art Unit 3645
Prosecution Insights