Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
Claim 6 recites “a module M1 configured to extract a ground point cloud, and compare two consecutive frames of point clouds…”; “a module M2 configured to estimate a linear velocity and an angular velocity…”; “a module M3 configured to estimate and construct a pose graph using a relative pose transformation…” The corresponding structure in the disclosure for performing the claimed (“the system and the apparatus as well as various modules thereof provided by the present invention can certainly be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers”).
Claim 7 recites “wherein in the module M1, the ground point cloud is extracted, and the two consecutive frames of point clouds are compared…”; “wherein a module M1.1 is configured to perform the following operation: ghost point removal…”; “and a module M1.2 is configured to perform the following operation: random point removal” The corresponding structure in the disclosure for performing the claimed (“the system and the apparatus as well as various modules thereof provided by the present invention can certainly be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers”).
Claim 8 recites “wherein in the module M2, a module M2.1 is configured to perform the following operation: static point extraction…”; “a module M2.2 is configured to perform the following operation: least squares estimation” The corresponding structure in the disclosure for performing the claimed (“the system and the apparatus as well as various modules thereof provided by the present invention can certainly be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers”).
Claim 9 recites “wherein in the module M3, pose graph optimization is performed…”; “wherein a module M3.1 is configured to perform the following operation: point cloud registration based on the normal distribution transformation…”; “a module M3.2 is configured to perform the following operation: velocity pre- integration…” The corresponding structure in the disclosure for performing the claimed (“the system and the apparatus as well as various modules thereof provided by the present invention can certainly be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers”).
Claim 10 recites “wherein a module M3.3 is configured to perform the following operation: loop closure detection…”; “a module M3.4 is configured to perform the following operation: graph optimization…” The corresponding structure in the disclosure for performing the claimed (“the system and the apparatus as well as various modules thereof provided by the present invention can certainly be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers”).
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3, and 8, are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 3 and 8 have a matrix equation where the dimensions of the vectors and matrices create ambiguity in how the equation is supposed to be implemented. The Examiner understand the [v ω] vector as a 6 x 1 vector but it is being applied to a 3 x 3 matrix which is not mathematically possible. The dimensions of the RsT pose matrix is unclear and if assumed to be 4 x 4 again it is unclear how it is supposed to interact with a 6 x 1 or 3 x 1 vector. Finally, the unit vector has m elements and again unclear how it is supposed to interact with a 4 x 1 vector.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 6, and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Ali (US 20230126333 A1) in view of Qu (2019) [Qu Z, Wei X-M, Chen S-Q (2019) An algorithm of image mosaic based on binary tree and eliminating distortion error. PLoS ONE 14(1): e0210354. https://doi.org/10.1371/journal.pone.0210354] further in view of Yang (2023) [Yang L, Ma H, Nie Z, Zhang H, Wang Z, Wang C. 3D LiDAR Point Cloud Registration Based on IMU Preintegration in Coal Mine Roadways. Sensors (Basel). 2023 Mar 26;23(7):3473. doi: 10.3390/s23073473. PMID: 37050535; PMCID: PMC10098805].
Regarding claim 1 Ali discloses
A pose graph SLAM computation method based on a 4D millimeter-wave radar, the method comprising (Paragraph 0033, "For example, in another embodiment, the scan aggregator and filter 42 and the scan matching and radar pose estimator module 46 may be used for three-dimensional radar based simultaneous localization and mapping (SLAM) applications"; Abstract, "The automated driving controller determines a pose graph based on the most recent hyper-local submap and neighboring radar point cloud scans"; Paragraph 0015, "converting the relative transform between the last two consecutive pose estimate into velocity based on a time difference between the last two consecutive pose estimates"): comparing two consecutive frames of point clouds (Paragraph 0015, "converting the relative transform between the last two consecutive pose estimate into velocity based on a time difference between the last two consecutive pose estimates") step S2: estimating a linear velocity and an angular velocity of a device based on Doppler velocity information in a 4D radar point cloud (Paragraph 0003, “A millimeter wave radar may measure the range, angle, and Doppler (radial velocity) of moving objects.”; Paragraph 0015, "converting the relative transform between the last two consecutive pose estimate into velocity based on a time difference between the last two consecutive pose estimates" where a linear and angular velocity can be calculated; Paragraph 0042, " It is to be appreciated that the maximum distance threshold is adjusted based on both the linear and angular velocity of the autonomous vehicle 10 (FIG. 1). Finally, the ICP scan matching sub-module 60 determines the initial estimated pose 80, which is used as a starting point to determine the final radar poses 54"); and loop closure detection (Paragraph 0006, “In an aspect, the automated driving controller executes instructions to execute a loop detection algorithm to detect the autonomous vehicle is currently situated in a previously visited location”); and estimating an optimal pose through graph optimization (Paragraph 0007, “In another aspect, in response to detecting the loop, the automated driving controller is instructed to execute a non-linear optimization routine to perform a global pose adjustment to a loop-closure pose graph to determine a loop-adjusted radar pose, and set the loop-adjusted radar pose as the final radar pose”). Ali does not disclose step S1: extracting a ground point cloud, and comparing two consecutive frames of point clouds, to remove ghost points below the ground and random points; and step S3: estimating and constructing a pose graph using a relative pose transformation that is obtained through point cloud registration performed based on a normal distribution transformation, pre-integration performed using an estimated device velocity.
Qu discloses
Step S1: extracting a ground point cloud, and comparing two consecutive frames of point clouds, to remove ghost points below the ground and random points (Section 2.2 First paragraph, "After extracting the A-KAZE feature points, two KD-trees are constructed for the reference image and the target image respectively. The next step is taking one of them in turn as the reference for KNN (K Nearest Neighbor) matching, then extract the public matching pairs of matching operations as the initial matching. Finally, the RANSAC algorithm is adopted to remove the outer points and estimate the affine transformation matrix between images" where KAZE is comparing different images as a process to remove outlier points which include at the ground and below, and Ali can use two consecutive frames where one acts as a reference).
Ali discloses ghost detections but not removing the ghost detections. Ali removing the ghost points between two consecutive frames would be advantageous for improving the calculation of the trajectory and removing false stationary objects, overall leading to better performance. As such, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Qu to add in ghost point removal to improve the trajectory calculation and accurate determination of stationary objects.
Yang discloses
Step S3: estimating and constructing a pose graph using a relative pose transformation that is obtained through point cloud registration performed based on a normal distribution transformation (Section 2.2, "The NDT algorithm is a typical distribution-based method that was first proposed in 2D LiDAR point cloud registration [7]. The LiDAR point cloud is represented by a group of Gaussian distributions with different probability density functions. To avoid the problem of incorrect correlation between data, the method gives the segmented smooth normal distribution representation of laser scanning data"), pre-integration performed using an estimated device velocity, (Abstract, "Therefore, for these environments, the traditional point cloud registration method to register directly will lead to problems, such as a decline in registration accuracy, z-axis drift, and map ghosting. To solve the above problems, we propose a point cloud registration method based on IMU preintegration with the sensor characteristics of LiDAR and IMU"; Introduction Paragraph 4, "we propose a point cloud registration method with ground segmentation and IMU preintegration based on the motion characteristics of the ground mobile robot").
Ali discloses a relative transform but not a Normal Distribution Transform (NDT). The advantages of using a NDT include computational efficiency, as the calculation is using probability densities instead of individual points, and outliers (or ghost points) are less likely to affect the calculation. As such, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Yang so that the transformation between points can be more efficient and more resistant to outliers.
Ali discloses the use of closed loop recognition but not the use of pre-integration. Using pre-integration would be useful for effective tracking between frames when in a very dynamic situation, and for keeping drift between frames low. As such, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Yang to include pre-integration to improve the tracking and reduce drift.
Regarding claim 6 Ali discloses
A pose graph SLAM computation system based on a 4D millimeter-wave radar, comprising (Paragraph 0033, "For example, in another embodiment, the scan aggregator and filter 42 and the scan matching and radar pose estimator module 46 may be used for three-dimensional radar based simultaneous localization and mapping (SLAM) applications"; Abstract, "The automated driving controller determines a pose graph based on the most recent hyper-local submap and neighboring radar point cloud scans"; Paragraph 0015, "converting the relative transform between the last two consecutive pose estimate into velocity based on a time difference between the last two consecutive pose estimates"): comparing two consecutive frames of point clouds (Paragraph 0015, "converting the relative transform between the last two consecutive pose estimate into velocity based on a time difference between the last two consecutive pose estimates"), a module M2 ("The automated driving controller determines an initial estimated pose by aligning a latest aggregated filtered data point cloud scan with the most recent hyper-local submap based on an iterative closest point (ICP) alignment algorithm") configured to estimate a linear velocity and an angular velocity of a device based on Doppler velocity information in a 4D radar point cloud (Paragraph 0003, “A millimeter wave radar may measure the range, angle, and Doppler (radial velocity) of moving objects.”; Paragraph 0015, "converting the relative transform between the last two consecutive pose estimate into velocity based on a time difference between the last two consecutive pose estimates" where a linear and angular velocity can be calculated; Paragraph 0042, " It is to be appreciated that the maximum distance threshold is adjusted based on both the linear and angular velocity of the autonomous vehicle 10 (FIG. 1). Finally, the ICP scan matching sub-module 60 determines the initial estimated pose 80, which is used as a starting point to determine the final radar poses 54"); and loop closure detection (Paragraph 0006, “In an aspect, the automated driving controller executes instructions to execute a loop detection algorithm to detect the autonomous vehicle is currently situated in a previously visited location”); and estimate an optimal pose through graph optimization (Paragraph 0007, “In another aspect, in response to detecting the loop, the automated driving controller is instructed to execute a non-linear optimization routine to perform a global pose adjustment to a loop-closure pose graph to determine a loop-adjusted radar pose, and set the loop-adjusted radar pose as the final radar pose”). Ali does not disclose a module M1 configured to extract a ground point cloud, and compare two consecutive frames of point clouds, to remove ghost points below the ground and random points; and a module M3 configured to estimate and construct a pose graph using a relative pose transformation that is obtained through point cloud registration performed based on a normal distribution transformation, pre-integration performed using an estimated device velocity,
Qu discloses
A module M1 configured to extract a ground point cloud, and compare two consecutive frames of point clouds, to remove ghost points below the ground and random points (Section 2.2 First paragraph, "After extracting the A-KAZE feature points, two KD-trees are constructed for the reference image and the target image respectively. The next step is taking one of them in turn as the reference for KNN (K Nearest Neighbor) matching, then extract the public matching pairs of matching operations as the initial matching. Finally, the RANSAC algorithm is adopted to remove the outer points and estimate the affine transformation matrix between images" where KAZE is comparing different images as a process to remove outlier points which include at the ground and below, and Ali can use two consecutive frames where one acts as a reference).
Ali discloses ghost detections but not removing the ghost detections. Ali removing the ghost points between two consecutive frames would be advantageous for improving the calculation of the trajectory and removing false stationary objects, overall leading to better performance. As such, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Qu to add in ghost point removal to improve the trajectory calculation and accurate determination of stationary objects.
Yang discloses
A module M3 configured to estimate and construct a pose graph using a relative pose transformation that is obtained through point cloud registration performed based on a normal distribution transformation (Section 2.2, "The NDT algorithm is a typical distribution-based method that was first proposed in 2D LiDAR point cloud registration [7]. The LiDAR point cloud is represented by a group of Gaussian distributions with different probability density functions. To avoid the problem of incorrect correlation between data, the method gives the segmented smooth normal distribution representation of laser scanning data"), pre-integration performed using an estimated device velocity (Abstract, "Therefore, for these environments, the traditional point cloud registration method to register directly will lead to problems, such as a decline in registration accuracy, z-axis drift, and map ghosting. To solve the above problems, we propose a point cloud registration method based on IMU preintegration with the sensor characteristics of LiDAR and IMU"; Introduction Paragraph 4, "we propose a point cloud registration method with ground segmentation and IMU preintegration based on the motion characteristics of the ground mobile robot").
Ali discloses a relative transform but not a Normal Distribution Transform (NDT). The advantages of using a NDT include computational efficiency, as the calculation is using probability densities instead of individual points, and outliers (or ghost points) are less likely to affect the calculation. As such, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Yang so that the transformation between points can be more efficient and more resistant to outliers.
Ali discloses the use of closed loop recognition but not the use of pre-integration. Using pre-integration would be useful for effective tracking between frames when in a very dynamic situation, and for keeping drift between frames low. As such, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Yang to include pre-integration to improve the tracking and reduce drift.
Regarding claim 11 Ali discloses
A pose graph SLAM computation method based on a 4D millimeter-wave radar, the method comprising (Paragraph 0033, "For example, in another embodiment, the scan aggregator and filter 42 and the scan matching and radar pose estimator module 46 may be used for three-dimensional radar based simultaneous localization and mapping (SLAM) applications"; Abstract, "The automated driving controller determines a pose graph based on the most recent hyper-local submap and neighboring radar point cloud scans"; Paragraph 0015, "converting the relative transform between the last two consecutive pose estimate into velocity based on a time difference between the last two consecutive pose estimates"): comparing two consecutive frames of point clouds (Paragraph 0015, "converting the relative transform between the last two consecutive pose estimate into velocity based on a time difference between the last two consecutive pose estimates") estimating a linear velocity and an angular velocity of a device based on Doppler velocity information associated with the 4D radar point cloud (Paragraph 0003, “A millimeter wave radar may measure the range, angle, and Doppler (radial velocity) of moving objects.”; Paragraph 0015, "converting the relative transform between the last two consecutive pose estimate into velocity based on a time difference between the last two consecutive pose estimates" where a linear and angular velocity can be calculated; Paragraph 0042, " It is to be appreciated that the maximum distance threshold is adjusted based on both the linear and angular velocity of the autonomous vehicle 10 (FIG. 1). Finally, the ICP scan matching sub-module 60 determines the initial estimated pose 80, which is used as a starting point to determine the final radar poses 54"); and loop closure detection (Paragraph 0006, “In an aspect, the automated driving controller executes instructions to execute a loop detection algorithm to detect the autonomous vehicle is currently situated in a previously visited location”), the optimal pose being estimated through graph optimization (Paragraph 0007, “In another aspect, in response to detecting the loop, the automated driving controller is instructed to execute a non-linear optimization routine to perform a global pose adjustment to a loop-closure pose graph to determine a loop-adjusted radar pose, and set the loop-adjusted radar pose as the final radar pose”). Ali does not disclose extracting a ground point cloud and comparing two consecutive frames of point clouds of a 4D radar point cloud generated by the 4D millimeter-wave radar to remove ghost points below the ground point cloud and random points from the 4D radar point cloud to create a filtered 4D radar point cloud; and estimating and constructing a pose graph using a relative pose transformation and estimation of an optimal pose, the relative pose transformation being obtained through point cloud registration performed based on a normal distribution transformation, pre-integration performed using an estimated device velocity,
Qu discloses
Extracting a ground point cloud and comparing two consecutive frames of point clouds of a 4D radar point cloud generated by the 4D millimeter-wave radar to remove ghost points below the ground point cloud and random points from the 4D radar point cloud to create a filtered 4D radar point cloud (Section 2.2 First paragraph, "After extracting the A-KAZE feature points, two KD-trees are constructed for the reference image and the target image respectively. The next step is taking one of them in turn as the reference for KNN (K Nearest Neighbor) matching, then extract the public matching pairs of matching operations as the initial matching. Finally, the RANSAC algorithm is adopted to remove the outer points and estimate the affine transformation matrix between images" where KAZE is comparing different images as a process to remove outlier points which include at the ground and below, and Ali can use two consecutive frames where one acts as a reference).
Ali discloses ghost detections but not removing the ghost detections. Ali removing the ghost points between two consecutive frames would be advantageous for improving the calculation of the trajectory and removing false stationary objects, overall leading to better performance. As such, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Qu to add in ghost point removal to improve the trajectory calculation and accurate determination of stationary objects.
Yang discloses
Estimating and constructing a pose graph using a relative pose transformation and estimation of an optimal pose, the relative pose transformation being obtained through point cloud registration performed based on a normal distribution transformation (Section 2.2, "The NDT algorithm is a typical distribution-based method that was first proposed in 2D LiDAR point cloud registration [7]. The LiDAR point cloud is represented by a group of Gaussian distributions with different probability density functions. To avoid the problem of incorrect correlation between data, the method gives the segmented smooth normal distribution representation of laser scanning data"), pre-integration performed using an estimated device velocity (Abstract, "Therefore, for these environments, the traditional point cloud registration method to register directly will lead to problems, such as a decline in registration accuracy, z-axis drift, and map ghosting. To solve the above problems, we propose a point cloud registration method based on IMU preintegration with the sensor characteristics of LiDAR and IMU"; Introduction Paragraph 4, "we propose a point cloud registration method with ground segmentation and IMU preintegration based on the motion characteristics of the ground mobile robot").
Ali discloses a relative transform but not a Normal Distribution Transform (NDT). The advantages of using a NDT include computational efficiency, as the calculation is using probability densities instead of individual points, and outliers (or ghost points) are less likely to affect the calculation. As such, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Yang so that the transformation between points can be more efficient and more resistant to outliers.
Ali discloses the use of closed loop recognition but not the use of pre-integration. Using pre-integration would be useful for effective tracking between frames when in a very dynamic situation, and for keeping drift between frames low. As such, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Yang to include pre-integration to improve the tracking and reduce drift.
Claims 2 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Ali (US 20230126333 A1) in view of Qu (2019) further in view of Yang (2023) further in view of Maile (DE 102018215753 A1) further in view of Yan (2022) [D. Yan, C. Zeng, L. Gui, S. Li, J. Luo and L. Song, "Elimination of Semi static Objects in SLAM Maps Based on Semantic Segmentation," 2022 IEEE 5th Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), Chongqing, China, 2022, pp. 1058-1063, doi: 10.1109/IMCEC55388.2022.10020123.].
Regarding claim 2 the combination of Ali, Qu, and Yang discloses
The pose graph SLAM computation method based on a 4D millimeter-wave radar according to claim 1. Ali discloses wherein in step S1,the ground point cloud is extracted, and the two consecutive frames of point clouds are compared, to remove the ghost points below the ground and the unstable random points, so as to reduce noise in the 4D radar point cloud (As established in claim 1), for the original radar point cloud P={Pi }, i E { 1, 2, ..., n } measured by a 4D radar, n is a number of points in the point cloud (Abstract, "The automated driving controller determines an initial estimated pose by aligning a latest aggregated filtered data point cloud scan with the most recent hyper-local submap based on an iterative closest point (ICP) alignment algorithm" where a point cloud has various points); a pose transformation between the current point cloud and the previous frame of point cloud is computed based on the device velocity, the previous frame of point cloud is transformed into a coordinate system of the current frame (Paragraph 0043, "The relative transform between the last two consecutive pose estimates associated with the latest aggregated filtered data point cloud scans 50 is then converted into velocity based on a time difference between the last two consecutive pose estimates associated with the latest radar point scans…Then, the relative transform is converted from velocity into position based on a time difference between the last pose estimate and a current pose estimate, where a difference in position between the last pose estimate and the current pose estimate is the predicted pose estimate"), and if the transformed previous frame of point cloud does not exist within a preset range near a point in the current frame, the point is classified as a random point and filtered out (Paragraph 0047, "The multi-view non-linear ICP adjustment sub-module 62 then determines the pose graph 100 by computing point-to-point correspondences for each of the neighboring radar point cloud scans 104 and the most recent hyper-local submap 84 based on a maximum correspondence distance threshold."); and a rotation transformation Rk_1 and a translation transformation tk-1 between the current point cloud Pk and the previous frame of point cloud Pk-1 adjacent to the current point cloud are computed: tk-1 = vk-1Δt (Paragraph 0043, "The relative transform between the last two consecutive pose estimates associated with the latest aggregated filtered data point cloud scans 50 is then converted into velocity based on a time difference between the last two consecutive pose estimates associated with the latest radar point scans. Specifically, the relative transform is converted into se(3) space based on a lie algebra log function using the time difference between the last two consecutive pose estimates” where se(3) space has rotations and translations) Rk-1= Exp(ωk-1Δt) wherein vk-1 and ωk-1 are respectively a device linear velocity estimate and a device angular velocity estimate of the previous frame, Δt is a time difference between the two frames, Exp(- ):3-> SO(3) is an exponential mapping of a three- dimensional rotation, R3 is a three-dimensional real number vector space, so(3) is a three-dimensional special orthogonal group , and Pj-1 is obtained by transforming the previous frame of point cloud into the coordinate system of the current frame using Rk_1 and tk_1 (Paragraph 0043, "Specifically, the relative transform is converted into special Euclidian group SE(3) space based on a lie algebra exponential function using the time difference between the last pose estimate and a current pose estimate"; Paragraph 0042, " It is to be appreciated that the maximum distance threshold is adjusted based on both the linear and angular velocity of the autonomous vehicle 10 (FIG. 1). Finally, the ICP scan matching sub-module 60 determines the initial estimated pose 80, which is used as a starting point to determine the final radar poses 54" where the invention is cognizant of the angular and linear velocity); and if no point in exists within a preset range of a point in the current frame, the current point is classified as a random point and filtered out (Paragraph 0047, "The multi-view non-linear ICP adjustment sub-module 62 then determines the pose graph 100 by computing point-to-point correspondences for each of the neighboring radar point cloud scans 104 and the most recent hyper-local submap 84 based on a maximum correspondence distance threshold"). Ali does not disclose wherein step S1 comprises the following steps: step S1.1: ghost point removal, wherein the ground point cloud is extracted from an original radar point cloud and the ghost points below the ground are filtered out; ground points are extracted using a random sample consensus algorithm, and the ghost points below the ground are filtered out, and step S1.2: random point removal, wherein the random points are identified and filtered out by comparing a current point cloud and a previous frame of point cloud adjacent to the current point cloud, for the current point cloud and the previous frame of point cloud adjacent to the current point cloud; a position of each point in a radar coordinate system is expressed as Pi= (xi, yi, zi)T = (rcosθ cosφ, ri sinθ cosφ , ri sinφ)T wherein ri is a distance of the point measured by the 4D radar θ is an azimuth angle of the point measured by the 4D radar, and φi is an elevation angle of the point measured by the 4D radar; And for the original radar point cloud, a point with a distance less than a threshold δr and a height within a threshold δh near a mounting height of the radar is retained, an upward normal vector of each point is computed using a principal component analysis method a point with a normal vector having an angle less than δn with a z-axis positive unit vector is retained.
Qu discloses
Wherein step S1 comprises the following steps: step S1.1: ghost point removal, wherein the ground point cloud is extracted from an original radar point cloud and the ghost points below the ground are filtered out; (Section 2.2 First paragraph, "After extracting the A-KAZE feature points, two KD-trees are constructed for the reference image and the target image respectively. The next step is taking one of them in turn as the reference for KNN (K Nearest Neighbor) matching, then extract the public matching pairs of matching operations as the initial matching. Finally, the RANSAC algorithm is adopted to remove the outer points and estimate the affine transformation matrix between images" which include ground points, and all ghost points including below the ground are extracted) ground points are extracted using a random sample consensus algorithm, and the ghost points below the ground are filtered out (Section 2.2 First paragraph, "After extracting the A-KAZE feature points, two KD-trees are constructed for the reference image and the target image respectively. The next step is taking one of them in turn as the reference for KNN (K Nearest Neighbor) matching, then extract the public matching pairs of matching operations as the initial matching. Finally, the RANSAC algorithm is adopted to remove the outer points and estimate the affine transformation matrix between images" which include ground points, and all ghost points including below the ground are extracted and AKAZE uses RANSAC); and step S1.2: random point removal, wherein the random points are identified and filtered out by comparing a current point cloud and a previous frame of point cloud adjacent to the current point cloud, for the current point cloud and the previous frame of point cloud adjacent to the current point cloud (Section 2.2 First paragraph, "After extracting the A-KAZE feature points, two KD-trees are constructed for the reference image and the target image respectively. The next step is taking one of them in turn as the reference for KNN (K Nearest Neighbor) matching, then extract the public matching pairs of matching operations as the initial matching. Finally, the RANSAC algorithm is adopted to remove the outer points and estimate the affine transformation matrix between images" where Ali can use this on consecutive frames so they would be adjacent).
Ali discloses ghost detections but not removing the ghost detections. Ali removing the ghost points between two consecutive frames would be advantageous for improving the calculation of the trajectory and removing false stationary objects, overall leading to better performance. Extracting points, such as features like the ground, are advantageous for tracking between multiple frames and recognizing objects/features of the environment. As such, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Qu to add in ghost point removal to improve the trajectory calculation and accurate determination of stationary objects and feature extraction for understanding the environment.
Maile discloses
A position of each point in a radar coordinate system is expressed as Pi= (xi, yi, zi)T = (rcosθ cosφ, ri sinθ cosφ , ri sinφ)T wherein ri is a distance of the point measured by the 4D radar θ is an azimuth angle of the point measured by the 4D radar, and φi is an elevation angle of the point measured by the 4D radar (Page 3 Paragraph three, "In a preferred embodiment, the preprocessing unit is designed to transform the sensor data from a spherical coordinate system into a Cartesian coordinate system. Additionally or alternatively, the preprocessing unit is designed to filter out noisy and unreliable sensor data before generating the sensor data set. The coordinate transformation enables an efficient calculation").
Ali discloses pose graphs in a coordinate system but it does not disclose interchanging between linear and spherical coordinates. Ali using both linear and spherical coordinates would be advantageous for using the most efficient system for the situation such as moving in a straight line or making a lot of turns or circular movements, depending on the system being used the calculation can be made easier. As such it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Maile to add in a flexibility of use between coordinate systems to make calculations easier in various scenarios.
Yan discloses
And for the original radar point cloud, a point with a distance less than a threshold δr and a height within a threshold δh near a mounting height of the radar is retained (Page 1061 Column 1 Paragraph 6, " to determine the horizontal distance 𝐷ℎ and normal distance 𝐷𝑛. If 𝐷ℎ < δh and 𝐷𝑛 <δn (δh,δn is Euclidean distance threshold and normal distance threshold respectively), then these points are added to the candidate cluster as candidate points [9]” where normal distance is height ; Page 1059 Column 2 Paragraph 1, "but their main application scenario is single frame point cloud ground segmentation, involving parameters such as laser radar installation location" which include mounting height), an upward normal vector of each point is computed using a principal component analysis method a point with a normal vector having an angle less than δn with a z-axis positive unit vector is retained, (Page 1059 Column 2 Paragraph 3, "Principal component analysis (PCA) algorithm is widely used in normal vector estimation because of its simplicity[6]"; Page 1060 column 1 paragraph 4 Equation 4, "For normal 𝐧𝑖. In practical application, the estimated normal vector is bidirectional, so redirection is required, as shown in Formula (4)" where it points up so the angle is zero).
Ali discloses using a distance threshold but not a distance and height threshold based on a normal vector. Ali using the distance and height thresholds along with a principal component analysis for the normal vector can add effective constraints to the pose graph that will help with tracking and recognizing the orientation of the moving object in the pose graph; the result will help with point removal in a way that Ali, by itself, does not address. The process would be able to differentiate between a point that is near the main mass of a moving object but does not have the proper orientation. As such, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Yan to incorporate the thresholds and the normal vector by principal analysis in order to differentiate between points that a near the main body of points but not associated with the main body.
Regarding claim 7 the combination of Ali, Qu, and Yang discloses
The pose graph SLAM computation system based on a 4D millimeter-wave radar according to claim 6. Ali discloses wherein in the module M1,the ground point cloud is extracted, and the two consecutive frames of point clouds are compared, to remove the ghost points below the ground and the unstable random points, so as to reduce noise in the 4D radar point cloud, wherein a module M1. 1 is configured to perform the following operation: (As established in claim 6); for the original radar point cloud 3= {Pi}, i E { 1, 2, ..., n} measured by a 4D radar, n is a number of points in the point cloud (Abstract, "The automated driving controller determines an initial estimated pose by aligning a latest aggregated filtered data point cloud scan with the most recent hyper-local submap based on an iterative closest point (ICP) alignment algorithm" where a point cloud has various points); a pose transformation between the current point cloud and the previous frame of point cloud is computed based on the device velocity, the previous frame of point cloud is transformed into a coordinate system of the current frame (Paragraph 0043, "The relative transform between the last two consecutive pose estimates associated with the latest aggregated filtered data point cloud scans 50 is then converted into velocity based on a time difference between the last two consecutive pose estimates associated with the latest radar point scans…Then, the relative transform is converted from velocity into position based on a time difference between the last pose estimate and a current pose estimate, where a difference in position between the last pose estimate and the current pose estimate is the predicted pose estimate"), and if the transformed previous frame of point cloud does not exist within a preset range near a point in the current frame, the point is classified as a random point and filtered out (Paragraph 0047, "The multi-view non-linear ICP adjustment sub-module 62 then determines the pose graph 100 by computing point-to-point correspondences for each of the neighboring radar point cloud scans 104 and the most recent hyper-local submap 84 based on a maximum correspondence distance threshold."); and a rotation transformation Rk_1 and a translation transformation tk-1 between the current point cloud Pk and the previous frame of point cloud Pk-1 adjacent to the current point cloud are computed: tk-1 = vk-1Δt (Paragraph 0043, "The relative transform between the last two consecutive pose estimates associated with the latest aggregated filtered data point cloud scans 50 is then converted into velocity based on a time difference between the last two consecutive pose estimates associated with the latest radar point scans. Specifically, the relative transform is converted into se(3) space based on a lie algebra log function using the time difference between the last two consecutive pose estimates” where se(3) space has rotations and translations) Rk-1= Exp(ωk-1Δt) wherein vk-1 and ωk-1 are respectively a device linear velocity estimate and a device angular velocity estimate of the previous frame, Δt is a time difference between the two frames, Exp(- ):3-> SO(3) is an exponential mapping of a three- dimensional rotation, R3 is a three-dimensional real number vector space, so(3) is a three-dimensional special orthogonal group , and Pj-1 is obtained by transforming the previous frame of point cloud into the coordinate system of the current frame using Rk_1 and tk_1 (Paragraph 0043, "Specifically, the relative transform is converted into special Euclidian group SE(3) space based on a lie algebra exponential function using the time difference between the last pose estimate and a current pose estimate"; Paragraph 0042, " It is to be appreciated that the maximum distance threshold is adjusted based on both the linear and angular velocity of the autonomous vehicle 10 (FIG. 1). Finally, the ICP scan matching sub-module 60 determines the initial estimated pose 80, which is used as a starting point to determine the final radar poses 54" where the invention is cognizant of the angular and linear velocity) and if no point in exists within a preset range of a point in the current frame, the current point is classified as a random point and filtered out (Paragraph 0047, "The multi-view non-linear ICP adjustment sub-module 62 then determines the pose graph 100 by computing point-to-point correspondences for each of the neighboring radar point cloud scans 104 and the most recent hyper-local submap 84 based on a maximum correspondence distance threshold"). Ali does not disclose ghost point removal, wherein the ground point cloud is extracted from an original radar point cloud and the ghost points below the ground are filtered out; a position of each point in a radar coordinate system is expressed as Pi= (xi, yi, zi)T = (rcosθ cosφ, ri sinθ cosφ , ri sinφ)T wherein ri is a distance of the point measured by the 4D radar θ is an azimuth angle of the point measured by the 4D radar, and φi is an elevation angle of the point measured by the 4D radar; And for the original radar point cloud, a point with a distance less than a threshold δr and a height within a threshold δh near a mounting height of the radar is retained, an upward normal vector of each point is computed using a principal component analysis method a point with a normal vector having an angle less than δn with a z-axis positive unit vector is retained; ground points are extracted using a random sample consensus algorithm, and the ghost points below the ground are filtered out; and a module M1.2 is configured to perform the following operation: random point removal, wherein the random points are identified and filtered out by comparing a current point cloud and a previous frame of point cloud adjacent to the current point cloud, for the current point cloud and the previous frame of point cloud adjacent to the current point cloud.
Qu discloses
Wherein step S1 comprises the following steps: step S1.1: ghost point removal, wherein the ground point cloud is extracted from an original radar point cloud and the ghost points below the ground are filtered out; (Section 2.2 First paragraph, "After extracting the A-KAZE feature points, two KD-trees are constructed for the reference image and the target image respectively. The next step is taking one of them in turn as the reference for KNN (K Nearest Neighbor) matching, then extract the public matching pairs of matching operations as the initial matching. Finally, the RANSAC algorithm is adopted to remove the outer points and estimate the affine transformation matrix between images" which include ground points, and all ghost points including below the ground are extracted) ground points are extracted using a random sample consensus algorithm, and the ghost points below the ground are filtered out (Section 2.2 First paragraph, "After extracting the A-KAZE feature points, two KD-trees are constructed for the reference image and the target image respectively. The next step is taking one of them in turn as the reference for KNN (K Nearest Neighbor) matching, then extract the public matching pairs of matching operations as the initial matching. Finally, the RANSAC algorithm is adopted to remove the outer points and estimate the affine transformation matrix between images" which include ground points, and all ghost points including below the ground are extracted and AKAZE uses RANSAC); and step S1.2: random point removal, wherein the random points are identified and filtered out by comparing a current point cloud and a previous frame of point cloud adjacent to the current point cloud, for the current point cloud and the previous frame of point cloud adjacent to the current point cloud (Section 2.2 First paragraph, "After extracting the A-KAZE feature points, two KD-trees are constructed for the reference image and the target image respectively. The next step is taking one of them in turn as the reference for KNN (K Nearest Neighbor) matching, then extract the public matching pairs of matching operations as the initial matching. Finally, the RANSAC algorithm is adopted to remove the outer points and estimate the affine transformation matrix between images" where Ali can use this on consecutive frames so they would be adjacent).
Ali discloses ghost detections but not removing the ghost detections. Ali removing the ghost points between two consecutive frames would be advantageous for improving the calculation of the trajectory and removing false stationary objects, overall leading to better performance. Extracting points, such as features like the ground, are advantageous for tracking between multiple frames and recognizing objects/features of the environment. As such, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Qu to add in ghost point removal to improve the trajectory calculation and accurate determination of stationary objects and feature extraction for understanding the environment.
Maile discloses
A position of each point in a radar coordinate system is expressed as Pi= (xi, yi, zi)T = (rcosθ cosφ, ri sinθ cosφ , ri sinφ)T wherein ri is a distance of the point measured by the 4D radar θ is an azimuth angle of the point measured by the 4D radar, and φi is an elevation angle of the point measured by the 4D radar (Page 3 Paragraph three, "In a preferred embodiment, the preprocessing unit is designed to transform the sensor data from a spherical coordinate system into a Cartesian coordinate system. Additionally or alternatively, the preprocessing unit is designed to filter out noisy and unreliable sensor data before generating the sensor data set. The coordinate transformation enables an efficient calculation").
Ali discloses pose graphs in a coordinate system but it does not disclose interchanging between linear and spherical coordinates. Ali using both linear and spherical coordinates would be advantageous for using the most efficient system for the situation such as moving in a straight line or making a lot of turns or circular movements, depending on the system being used the calculation can be made easier. As such it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Maile to add in a flexibility of use between coordinate systems to make calculations easier in various scenarios.
Yan discloses
And for the original radar point cloud, a point with a distance less than a threshold δr and a height within a threshold δh near a mounting height of the radar is retained (Page 1061 Column 1 Paragraph 6, " to determine the horizontal distance 𝐷ℎ and normal distance 𝐷𝑛. If 𝐷ℎ < δh and 𝐷𝑛 <δn (δh,δn is Euclidean distance threshold and normal distance threshold respectively), then these points are added to the candidate cluster as candidate points [9]” where normal distance is height ; Page 1059 Column 2 Paragraph 1, "but their main application scenario is single frame point cloud ground segmentation, involving parameters such as laser radar installation location" which include mounting height), an upward normal vector of each point is computed using a principal component analysis method a point with a normal vector having an angle less than δn with a z-axis positive unit vector is retained, (Page 1059 Column 2 Paragraph 3, "Principal component analysis (PCA) algorithm is widely used in normal vector estimation because of its simplicity[6]"; Page 1060 column 1 paragraph 4 Equation 4, "For normal 𝐧𝑖. In practical application, the estimated normal vector is bidirectional, so redirection is required, as shown in Formula (4)" where it points up so the angle is zero).
Ali discloses using a distance threshold but not a distance and height threshold based on a normal vector. Ali using the distance and height thresholds along with a principal component analysis for the normal vector can add effective constraints to the pose graph that will help with tracking and recognizing the orientation of the moving object in the pose graph; the result will help with point removal in a way that Ali, by itself, does not address. The process would be able to differentiate between a point that is near the main mass of a moving object but does not have the proper orientation. As such, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Yan to incorporate the thresholds and the normal vector by principal analysis in order to differentiate between points that a near the main body of points but not associated with the main body.
Claims 3 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Ali (US 20230126333 A1) in view of Qu (2019) further in view of Yang (2023) further in view of Maile (DE 102018215753 A1).
Regarding claim 3 the combination of Ali, Qu, and Yang discloses
The pose graph SLAM computation method based on a 4D millimeter-wave radar according to claim 1. Ali discloses wherein in step S2, the following steps are comprised: step S2.1:, wherein for an ith point in the 4D radar point cloud, a relationship between a Doppler velocity of the point and a velocity of the radar is as follows: Vr =-diVs wherein vr,i is the Doppler velocity of the ithpoint, di= unit vector is a unit direction vector of the point relative to the radar, and vs is the velocity of the radar (Paragraph 0043, "The relative transform between the last two consecutive pose estimates associated with the latest aggregated filtered data point cloud scans 50 is then converted into velocity based on a time difference between the last two consecutive pose estimates associated with the latest radar point scans” where the velocities would be negative of each other i.e. a car going forward in their frame of reference sees everything else approaching); and a relationship between the velocity of the radar and the device velocity is as follows: Vs = RsT (V + ω x t5) wherein t and RS are respectively a mounting position and a pose of the 4D radar in a device coordinate system, veR3 and ωeR3 are respectively the linear velocity and the angular velocity of the device, and for all static points in the 4D radar point cloud, the Doppler velocity and the device velocity both conform to the following equation: -I3x3 -t^], wherein m is a number of all the static points in the point cloud, I3x3 is a 3 x3 unit matrix, t^eR3x3 is an antisymmetric matrix of ts, Ra3X represents a set of 3 x 3 real matrices, and dynamic outliers are removed and the static points are extracted based on relationships between Doppler velocities of all the static points in the radar point cloud and the device velocity (Paragraph 0043, "The relative transform between the last two consecutive pose estimates associated with the latest aggregated filtered data point cloud scans 50 is then converted into velocity based on a time difference between the last two consecutive pose estimates associated with the latest radar point scans” where this equation includes a zero offset for t for example where the radar is mounted on a vehicle and considered at origin and the equation turns into a vector transformation where in this case the relationship between the device and radar velocity is simply a negative ); and step S2.2: least squares estimation, wherein after the point clouds are recognized, the device velocity is computed based on the relationships between the Doppler velocities of all the static points (Paragraph 0003, "A millimeter wave radar may measure the range, angle, and Doppler (radial velocity) of moving objects. A radar point cloud may be determined based on the data collected by the millimeter wave radar based on various clustering and tracking algorithms,"; Paragraph 0043, "The relative transform between the last two consecutive pose estimates associated with the latest aggregated filtered data point cloud scans 50 is then converted into velocity based on a time difference between the last two consecutive pose estimates associated with the latest radar point scans” where velocity is calculated from the poses and the poses can include static points; Paragraph 0058, "As mentioned above, one example of a non-linear optimization routine for determining the loop-adjusted radar poses 86 is a non-linear least-squares routine such as, but not limited to, the Levenberg-Marquardt algorithm" where if this process is happening near a closed loop it is using the least squares) and the device velocity using the least square method (Paragraph 0058, "As mentioned above, one example of a non-linear optimization routine for determining the loop-adjusted radar poses 86 is a non-linear least-squares routine such as, but not limited to, the Levenberg-Marquardt algorithm" where the device is on the vehicle). Ali does not disclose static point extraction or spherical coordinates (cosθ cosφ, sinθ cosφ , sinφ ) for a unit vector, using the random sample consensus algorithm
Qu discloses
Static point extraction and using the random sample consensus algorithm (Section 2.2 First paragraph, "After extracting the A-KAZE feature points, two KD-trees are constructed for the reference image and the target image respectively. The next step is taking one of them in turn as the reference for KNN (K Nearest Neighbor) matching, then extract the public matching pairs of matching operations as the initial matching. Finally, the RANSAC algorithm is adopted to remove the outer points and estimate the affine transformation matrix between images" where KAZE can extract static points).
Ali includes the use of static objects in its calculations but does not disclose the extraction of static points or using the random sample consensus algorithm for the velocities. Extracting points, such as features like the ground, are advantageous for tracking between multiple frames and recognizing objects/features of the environment. Ali already uses velocity but incorporating a random sample consensus algorithm can improve the calculation with an added layer against outliers as a check against current estimates and as a means of processing frames with a high number of outliers. As such, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Qu to add in accurate determination of stationary objects and feature extraction for understanding the environment and improved/validation method of outlier mitigation.
Maile discloses
Spherical coordinates (cosθ cosφ, sinθ cosφ , sinφ ) for a unit vector (Page 3 Paragraph three, "In a preferred embodiment, the preprocessing unit is designed to transform the sensor data from a spherical coordinate system into a Cartesian coordinate system”).
Ali discloses pose graphs in a coordinate system but it does not disclose interchanging between linear and spherical coordinates. Ali using both linear and spherical coordinates would be advantageous for using the most efficient system for the situation such as moving in a straight line or making a lot of turns or circular movements, depending on the system being used the calculation can be made easier. As such it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Maile to add in a flexibility of use between coordinate systems to make calculations easier in various scenarios.
Regarding claim 8 the combination of Ali, Qu, and Yang discloses
The pose graph SLAM computation method based on a 4D millimeter-wave radar according to claim 6. Ali discloses wherein in the module M2 a module 2.1 (Abstract, "The automated driving controller determines an initial estimated pose by aligning a latest aggregated filtered data point cloud scan with the most recent hyper-local submap based on an iterative closest point (ICP) alignment algorithm") is configured to perform the following operation: wherein for an ith point in the 4D radar point cloud, a relationship between a Doppler velocity of the point and a velocity of the radar is as follows: Vr =-diVs wherein vr,i is the Doppler velocity of the ithpoint, di= unit vector is a unit direction vector of the point relative to the radar, and vs is the velocity of the radar (Paragraph 0043, "The relative transform between the last two consecutive pose estimates associated with the latest aggregated filtered data point cloud scans 50 is then converted into velocity based on a time difference between the last two consecutive pose estimates associated with the latest radar point scans” where the velocities would be negative of each other i.e. a car going forward in their frame of reference sees everything else approaching); and a relationship between the velocity of the radar and the device velocity is as follows: Vs = RsT (V + ω x t5) wherein t and RS are respectively a mounting position and a pose of the 4D radar in a device coordinate system, veR3 and ωeR3 are respectively the linear velocity and the angular velocity of the device, and for all static points in the 4D radar point cloud, the Doppler velocity and the device velocity both conform to the following equation: -I3x3 -t^], wherein m is a number of all the static points in the point cloud, I3x3 is a 3 x3 unit matrix, t^eR3x3 is an antisymmetric matrix of ts, Ra3X represents a set of 3 x 3 real matrices, and dynamic outliers are removed and the static points are extracted based on relationships between Doppler velocities of all the static points in the radar point cloud and the device velocity (Paragraph 0043, "The relative transform between the last two consecutive pose estimates associated with the latest aggregated filtered data point cloud scans 50 is then converted into velocity based on a time difference between the last two consecutive pose estimates associated with the latest radar point scans” where this equation includes a zero offset for t for example where the radar is mounted on a vehicle and considered at origin and the equation turns into a vector transformation where in this case the relationship between the device and radar velocity is simply a negative ); and a module M2.2 (Abstract, "The automated driving controller determines an initial estimated pose by aligning a latest aggregated filtered data point cloud scan with the most recent hyper-local submap based on an iterative closest point (ICP) alignment algorithm") is configured to perform the following operation: least squares estimation, wherein after the point clouds are recognized, the device velocity is computed based on the relationships between the Doppler velocities of all the static points (Paragraph 0003, "A millimeter wave radar may measure the range, angle, and Doppler (radial velocity) of moving objects. A radar point cloud may be determined based on the data collected by the millimeter wave radar based on various clustering and tracking algorithms,"; Paragraph 0043, "The relative transform between the last two consecutive pose estimates associated with the latest aggregated filtered data point cloud scans 50 is then converted into velocity based on a time difference between the last two consecutive pose estimates associated with the latest radar point scans” where velocity is calculated from the poses and the poses can include static points; Paragraph 0058, "As mentioned above, one example of a non-linear optimization routine for determining the loop-adjusted radar poses 86 is a non-linear least-squares routine such as, but not limited to, the Levenberg-Marquardt algorithm" where if this process is happening near a closed loop it is using the least squares) and the device velocity using the least square method (Paragraph 0058, "As mentioned above, one example of a non-linear optimization routine for determining the loop-adjusted radar poses 86 is a non-linear least-squares routine such as, but not limited to, the Levenberg-Marquardt algorithm" where the device is on the vehicle). Ali does not disclose static point extraction or spherical coordinates (cosθ cosφ, sinθ cosφ , sinφ ) for a unit vector, using the random sample consensus algorithm
Qu discloses
Static point extraction and using the random sample consensus algorithm (Section 2.2 First paragraph, "After extracting the A-KAZE feature points, two KD-trees are constructed for the reference image and the target image respectively. The next step is taking one of them in turn as the reference for KNN (K Nearest Neighbor) matching, then extract the public matching pairs of matching operations as the initial matching. Finally, the RANSAC algorithm is adopted to remove the outer points and estimate the affine transformation matrix between images" where KAZE can extract static points).
Ali includes the use of static objects in its calculations but does not disclose the extraction of static points or using the random sample consensus algorithm for the velocities. Extracting points, such as features like the ground, are advantageous for tracking between multiple frames and recognizing objects/features of the environment. Ali already uses velocity but incorporating a random sample consensus algorithm can improve the calculation with an added layer against outliers as a check against current estimates and as a means of processing frames with a high number of outliers. As such, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Qu to add in accurate determination of stationary objects and feature extraction for understanding the environment and improved/validation method of outlier mitigation.
Maile discloses
Spherical coordinates (cosθ cosφ, sinθ cosφ , sinφ ) for a unit vector (Page 3 Paragraph three, "In a preferred embodiment, the preprocessing unit is designed to transform the sensor data from a spherical coordinate system into a Cartesian coordinate system”).
Ali discloses pose graphs in a coordinate system but it does not disclose interchanging between linear and spherical coordinates. Ali using both linear and spherical coordinates would be advantageous for using the most efficient system for the situation such as moving in a straight line or making a lot of turns or circular movements, depending on the system being used the calculation can be made easier. As such it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Maile to add in a flexibility of use between coordinate systems to make calculations easier in various scenarios.
Claims 12 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Ali (US 20230126333 A1) in view of Qu (2019) further in view of Yang (2023) further in view of Yan (2022) [D. Yan, C. Zeng, L. Gui, S. Li, J. Luo and L. Song, "Elimination of Semi static Objects in SLAM Maps Based on Semantic Segmentation," 2022 IEEE 5th Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), Chongqing, China, 2022, pp. 1058-1063, doi: 10.1109/IMCEC55388.2022.10020123.].
Regarding claim 12 the combination of Ali, Qu, and Yang discloses
The method of claim 11. Ali discloses wherein removing the random points comprises comparing a current point cloud frame and a previous frame of the 4D radar point cloud adjacent to the current point cloud frame by computing a pose transformation between the current point cloud frame and the previous frame based on the device velocity to transform the previous frame into a coordinate system of the current point cloud frame (Paragraph 0043, "The relative transform between the last two consecutive pose estimates associated with the latest aggregated filtered data point cloud scans 50 is then converted into velocity based on a time difference between the last two consecutive pose estimates associated with the latest radar point scans…Then, the relative transform is converted from velocity into position based on a time difference between the last pose estimate and a current pose estimate, where a difference in position between the last pose estimate and the current pose estimate is the predicted pose estimate"), and if the previous frame after being transformed does not exist within a preset range near a point in the current point cloud frame, the point in the current point cloud frame is classified as a random point and filtered out (Paragraph 0047, "The multi-view non-linear ICP adjustment sub-module 62 then determines the pose graph 100 by computing point-to-point correspondences for each of the neighboring radar point cloud scans 104 and the most recent hyper-local submap 84 based on a maximum correspondence distance threshold" which would also apply if the single point wasn’t near any points in the previous frame). Ali does not disclose wherein extracting the ground point cloud comprises extracting the ground point cloud using a random sample consensus algorithm on the 4D radar point cloud measured by the 4D millimeter- wave radar, wherein the 4D radar point cloud includes a number of points positioned in a radar coordinate system, removing the ghost points from the 4D radar point cloud further comprising retaining each point of the number of points with a distance less than a threshold δr and a height within a threshold δh and an upward normal vector having an angle less than a threshold angle δn with a z-axis positive unit vector, the upward normal vector being computed using a principal component analysis method
Qu discloses
Wherein extracting the ground point cloud comprises extracting the ground point cloud using a random sample consensus algorithm on the 4D radar point cloud measured by the 4D millimeter- wave radar, wherein the 4D radar point cloud includes a number of points positioned in a radar coordinate system (Section 2.2 First paragraph, "After extracting the A-KAZE feature points, two KD-trees are constructed for the reference image and the target image respectively. The next step is taking one of them in turn as the reference for KNN (K Nearest Neighbor) matching, then extract the public matching pairs of matching operations as the initial matching. Finally, the RANSAC algorithm is adopted to remove the outer points and estimate the affine transformation matrix between images" which include ground points, and all ghost points including below the ground are extracted and AKAZE uses RANSAC),
Yan discloses
Removing the ghost points from the 4D radar point cloud further comprising retaining each point of the number of points with a distance less than a threshold δr and a height within a threshold δh (Page 1061 Column 1 Paragraph 6, " to determine the horizontal distance 𝐷ℎ and normal distance 𝐷𝑛. If 𝐷ℎ < δh and 𝐷𝑛 <δn (δh,δn is Euclidean distance threshold and normal distance threshold respectively), then these points are added to the candidate cluster as candidate points [9]” where normal distance is height ; Page 1059 Column 2 Paragraph 1, "but their main application scenario is single frame point cloud ground segmentation, involving parameters such as laser radar installation location" which include mounting height) and an upward normal vector having an angle less than a threshold angle δn with a z-axis positive unit vector, the upward normal vector being computed using a principal component analysis method (Page 1059 Column 2 Paragraph 3, "Principal component analysis (PCA) algorithm is widely used in normal vector estimation because of its simplicity[6]"; Page 1060 column 1 paragraph 4 Equation 4, "For normal 𝐧𝑖. In practical application, the estimated normal vector is bidirectional, so redirection is required, as shown in Formula (4)" where it points up so the angle is zero).
Ali discloses using a distance threshold but not a distance and height threshold based on a normal vector. Ali using the distance and height thresholds along with a principal component analysis for the normal vector can add effective constraints to the pose graph that will help with tracking and recognizing the orientation of the moving object in the pose graph; the result will help with point removal in a way that Ali, by itself, does not address. The process would be able to differentiate between a point that is near the main mass of a moving object but does not have the proper orientation. As such, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify Ali with Yan to incorporate the thresholds and the normal vector by principal analysis in order to differentiate between points that a near the main body of points but not associated with the main body.
Regarding claim 13 the combination of Ali, Qu, Yang, and Yan discloses
The method of claim 12. Ali further discloses wherein computing the pose transformation comprises: computing a rotation transformation (Rk-1) between the current point cloud frame (Pk) and the previous frame (Pk-1) based on an exponential function of a product of a device angular velocity estimate (ωk_1) of the previous frame (Pk_1) and a time difference (Δt) between the current point cloud frame (Pk) and the previous frame (Pk_1) (Paragraph 0043, "Specifically, the relative transform is converted into special Euclidian group SE(3) space based on a lie algebra exponential function using the time difference between the last pose estimate and a current pose estimate" which includes translations and rotations; Paragraph 0047, "however, it is to be appreciated that the maximum correspondence distance threshold is adjusted based on both the linear and angular velocity of the autonomous vehicle 10 (FIG. 1)" where se(3) considers the maximum correspondence when doing the transform); computing a translation transformation tk-1 between the current point cloud frame (Pk) and the previous frame (Pk_1) based on a product of a device linear velocity estimate (Vk-1) of the previous frame (Pk_1) and the time difference (Δt) between the current point cloud frame (Pk) and the previous frame (Pk_1) (Paragraph 0043, "Specifically, the relative transform is converted into special Euclidian group SE(3) space based on a lie algebra exponential function using the time difference between the last pose estimate and a current pose estimate" which includes translations and rotations; Paragraph 0047, "however, it is to be appreciated that the maximum correspondence distance threshold is adjusted based on both the linear and angular velocity of the autonomous vehicle 10 (FIG. 1)" where se(3) considers the maximum correspondence when doing the transform); and transforming the previous frame (Pk_1) into the coordinate system of the current point cloud frame (Pk) using the rotation transformation (Rkl) and the translation transformation tk-1 to provide a transformed previous frame (P’k_1) (Paragraph 0043, "Specifically, the relative transform is converted into special Euclidian group SE(3) space based on a lie algebra exponential function using the time difference between the last pose estimate and a current pose estimate" which includes translations and rotations) wherein the current point is classified as a random point and filtered out if no point in the transformed previous frame (P’k_1) exists within a preset range of the current point in the current point cloud frame (Pk) (Paragraph 0047, "The multi-view non-linear ICP adjustment sub-module 62 then determines the pose graph 100 by computing point-to-point correspondences for each of the neighboring radar point cloud scans 104 and the most recent hyper-local submap 84 based on a maximum correspondence distance threshold.").
Allowable Subject Matter
Claims 4, 5, 9, and 10 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: Claims 4 and 9 recite similar limitations which include: pose graph optimization; relative pose transformation; velocity pre-integration; loop closure detection; optimal pose graph estimation; pose graph registration based on a normal distribution; a relative transformation by matching with a keyframe submap; a radar submap established by a plurality of keyframe point clouds and a sliding window; selecting a new keyframe if a translation or rotation exceeds a threshold; when a number of keyframes exceed the window, discarding the earliest keyframe; dividing the submap into grids and placing a point cloud in a grid; measurement uncertainty of each point considered when computing the mean and covariance of the normal distribution of the point cloud; computing a relative pose transformation that maximizes the distribution probability; calculating an error of the pose transformation; a pose transformation including and additional reliability estimate from the pre-integration velocity; an estimate of linear and angular velocities with true values and zero-mean Gaussian white noise errors; determining a relative transformation (translation/rotation) by integration and an accompanying error. Ali discloses pose graph optimization; relative pose transformation; loop closure detection; optimal pose graph estimation; a relative transformation by matching with a keyframe submap; selecting a new keyframe if a translation or rotation exceeds a threshold; a pose transformation computed based on the estimated velocity. Ali does not disclose the other limitations. Yang discloses velocity pre-integration; and pose graph registration based on a normal distribution; a pose transformation including and additional reliability estimate from the pre-integration velocity; but not the other limitations. Yuan discloses a radar submap established by a plurality of keyframe point clouds and a sliding window; but does not disclose the other limitations such as the pre-integration or any of the error or uncertainty limitations or the covariance or grid limitations. Zhong discloses when a number of frames exceed the window, discarding the earliest frame; but does not disclose a sliding window of keyframes nor any of the errors, uncertainties, covariances, nor the limitations such as the pre-integration. Yan discloses dividing the submap into grids and placing a point cloud in a grid; but doesn’t disclose the other limitations such as the pre-integration or sliding window or any of the errors, uncertainties, or the covariances. None of the references mentioned here disclose the other limitations of: measurement uncertainty of each point considered when computing the mean and covariance of the normal distribution of the point cloud; computing a relative pose transformation that maximizes the distribution (normal/gaussian) probability; calculating an error of the pose transformation; an estimate of linear and angular velocities with true values and zero-mean Gaussian white noise errors; determining a relative transformation (translation/rotation) by integration and an accompanying error. Moreover, even assuming arguendo that the features of the claims exist individually, the combination of features as claimed would not have been obvious to one of ordinary skill in the art because any combination of the evidence obtained to reach the combination of features as claimed would require a substantial reconstruction of Applicant’s claimed invention relying on improper hindsight bias. As such, claims 4 and 9, and their dependents 5 and 10, would be allowable if they weren’t dependent on a rejected claim.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PETER D DOZE whose telephone number is (571)272-0392. The examiner can normally be reached Monday-Friday 9:00am - 6:00pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Resha Desai can be reached at (571) 270-7792. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PETER DAVON DOZE/Examiner, Art Unit 3648
/RESHA DESAI/Supervisory Patent Examiner, Art Unit 3648
Prosecution Insights