DETAILED ACTION
This communication is a Non-Final Office Action on the Merits. Claims 1-20 as originally filed are currently pending and have been considered as follows:
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The lengthy specification has not been checked to the extent necessary to determine the
presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of
which applicant may become aware in the specification.
The specification is objected to because of the following informalities:
“After the actual mounting positions of the at least two 2D laser sensors are obtained, the mounting position error of the at least two 2D laser sensors is obtained by subtracting the actual mounting positions of the at least two 2D laser sensors.” in ¶40 appears incomplete or incorrect. A mounting position error would normally be obtained by comparing/subtracting actual mounting positions from preset/registered mounting positions, not merely “subtracting the actual mounting positions.”
“Perform calculation according to a navigation QR code and kinematic data and based on an extended Kalman filter,” in ¶45 should read “Perform calculation according to a navigation QR code and pose data and based on an extended Kalman filter,”
“calculation may be performed according to the navigation QR code and the kinematic data and based on the extended Kalman filter,” in ¶46 should read “calculation may be performed according to the navigation QR code and the pose data and based on the extended Kalman filter,”
“according to the navigation QR code and the kinematic data obtained by the motion sensor” in ¶52 and ¶60 should read “according to the navigation QR code and the pose data obtained by the motion sensor”
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 3, 11, and 19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim(s) 3 recite “performing pose correction and registering on two frames of point clouds…”. Claims 11 and 19 similarly recite: “perform pose correction and registering on two frames of point clouds…”. The phrase “performing/perform pose correction and registering on two frames of point clouds” is unclear because it is not apparent whether the claim requires pose correction and point-cloud registration as separate operations, a single combined operation, registration between two point-cloud frames, or registration performed “on” some unspecified object. Accordingly, the metes and bounds of the online calibration step are unclear. Therefore, claim(s) 3, 11, and 19 are rejected.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
1. A robot position determination method, comprising:
obtaining laser point cloud data by using a laser sensor carried by a robot;
obtaining pose data of the robot by using a motion sensor carried by the robot;
performing calculation according to a navigation QR code and the pose data and based on an extended Kalman filter, to obtain a prior pose of the robot;
matching the laser point cloud data with a robot map based on the prior pose of the robot, to obtain a first pose of the robot; and
fusing the first pose of the robot with a second pose currently outputted by the extended Kalman filter, to obtain a final pose of the robot.
101 Analysis - Step 1: Statutory category – Yes
The claims recites a method including at least one step. The claims falls within one of the four statutory categories. MPEP 2106.03
Step 2A Prong one evaluation: Judicial Exception – Yes – Mental processes
Claim(s) is to be analyzed to determine whether it recites subject matter that falls within one of the following groups of abstract ideas: a) mathematical concepts, b) mental processes, and/or c) certain methods of organizing human activity.
The Office submits that the foregoing bolded limitation(s) constitutes judicial exceptions in terms of “mental processes” because under its broadest reasonable interpretation, the claim covers performance using mental processes.
The Office submits that the foregoing bolded limitation(s) constitutes judicial exceptions in terms of “mental processes” because under its broadest reasonable interpretation, the claim covers performance using mental processes.
The claims recite the limitations “obtaining…”, “performing calculation…”, and “matching…”. The “obtaining…”, “performing calculation…”, and “matching…” limitations, as drafted, are processes that, under their broadest reasonable interpretation, cover performance of the limitation in the mind but for the recitation of the bolded limitations, nothing in the claims precludes the step from practically being performed in the mind. For example, but for the underlined portions, such as “laser sensor”, “motion sensor”, and “robot” language, the claim encompasses looking at data collected and forming a simple judgement. The mere nominal recitation of the underlined portions do not take the claim limitations out of the mental process grouping. Thus, the claims recite a mental process.
101 Analysis - Step 2A Prong two evaluation: Practical Application - No
Claim(s) is evaluated whether as a whole it integrates the recited judicial exception into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claims beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”)
The claims recite the additional elements of “laser sensor”, “motion sensor”, and “robot” that performs the “obtaining…”, “performing calculation…”, and “matching…” steps. The steps by the additional elements are recited at a high level of generality and merely automates the steps, therefore acting as a generic computer to perform the abstract idea. The additional elements are claimed generically and is operating in its ordinary capacity and does not use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claims are more than a drafting effort designed to monopolize the exception. The additional limitations are no more than mere instructions to apply the exception using a computer.
Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
The claims are directed to the abstract idea.
Step 2B evaluation: Inventive Concept: - No
The claim(s) are evaluated whether the claim as a whole amount to significantly more than the recited exception, i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claims.
As discussed with respect to Step 2A Prong Two, the additional elements in the claims amount to no more than mere instructions to apply the exception using a generic computer component. The same analysis applies here in 2B, i.e., mere instructions to apply an exception on a generic computer cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B, MPEP 2106.05(f).
Therefore, Claim 1 is ineligible.
Claim 9 is rejected using the same rationale, mutatis mutandis, applied to Claim 1 above, respectively.
17. A non-transitory computer-readable storage medium, storing executable code, wherein when the executable code is executed by a processor of an electronic device, the processor is enabled to:
obtain laser point cloud data by using a laser sensor carried by a robot;
obtain pose data of the robot by using a motion sensor carried by the robot;
perform calculation according to a navigation QR code and the pose data and based on an extended Kalman filter, to obtain a prior pose of the robot;
match the laser point cloud data with a robot map based on the prior pose of the robot, to obtain a first pose of the robot; and
fuse the first pose of the robot with a second pose currently outputted by the extended Kalman filter, to obtain a final pose of the robot;
wherein when the processor is enabled to fuse the first pose of the robot with a second pose currently outputted by the extended Kalman filter, to obtain a final pose of the robot, the processor is enabled to:
calculate a difference between the first pose of the robot and the prior pose of the robot, to obtain a pose deviation; and
calculate a sum of the pose deviation and the second pose currently outputted by the extended Kalman filter, to obtain the final pose of the robot.
101 Analysis - Step 1: Statutory category – Yes
The claims recites a non-transitory computer-readable storage medium including at least one step. The claims falls within one of the four statutory categories. MPEP 2106.03
Step 2A Prong one evaluation: Judicial Exception – Yes – Mental processes
Claim(s) is to be analyzed to determine whether it recites subject matter that falls within one of the following groups of abstract ideas: a) mathematical concepts, b) mental processes, and/or c) certain methods of organizing human activity.
The Office submits that the foregoing bolded limitation(s) constitutes judicial exceptions in terms of “mental processes” because under its broadest reasonable interpretation, the claim covers performance using mental processes.
The Office submits that the foregoing bolded limitation(s) constitutes judicial exceptions in terms of “mental processes” because under its broadest reasonable interpretation, the claim covers performance using mental processes.
The claims recite the limitations “obtain…”, “perform calculation…”, and “match…”. The “obtain…”, “perform calculation…”, and “match…” limitations, as drafted, are processes that, under their broadest reasonable interpretation, cover performance of the limitation in the mind but for the recitation of the bolded limitations, nothing in the claims precludes the step from practically being performed in the mind. For example, but for the underlined portions, such as “laser sensor”, “motion sensor”, and “robot” language, the claim encompasses looking at data collected and forming a simple judgement. The mere nominal recitation of the underlined portions do not take the claim limitations out of the mental process grouping. Thus, the claims recite a mental process.
101 Analysis - Step 2A Prong two evaluation: Practical Application - No
Claim(s) is evaluated whether as a whole it integrates the recited judicial exception into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claims beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”)
The claims recite the additional elements of “laser sensor”, “motion sensor”, and “robot” that performs the “obtain…”, “perform calculation…”, and “match…” steps. The steps by the additional elements are recited at a high level of generality and merely automates the steps, therefore acting as a generic computer to perform the abstract idea. The additional elements are claimed generically and is operating in its ordinary capacity and does not use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claims are more than a drafting effort designed to monopolize the exception. The additional limitations are no more than mere instructions to apply the exception using a computer.
Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
The claims are directed to the abstract idea.
Step 2B evaluation: Inventive Concept: - No
The claim(s) are evaluated whether the claim as a whole amount to significantly more than the recited exception, i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claims.
As discussed with respect to Step 2A Prong Two, the additional elements in the claims amount to no more than mere instructions to apply the exception using a generic computer component. The same analysis applies here in 2B, i.e., mere instructions to apply an exception on a generic computer cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B, MPEP 2106.05(f).
Therefore, Claim 17 is ineligible.
Dependent claim(s) 2-8, 10-16, and 18-20 do not recite any further limitations that cause the claim(s) to be patent eligible. Rather, the limitations of dependent claims are directed toward additional aspects of the judicial exception and/or generic additional elements that do not integrate the judicial exception into a practical application. Claims 2-8, 10-16, and 18-20 recite limitations that are insignificant extra-solution activity as they are nominally or tangentially related to the invention and well-known. Therefore, dependent claim(s) 2-8, 10-16, and 18-20 are not patent eligible under the same rationale as provided for in the rejection of claim(s) 1, 9, and 17.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Myeong (KR Pub. No. 20180076815) in view of Liang (CN Pub. No. 113686340).
As per Claim 1, Myeong discloses of a robot position recognition device, comprising:
obtaining laser point cloud data by using a laser sensor carried by a robot; (as per “a step of generating a map corresponding to the indoor space using the optimized graph; and a step of matching the generated map with data scanned using a laser scanner” in ¶18, as per “The matching unit (154) matches the generated map with the data scanned by the laser scanner. The position of the robot (120) can be estimated based on the matched scanned data and map. Data scanned by the laser scanner can be provided from the robot (120)” in ¶45, as per ¶68)
obtaining pose data of the robot by using a motion sensor carried by the robot; (as per “At least one sensor installed on the robot may include at least one of odometry, a gyroscope, LiDAR (Light Detection And Ranging), or a camera” in ¶20, as per “, a location recognition method according to an embodiment of the present invention obtains information about a plurality of QR markers (110) placed at preset locations in an indoor space (100) and information sensed by at least one sensor installed on a robot (120) (S210)” in ¶48, as per “The graph generation unit (151) generates a graph to estimate the position of the robot (120). For example, the position of the robot (120) can be estimated based on the position information of each of the multiple QR markers (110) and the information sensed by the odometry sensor installed on the robot (120)” in ¶43)
performing calculation according to a navigation QR code and the pose data (as per “according to an embodiment of the present invention comprises: a step of obtaining information about at least one QR marker placed at a preset location in an indoor space and information sensed by at least one sensor installed on the robot; and a step of estimating the position of the robot using the information about the QR marker and the information sensed by the at least one sensor installed on the robot” in ¶14, as per “The step of estimating the position of the robot may include: a step of generating a graph using information about the QR marker and information sensed by at least one sensor installed on the robot as constraints; and a step of optimizing the graph to estimate the position of the robot” in ¶17, as per ¶43)
matching the laser point cloud data with a robot map, to obtain a first pose of the robot; (as per “above estimation unit may include a map generation unit that generates a map corresponding to the indoor space using the above optimized graph, and a matching unit that matches the generated map with data scanned using a laser scanner” in ¶25, as per “The matching unit (154) matches the generated map with the data scanned by the laser scanner. The position of the robot (120) can be estimated based on the matched scanned data and map. Data scanned by the laser scanner can be provided from the robot (120)” in ¶45, as per ¶66)
Myeong fails to expressly disclose:
based on an extended Kalman filter, to obtain a prior pose of the robot;
based on the prior pose of the robot
fusing the first pose of the robot with a second pose currently outputted by the extended Kalman filter, to obtain a final pose of the robot.
Liang discloses of a multi-sensor fusion localization method, comprising:
based on an extended Kalman filter, to obtain a prior pose of the robot; (as per “The SLO-VLP pose of the mobile terminal is obtained based on SLO-VLP, and the SLO-VLP pose is used as the initial pose of the particles in the adaptive Monte Carlo localization algorithm. Based on the initial pose, the AMCL pose is obtained” in ¶6, as per “The SLO-VLP pose is used as the initial pose of the AMCL filter particles. The pose information acquired by the odometer sensor and the lidar data are input into the AMCL filter, and the AMCL pose is obtained by outputting the filter.” in ¶18, as per “Step S3: Use the observed pose obtained in steps S1 and S2 as the input value of the EKF algorithm to correct the mobile terminal pose estimate calculated by the odometer sensor. The methods include: Step S31: Calculate the attitude estimate based on the odometry sensor, which is measured by the odometry inside the robot.” in ¶67-¶68)
based on the prior pose of the robot (as per “The SLO-VLP pose is used as the initial pose of the AMCL filter particles. The pose information acquired by the odometer sensor and the lidar data are input into the AMCL filter, and the AMCL pose is obtained by outputting the filter.” in ¶18, as per “Step S2: Use the SLO-VLP pose obtained in step S1 as the initial pose of the particle in the Adaptive Monte Carlo Localization Algorithm (AMCL) to obtain the position estimated by AMCL… Step S21: Use the SLO-VLP attitude S<sub>t</sub> obtained in step S1 as the initial position of the AMCL filter particles; Step S3: Use the observed pose obtained in steps S1 and S2 as the input value of the EKF algorithm to correct the mobile terminal pose estimate calculated by the odometer sensor” in ¶64-¶67)
fusing the first pose of the robot with a second pose currently outputted by the extended Kalman filter, to obtain a final pose of the robot. (as per “The SLO-VLP pose of the mobile terminal is obtained based on SLO-VLP, and the SLO-VLP pose is used as the initial pose of the particles in the adaptive Monte Carlo localization algorithm. Based on the initial pose, the AMCL pose is obtained. The pose values of the mobile terminal are obtained by using the SLO-VLP pose and the AMCL pose as input values for the EKF algorithm” in Claim 1, as per “Obtaining the pose value of the mobile terminal includes: Pose estimation values are obtained through the odometer sensor of the mobile terminal; Based on the SLO-VLP pose and the AMCL pose, pose observations are obtained, and the pose estimates are corrected using a Kalman filter to obtain the pose value of the mobile terminal.” in Claim 6, as per ¶22-¶23)
In this way, Liang is concerned with improving positioning accuracy by using observed pose information from SLO-VLP and AMCL as EKF inputs to correct an odometer-based mobile terminal pose estimate (as per ¶67-¶68). Like Myeong, Liang is concerned with localization of a mobile robot or terminal using robot-mounted sensor data.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the system(s) of Myeong with the EKF-based multi-sensor fusion of Liang to obtain a prior pose and fuse lidar/map-based localization information with motion-sensor pose information. Such modification also reduces accumulated odometer error and improves the reliability of the final estimated pose (as per ¶67-¶78).
Claim 9 is rejected using the same rationale, mutatis mutandis, applied to Claim 1 above, respectively.
Claim(s) 2 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Myeong (KR Pub. No. 20180076815) in view of Liang (CN Pub. No. 113686340) in further view of Jiang (CN Pub. No. 111121625).
As per Claim 2, the combination of Myeong and Liang teaches or suggests all limitations of Claim 1. Myeong and Liang fail to expressly disclose wherein the laser sensor comprises at least two 2D laser sensors deployed on the robot, and the method further comprises:
calibrating the at least two 2D laser sensors offline or online before the obtaining laser point cloud data by using a laser sensor carried by a robot, to obtain a mounting position error of the at least two 2D laser sensors.
Jiang discloses of calibration of diagonally arranged dual lidar radars, wherein the laser sensor comprises at least two 2D laser sensors deployed on the robot (as per “for relative position calibration of diagonally arranged dual lidar radars” in ¶8, as per “Install two lidars, namely radar A and radar B, on the backpack AGV (Automated Guided Vehicle) to ensure that the AGV scans the 360° area environment at all times during the movement” in ¶9) and the method further comprises:
calibrating the at least two 2D laser sensors offline or online before the obtaining laser point cloud data by using a laser sensor carried by a robot, to obtain a mounting position error of the at least two 2D laser sensors. (as per “Extract the point cloud features of the two walls scanned by radar A and radar B, and obtain two sets of linear feature equations by fitting the least squares method. Then manually rotate the AGV body to ensure that radar A and radar B can scan the wall features of wall 1 and wall 2 at the same time. Perform N rotation actions of the AGV body to obtain N sets of linear feature equations of wall 1 and wall 2 in the coordinate system of radar A and radar B.” in ¶11, as per “Obtain the relative positions of the two lidars based on the average angle deviation and average offset distance” in ¶14, as per “Radar A and Radar B are installed diagonally on the AGV, and the scanning area of each radar is a 270° sector, ensuring that the AGV always scans the 360° area environment during its movement” in ¶58)
In this way, Jiang is concerned with calibrating the relative position of two lidars installed on an AGV body, where two lidars are used so the AGV can scan the surrounding environment in all directions (as per ¶5 and ¶8-¶14). Like Myeong and Liang, Jiang is concerned with mobile robot/AGV localization using lidar data.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the system(s) of Myeong and Liang with the dual-lidar calibration of Jiang to provide calibrated point cloud data from multiple robot-mounted laser sensors. Such modification also improves all-direction environmental sensing and reduces relative mounting error between the lidars before the laser data is used for localization (as per ¶8-¶14).
Claim 10 is rejected using the same rationale, mutatis mutandis, applied to Claim 2 above, respectively.
Claim(s) 3 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Myeong (KR Pub. No. 20180076815) in view of Liang (CN Pub. No. 113686340) in view of Jiang (CN Pub. No. 111121625) in further view of Li (CN Pub. No. 112598757).
As per Claim 3, the combination of Myeong, Liang, and Jiang teaches or suggests all limitations of Claim 2. Myeong, Liang, and Jiang fail to expressly disclose wherein the calibrating the at least two 2D laser sensors online to obtain a mounting position error of the at least two 2D laser sensors comprises:
performing feature point matching on image data obtained by a visual device, to obtain a reprojection error corresponding to a feature point;
performing pose correction and registering on two frames of point clouds in laser point cloud data obtained by any one of the at least two 2D laser sensors, and calculating a relative pose between the two frames of point clouds;
calculating a pose deviation between the two frames of point clouds based on the pose data obtained by the motion sensor; and
performing iterative optimization based on the reprojection error, the relative pose, and the pose deviation for a solution within specified duration, and obtaining actual mounting positions of the at least two 2D laser sensors, to obtain the mounting position error of the at least two 2D laser sensors.
Li discloses of a multi-sensor temporal-space calibration method, comprising:
performing feature point matching on image data obtained by a visual device, to obtain a reprojection error corresponding to a feature point; (as per “The image data is semantically segmented and feature points are extracted. The feature points are matched according to the results of the semantic segmentation to construct a reprojection error equation. The first time deviation between the IMU sensor and the camera is introduced into the reprojection error equation.” in ¶9, as per “Step S103: Perform semantic segmentation and feature point extraction on the image data, match the feature points according to the semantic segmentation results, construct a reprojection error equation, and introduce the first time deviation between the IMU sensor and the camera into the reprojection error equation;” in ¶21)
performing pose correction and registering on two frames of point clouds in laser point cloud data obtained by any one of the at least two 2D laser sensors, and calculating a relative pose between the two frames of point clouds; (as per “Introducing a second time deviation between the IMU sensor and the lidar, the pose of two frames of point cloud data is corrected, the corrected two frames of point cloud data are registered, and the relative pose between the two frames of point cloud data is calculated” in ¶10, as per “Step S104: Introducing a second time deviation between the IMU sensor and the lidar, pose correction is performed on two frames of point cloud data; the corrected two frames of point cloud data are then registered; and the relative pose between the two frames of point cloud data is calculated… Step S1041: First, calculate the motion velocity of the two frames of point cloud and introduce the second time deviation between the IMU sensor and the lidar. Based on the motion velocity and the second time deviation, calculate the true pose of the two frames of point cloud, correct the true pose of the two frames of point cloud, and obtain the true coordinates of each point in each frame of point cloud based on the true pose of each frame of point cloud” in ¶ 56-¶57)
calculating a pose deviation between the two frames of point clouds based on the pose data obtained by the motion sensor; (as per “calculate the first pose of the two frames of images through pre-integration, acquire IMU data between two frames of point clouds, calculate the second pose of the two frames of point clouds through pre-integration, and calculate the pose deviation between the first pose and the second pose” in ¶11)
performing iterative optimization based on the reprojection error, the relative pose, and the pose deviation for a solution within specified duration, and obtaining actual mounting positions of the at least two 2D laser sensors, to obtain the mounting position error of the at least two 2D laser sensors. (as per “A sliding window is set, and the reprojection error equation, the relative pose, and the pose deviation within the sliding window are iteratively optimized to achieve multi-sensor time space calibration” in ¶12, as per “Step S106: Set a sliding window, and perform iterative optimization to solve the reprojection error equation, the relative pose, and the pose deviation within the sliding window to achieve multi-sensor time-space calibration. Specifically, if all data are used for iterative optimization during the iterative optimization process, it will not only increase the computational load of the system, but may even introduce a large error, leading to failure of multi-sensor time-space calibration” in ¶97)
In this way, Li is concerned with multi-sensor time-space calibration using camera image data, LiDAR point cloud data, and IMU data, including reprojection error, point-cloud relative pose, and pose deviation constraints (as per ¶8-¶12). Like Myeong, Liang, and Jiang, Li is concerned with localization or detection systems using lidar and motion-sensor data.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the system(s) of Myeong, Liang, and Jiang with the online calibration technique of Li to calibrate sensor mounting relationships using image feature matching, lidar point-cloud registration, and IMU-based pose deviation. Such modification also maintains sensor calibration during operation and improves multi-sensor fusion and positioning accuracy (as per ¶20).
Claim 11 is rejected using the same rationale, mutatis mutandis, applied to Claim 3 above, respectively.
Claim(s) 4 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Myeong (KR Pub. No. 20180076815) in view of Liang (CN Pub. No. 113686340) in further view of Li (CN Pub. No. 112598757).
As per Claim 4, the combination of Myeong and Liang teaches or suggests all limitations of Claim 1. Myeong and Liang fail to expressly disclose aligning, after the obtaining laser point cloud data by using a laser sensor carried by a robot and the obtaining pose data of the robot by using a motion sensor carried by the robot, the laser point cloud data with the pose data of the robot temporally.
Li discloses of a multi-sensor temporal-space calibration method, comprising:
aligning, after the obtaining laser point cloud data (as per “IMU data is acquired through an IMU sensor, image data is acquired through a camera, and point cloud data is acquired through a LiDAR” in ¶8) by using a laser sensor carried by a robot and the obtaining pose data of the robot by using a motion sensor carried by the robot, the laser point cloud data with the pose data of the robot temporally. (as per “The image data is semantically segmented and feature points are extracted. The feature points are matched according to the results of the semantic segmentation to construct a reprojection error equation. The first time deviation between the IMU sensor and the camera is introduced into the reprojection error equation… Introducing a second time deviation between the IMU sensor and the lidar, the pose of two frames of point cloud data is corrected, the corrected two frames of point cloud data are registered, and the relative pose between the two frames of point cloud data is calculated” in ¶9-¶10, as per “Because detection systems inevitably suffer from issues such as trigger delay, transmission delay, and clock asynchrony, each sensor... There will inevitably be a time offset between the sensors. In order to improve the effectiveness of multi-sensor data fusion, it is necessary to ensure the time offset between each sensor)” in n0019)
In this way, Li is concerned with multi-sensor time-space calibration using camera image data, LiDAR point cloud data, and IMU data, including reprojection error, point-cloud relative pose, and pose deviation constraints (as per ¶8-¶12). Like Myeong and Liang, Li is concerned with localization or detection systems using lidar and motion-sensor data.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the system(s) of Myeong and Liang with the temporal alignment technique of Li to align LiDAR point cloud data with motion-sensor pose data. Such modification also improves the effectiveness of multi-sensor data fusion by reducing timing mismatch between the sensors (as per n0019 and ¶20).
Claim 12 is rejected using the same rationale, mutatis mutandis, applied to Claim 4 above, respectively.
Claim(s) 5 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Myeong (KR Pub. No. 20180076815) in view of Liang (CN Pub. No. 113686340) in view of Li (CN Pub. No. 112598757) in further view of Dong (CN Pub. No. 112083433).
As per Claim 5, the combination of Myeong, Liang, and Li teaches or suggests all limitations of Claim 4. Myeong, Liang, and Li fail to expressly disclose wherein the aligning the laser point cloud data with the pose data of the robot temporally comprises:
aligning timestamps of the laser point cloud data and the pose data of the robot by using a linear interpolation algorithm.
Dong discloses a LiDAR distortion correction method for a two-wheeled mobile robot equipped with a LiDAR, an odometer, and an IMU, comprising:
aligning timestamps of the laser point cloud data and the pose data of the robot by using a linear interpolation algorithm. (as per “Obtain the original pose data of each laser point in a frame of lidar, where the time of the starting laser point in the lidar frame is t<sub>s</sub> and the time of the last laser point is t<sub>e</sub>.” in ¶11, as per “Obtain the first pose data of the odometer within the first time period, where the start time of the first time period is t<sub>a</sub>, the end time is t<sub>b</sub>, and t<sub>a</sub> < t<sub>s</sub> < t<sub>e</sub> < t<sub>b</sub>.” in ¶12, as per “Let t<sub>g</sub> < t<sub>s</sub> < t<sub>p</sub>, and there is no corresponding compensation data at time t<sub>s</sub> that is equal to the data after the first distortion correction. There are corresponding compensation data at times t<sub>g</sub> and t<sub>p</sub> that are equal to the data after the first distortion correction.” in ¶35, as per “Linear interpolation is used to insert <img file="FDA0002594155970000039.JPG" frnum="0001" he="99" id="0055" img-content="drawing" img-format="tif" inline="no" orientation="portrait" pgnum="0055" wi="53"/> data points between two adjacent interpolation points among these 5 interpolation points. All the data generated after interpolation is stored in the queue Lidarlist3 as the final distortion-corrected data of the LiDAR, thus completing the secondary distortion correction.” in claim 7)
In this way, Dong is concerned with correcting LiDAR distortion for a mobile robot using LiDAR, odometer, and IMU data, including LiDAR frame timing and odometer pose timing (as per ¶11-¶12). Like Myeong, Liang, and Li, Dong is concerned with mobile robot localization using lidar data and robot pose data.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the system(s) of Myeong, Liang, and Li with the linear-interpolation alignment technique of Dong to align LiDAR scan timestamps with robot pose timestamps. Such modification also provides corrected LiDAR data for robot mapping or localization by interpolating pose/compensation data corresponding to LiDAR scan times (as per ¶35 and claim 7).
Claim 13 is rejected using the same rationale, mutatis mutandis, applied to Claim 5 above, respectively.
Claim(s) 6 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Myeong (KR Pub. No. 20180076815) in view of Liang (CN Pub. No. 113686340) in further view of Zhao (CN Pub. No. 113168717).
As per Claim 6, the combination of Myeong and Liang teaches or suggests all limitations of Claim 1. Myeong and Liang fail to expressly disclose eliminating, before the matching the laser point cloud data with a robot map based on the prior pose of the robot, laser point cloud data corresponding to a movable target from the laser point cloud data by using a data association algorithm.
Zhao discloses a point cloud matching method, comprising:
eliminating, before the matching the laser point cloud data with a robot map based on the prior pose of the robot, laser point cloud data corresponding to a movable target from the laser point cloud data by using a data association algorithm. (as per “Based on the similarity of point cloud clusters, the point cloud clusters of each object in the second frame point cloud are associated with the point cloud clusters of each object in the first frame point cloud.” in ¶9, as per “In one possible implementation of the first aspect, prior to performing point cloud matching, the method further includes: Filter out point cloud clusters corresponding to dynamic objects in the first frame point cloud; and/or Filter out the point cloud clusters corresponding to dynamic objects in the second frame point cloud.” in ¶29-¶31, as per “Before performing point cloud matching, the velocity of the point cloud clusters of the same object in consecutive frames of point clouds can be observed and estimated to determine whether the object is a dynamic object. The point cloud clusters of the corresponding dynamic objects can then be filtered out from each frame of point clouds to eliminate the influence of dynamic objects on point cloud matching.” in ¶32, as per “S603: Filter out dynamic objects. Based on the correlation results, filter out dynamic objects in the point cloud of frame T1 and the point cloud of frame T2.” in ¶161)
In this way, Zhao is concerned with improving point cloud matching in dynamic environments by associating point cloud clusters between consecutive frames and filtering out dynamic-object clusters before point cloud matching (as per ¶29-¶32). Like Myeong and Liang, Zhao is concerned with lidar-based localization or SLAM using point cloud matching.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the system(s) of Myeong and Liang with the dynamic-object filtering of Zhao to eliminate movable-target point cloud data before matching laser point cloud data with a map. Such modification also reduces the influence of moving objects on point cloud matching and improves the accuracy of the resulting pose transformation (as per ¶32 and ¶161).
Claim 14 is rejected using the same rationale, mutatis mutandis, applied to Claim 6 above, respectively.
Claim(s) 7 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Myeong (KR Pub. No. 20180076815) in view of Liang (CN Pub. No. 113686340) in further view of Xie (CN Pub. No. 105547305).
As per Claim 7, the combination of Myeong and Liang teaches or suggests all limitations of Claim 1. Myeong fails to expressly disclose wherein the robot map is a two-dimensional grid map, and the matching the laser point cloud data with a robot map based on the prior pose of the robot, to obtain a first pose of the robot comprises:
determining a plurality of candidate poses in pose search space based on the prior pose of the robot;
projecting the laser point cloud data to the two-dimensional grid map based on each of the plurality of candidate poses, and calculating a matching score of each candidate pose on the two-dimensional grid map; and
determining a candidate pose with a highest matching score in the plurality of candidate poses on the two-dimensional grid map as the first pose of the robot.
See Claim 1 for teachings of Liang. Liang further discloses:
the matching the laser point cloud data with a robot map based on the prior pose of the robot, (as per “The SLO-VLP pose is used as the initial pose of the AMCL filter particles. The pose information acquired by the odometer sensor and the lidar data are input into the AMCL filter, and the AMCL pose is obtained by outputting the filter.” in ¶18, as per “Step S2: Use the SLO-VLP pose obtained in step S1 as the initial pose of the particle in the Adaptive Monte Carlo Localization Algorithm (AMCL) to obtain the position estimated by AMCL… Step S21: Use the SLO-VLP attitude S<sub>t</sub> obtained in step S1 as the initial position of the AMCL filter particles… Step S3: Use the observed pose obtained in steps S1 and S2 as the input value of the EKF algorithm to correct the mobile terminal pose estimate calculated by the odometer sensor” in ¶64-¶67)
In this way, Liang is concerned with improving positioning accuracy by using observed pose information from SLO-VLP and AMCL as EKF inputs to correct an odometer-based mobile terminal pose estimate (as per ¶67-¶68). Like Myeong, Liang is concerned with localization of a mobile robot or terminal using robot-mounted sensor data.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the system(s) of Myeong with the EKF-based multi-sensor fusion of Liang to obtain a prior pose and fuse lidar/map-based localization information with motion-sensor pose information. Such modification also reduces accumulated odometer error and improves the reliability of the final estimated pose (as per ¶67-¶78).
Myeong and Liang fail to expressly disclose wherein the robot map is a two-dimensional grid map to obtain a first pose of the robot comprises:
determining a plurality of candidate poses in pose search space based on the prior pose of the robot;
projecting the laser point cloud data to the two-dimensional grid map based on each of the plurality of candidate poses, and calculating a matching score of each candidate pose on the two-dimensional grid map; and
determining a candidate pose with a highest matching score in the plurality of candidate poses on the two-dimensional grid map as the first pose of the robot.
Xie discloses of a pose calculation method based on wireless positioning and laser map matching, wherein the robot map is a two-dimensional grid map, (as per “Define the map coordinate system A<sub> m </sub>: with the bottom left corner of the map as the origin, the horizontal direction to the right as the x-axis, and the vertical direction upward as the y-axis” in ¶45) to obtain a first pose of the robot comprises:
determining a plurality of candidate poses in pose search space based on the prior pose of the robot; (as per “Based on the installation location of the wireless positioning tag on the positioning device and the magnitude of the wireless positioning error, estimate the candidate location area of the positioning device,” in ¶17, as per “When calculating candidate locations based on a global map, all unobstructed pixels in the map are generally considered as candidate locations. To improve the accuracy of simulation data calculations and reduce the number of candidate location points, the map can be preprocessed as follows: the boundary lines of obstacles are expanded, and pixels that are not expected to be candidate location points are changed to uncertain environmental values, i.e., grayscale pixels” in ¶81)
projecting the laser point cloud data to the two-dimensional grid map based on each of the plurality of candidate poses, and calculating a matching score of each candidate pose on the two-dimensional grid map; (as per “Measure a frame of laser scan data. At each unobstructed pixel in the candidate location area, match the measured laser scan data with the simulated laser scan data at that pixel to obtain the yaw angle and matching distance” in ¶18, as per “find the candidate positioning points in the global map, and calculate the simulated laser scanning data of each positioning point through simulation, which is then used for map matching and positioning” in ¶24, as per ¶84)
determining a candidate pose with a highest matching score in the plurality of candidate poses on the two-dimensional grid map as the first pose of the robot. (as per “Select the pixel with the smallest matching distance of the laser scanning data. Its corresponding position is the current position of the positioning device. The yaw angle obtained by matching the laser scanning data at this pixel is the yaw angle of the positioning device” in ¶19, as per “the preferred method is to determine the current position pixel and yaw angle of the positioning device by calculating the minimum average distance difference between the laser scan data and the simulated laser data of each pixel in the candidate position area under each yaw angle condition” in ¶26, as per ¶95-¶97)
In this way, Xie is concerned with laser map matching by determining candidate locations in a pixel-based global map, matching measured laser scan data with simulated laser scan data at the candidate locations, and selecting the candidate having the best match (as per ¶17-¶19). Like Myeong and Liang, Xie is concerned with localization using laser scan data and map matching.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the system(s) of Myeong and Liang with the candidate-pose search and laser map matching of Xie to evaluate laser scan data over multiple candidate poses in a two-dimensional map. Such modification also reduces the search area and improves pose calculation by selecting the candidate pose having the smallest matching distance, i.e., the best laser-map match (as per ¶24, ¶81, and ¶95-¶96).
Claim 15 is rejected using the same rationale, mutatis mutandis, applied to Claim 7 above, respectively.
Claim(s) 8 and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Myeong (KR Pub. No. 20180076815) in view of Liang (CN Pub. No. 113686340) in further view of Jeong (US Pat. No. 11859994).
As per Claim 8, the combination of Myeong and Liang teaches or suggests all limitations of Claim 1. Myeong and Liang fail to expressly disclose wherein the fusing the first pose of the robot with a second pose currently outputted by the extended Kalman filter, to obtain a final pose of the robot comprises:
calculating a difference between the first pose of the robot and the prior pose of the robot, to obtain a pose deviation;
calculating a sum of the pose deviation and the second pose currently outputted by the extended Kalman filter, to obtain the final pose of the robot.
Jeong discloses of landmark-based localization methods and architectures for an autonomous vehicle, comprising:
calculating a difference between the first pose of the robot and the prior pose of the robot, to obtain a pose deviation; (as per “Moreover, landmark module 254 can process the local pose instance 292A generated by local pose module 292 and a stored mapping of an environment of vehicle 100 (e.g., in stored mapping(s) database 160A) to generate a pose-based predicted location 254B of the landmark in the surrounding environment of vehicle 100. The local pose instance 292A can provide landmark module 254 with information that indicates a tile that vehicle 100 is located. Landmark module 254 can identify the stored mapping of the environment of vehicle 100 based on the information that indicates the tile that vehicle 100 is located” in C25L1-15, as per “Online calibration module 256 can generate a correction instance(s) 256A based on comparing the first predicted location (e.g., LIDAR-based predicted location) of the landmark and the second predicted location (e.g., pose-based predicted location) of the landmark. More particularly, online calibration module 256 can compare the predicted locations to determine an error in the determined local pose instance of vehicle 100 based on a difference between the first predicted location and the second predicted location from the comparing” in C18L25-45)
calculating a sum of the pose deviation and the second pose currently outputted by the extended Kalman filter, to obtain the final pose of the robot. (as per “Local pose module 292 can include in propagated filter(s) that incorporates the most recent version of sensor data in instances of the second sensor data (i.e., anytime machinery). Further, local pose module 292 can receive a correction instance(s) 256A generated by online calibration module 256 as described above with respect to FIG. 2A. Moreover, local pose module 292 can process, using a state estimation model that is filter-based (e.g., Kalman filter, extended Kalman filter, dual Kalman filter, or other filter-based techniques) or observer-based (e.g., recursive least squares or other observer-based techniques), the instance of the second sensor data (including IMU data 182A and wheel encoder data 184A) and optionally the correction instance(s) 256A to generate output” in C24L5-25, as per “the system generates a correction instance based on comparing the pose-based predicted location and the LIDAR-based predicted location. The system can determine an offset based on the difference between the pose-based predicted location and the LIDAR-based predicted location, and can generate the correction instance based on the determined offset” in C32L10-25, as per “Thus, the additional local pose instance(s) are generated based on the determined offset. Notably, multiple additional pose instances can be generated based on the generated correction instance until a further correction instance is generated. The correction instance and the further correction instance can be combined, and further local pose instances can be generated based on the combined correction instance” in C32L25-40)
In this way, Jeong is concerned with correcting vehicle localization by comparing a LIDAR-based predicted landmark location with a pose-based predicted landmark location, determining a difference, and generating a correction offset based on that difference (as per C18L25-45 and C32L10-25). Like Myeong and Liang, Jeong is concerned with localization using lidar data, map information, and motion-sensor-based pose estimation.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the system(s) of Myeong and Liang with the correction-offset technique of Jeong to calculate a pose deviation from a difference between lidar/map-based localization and pose-based localization and apply that deviation to a filter-based pose estimate. Such modification also improves the accuracy of subsequent local pose instances used for vehicle control (as per C24L5-25 and C32L25-40).
Claim 16 is rejected using the same rationale, mutatis mutandis, applied to Claim 8 above, respectively.
As per Claim 17, Myeong discloses of a robot position recognition device, comprising:
non-transitory computer-readable storage medium, storing executable code, (as per “Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory” in ¶71)
obtain laser point cloud data by using a laser sensor carried by a robot; (as per “a step of generating a map corresponding to the indoor space using the optimized graph; and a step of matching the generated map with data scanned using a laser scanner” in ¶18, as per “The matching unit (154) matches the generated map with the data scanned by the laser scanner. The position of the robot (120) can be estimated based on the matched scanned data and map. Data scanned by the laser scanner can be provided from the robot (120)” in ¶45, as per ¶68)
obtain pose data of the robot by using a motion sensor carried by the robot; (as per “At least one sensor installed on the robot may include at least one of odometry, a gyroscope, LiDAR (Light Detection And Ranging), or a camera” in ¶20, as per “, a location recognition method according to an embodiment of the present invention obtains information about a plurality of QR markers (110) placed at preset locations in an indoor space (100) and information sensed by at least one sensor installed on a robot (120) (S210)” in ¶48, as per “The graph generation unit (151) generates a graph to estimate the position of the robot (120). For example, the position of the robot (120) can be estimated based on the position information of each of the multiple QR markers (110) and the information sensed by the odometry sensor installed on the robot (120)” in ¶43)
perform calculation according to a navigation QR code and the pose data (as per “according to an embodiment of the present invention comprises: a step of obtaining information about at least one QR marker placed at a preset location in an indoor space and information sensed by at least one sensor installed on the robot; and a step of estimating the position of the robot using the information about the QR marker and the information sensed by the at least one sensor installed on the robot” in ¶14, as per “The step of estimating the position of the robot may include: a step of generating a graph using information about the QR marker and information sensed by at least one sensor installed on the robot as constraints; and a step of optimizing the graph to estimate the position of the robot” in ¶17, as per ¶43)
match the laser point cloud data with a robot map, to obtain a first pose of the robot; (as per “above estimation unit may include a map generation unit that generates a map corresponding to the indoor space using the above optimized graph, and a matching unit that matches the generated map with data scanned using a laser scanner” in ¶25, as per “The matching unit (154) matches the generated map with the data scanned by the laser scanner. The position of the robot (120) can be estimated based on the matched scanned data and map. Data scanned by the laser scanner can be provided from the robot (120)” in ¶45, as per ¶66)
Myeong fails to expressly disclose:
based on an extended Kalman filter, to obtain a prior pose of the robot;
based on the prior pose of the robot
fuse the first pose of the robot with a second pose currently outputted by the extended Kalman filter, to obtain a final pose of the robot;
wherein when the processor is enabled to fuse the first pose of the robot with a second pose currently outputted by the extended Kalman filter, to obtain a final pose of the robot,
calculate a difference between the first pose of the robot and the prior pose of the robot, to obtain a pose deviation; and
calculate a sum of the pose deviation and the second pose currently outputted by the extended Kalman filter, to obtain the final pose of the robot.
Liang discloses of a multi-sensor fusion localization method, comprising:
based on an extended Kalman filter, to obtain a prior pose of the robot; (as per “The SLO-VLP pose of the mobile terminal is obtained based on SLO-VLP, and the SLO-VLP pose is used as the initial pose of the particles in the adaptive Monte Carlo localization algorithm. Based on the initial pose, the AMCL pose is obtained” in ¶6, as per “The SLO-VLP pose is used as the initial pose of the AMCL filter particles. The pose information acquired by the odometer sensor and the lidar data are input into the AMCL filter, and the AMCL pose is obtained by outputting the filter.” in ¶18, as per “Step S3: Use the observed pose obtained in steps S1 and S2 as the input value of the EKF algorithm to correct the mobile terminal pose estimate calculated by the odometer sensor. The methods include: Step S31: Calculate the attitude estimate based on the odometry sensor, which is measured by the odometry inside the robot.” in ¶67-¶68)
based on the prior pose of the robot (as per “The SLO-VLP pose is used as the initial pose of the AMCL filter particles. The pose information acquired by the odometer sensor and the lidar data are input into the AMCL filter, and the AMCL pose is obtained by outputting the filter.” in ¶18, as per “Step S2: Use the SLO-VLP pose obtained in step S1 as the initial pose of the particle in the Adaptive Monte Carlo Localization Algorithm (AMCL) to obtain the position estimated by AMCL… Step S21: Use the SLO-VLP attitude S<sub>t</sub> obtained in step S1 as the initial position of the AMCL filter particles; Step S3: Use the observed pose obtained in steps S1 and S2 as the input value of the EKF algorithm to correct the mobile terminal pose estimate calculated by the odometer sensor” in ¶64-¶67)
fuse the first pose of the robot with a second pose currently outputted by the extended Kalman filter, to obtain a final pose of the robot; (as per “The SLO-VLP pose of the mobile terminal is obtained based on SLO-VLP, and the SLO-VLP pose is used as the initial pose of the particles in the adaptive Monte Carlo localization algorithm. Based on the initial pose, the AMCL pose is obtained. The pose values of the mobile terminal are obtained by using the SLO-VLP pose and the AMCL pose as input values for the EKF algorithm” in Claim 1, as per “Obtaining the pose value of the mobile terminal includes: Pose estimation values are obtained through the odometer sensor of the mobile terminal; Based on the SLO-VLP pose and the AMCL pose, pose observations are obtained, and the pose estimates are corrected using a Kalman filter to obtain the pose value of the mobile terminal.” in Claim 6, as per ¶22-¶23)
wherein when the processor is enabled to fuse the first pose of the robot with a second pose currently outputted by the extended Kalman filter, to obtain a final pose of the robot, (as per “The SLO-VLP pose of the mobile terminal is obtained based on SLO-VLP, and the SLO-VLP pose is used as the initial pose of the particles in the adaptive Monte Carlo localization algorithm. Based on the initial pose, the AMCL pose is obtained. The pose values of the mobile terminal are obtained by using the SLO-VLP pose and the AMCL pose as input values for the EKF algorithm” in Claim 1, as per “Obtaining the pose value of the mobile terminal includes: Pose estimation values are obtained through the odometer sensor of the mobile terminal; Based on the SLO-VLP pose and the AMCL pose, pose observations are obtained, and the pose estimates are corrected using a Kalman filter to obtain the pose value of the mobile terminal.” in Claim 6, as per ¶22-¶23)
In this way, Liang is concerned with improving positioning accuracy by using observed pose information from SLO-VLP and AMCL as EKF inputs to correct an odometer-based mobile terminal pose estimate (as per ¶67-¶68). Like Myeong, Liang is concerned with localization of a mobile robot or terminal using robot-mounted sensor data.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the system(s) of Myeong with the EKF-based multi-sensor fusion of Liang to obtain a prior pose and fuse lidar/map-based localization information with motion-sensor pose information. Such modification also reduces accumulated odometer error and improves the reliability of the final estimated pose (as per ¶67-¶78).
Myeong and Liang fail to expressly disclose:
calculate a difference between the first pose of the robot and the prior pose of the robot, to obtain a pose deviation; and
calculate a sum of the pose deviation and the second pose currently outputted by the extended Kalman filter, to obtain the final pose of the robot.
Jeong discloses of landmark-based localization methods and architectures for an autonomous vehicle, comprising:
calculate a difference between the first pose of the robot and the prior pose of the robot, to obtain a pose deviation; (as per “Moreover, landmark module 254 can process the local pose instance 292A generated by local pose module 292 and a stored mapping of an environment of vehicle 100 (e.g., in stored mapping(s) database 160A) to generate a pose-based predicted location 254B of the landmark in the surrounding environment of vehicle 100. The local pose instance 292A can provide landmark module 254 with information that indicates a tile that vehicle 100 is located. Landmark module 254 can identify the stored mapping of the environment of vehicle 100 based on the information that indicates the tile that vehicle 100 is located” in C25L1-15, as per “Online calibration module 256 can generate a correction instance(s) 256A based on comparing the first predicted location (e.g., LIDAR-based predicted location) of the landmark and the second predicted location (e.g., pose-based predicted location) of the landmark. More particularly, online calibration module 256 can compare the predicted locations to determine an error in the determined local pose instance of vehicle 100 based on a difference between the first predicted location and the second predicted location from the comparing” in C18L25-45)
calculate a sum of the pose deviation and the second pose currently outputted by the extended Kalman filter, to obtain the final pose of the robot. (as per “Local pose module 292 can include in propagated filter(s) that incorporates the most recent version of sensor data in instances of the second sensor data (i.e., anytime machinery). Further, local pose module 292 can receive a correction instance(s) 256A generated by online calibration module 256 as described above with respect to FIG. 2A. Moreover, local pose module 292 can process, using a state estimation model that is filter-based (e.g., Kalman filter, extended Kalman filter, dual Kalman filter, or other filter-based techniques) or observer-based (e.g., recursive least squares or other observer-based techniques), the instance of the second sensor data (including IMU data 182A and wheel encoder data 184A) and optionally the correction instance(s) 256A to generate output” in C24L5-25, as per “the system generates a correction instance based on comparing the pose-based predicted location and the LIDAR-based predicted location. The system can determine an offset based on the difference between the pose-based predicted location and the LIDAR-based predicted location, and can generate the correction instance based on the determined offset” in C32L10-25, as per “Thus, the additional local pose instance(s) are generated based on the determined offset. Notably, multiple additional pose instances can be generated based on the generated correction instance until a further correction instance is generated. The correction instance and the further correction instance can be combined, and further local pose instances can be generated based on the combined correction instance” in C32L25-40)
In this way, Jeong is concerned with correcting vehicle localization by comparing a LIDAR-based predicted landmark location with a pose-based predicted landmark location, determining a difference, and generating a correction offset based on that difference (as per C18L25-45 and C32L10-25). Like Myeong and Liang, Jeong is concerned with localization using lidar data, map information, and motion-sensor-based pose estimation.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the system(s) of Myeong and Liang with the correction-offset technique of Jeong to calculate a pose deviation from a difference between lidar/map-based localization and pose-based localization and apply that deviation to a filter-based pose estimate. Such modification also improves the accuracy of subsequent local pose instances used for vehicle control (as per C24L5-25 and C32L25-40).
Claim(s) 18 is rejected under 35 U.S.C. 103 as being unpatentable over Myeong (KR Pub. No. 20180076815) in view of Liang (CN Pub. No. 113686340) in view of Jeong (US Pat. No. 11859994) in further view of Jiang (CN Pub. No. 111121625).
As per Claim 18, the combination of Myeong, Liang, and Jeong teaches or suggests all limitations of Claim 17. Myeong, Liang, and Jeong fail to expressly disclose wherein the laser sensor comprises at least two 2D laser sensors deployed on the robot, and the processor is further enabled to:
calibrate the at least two 2D laser sensors offline or online before the obtaining laser point cloud data by using a laser sensor carried by a robot, to obtain a mounting position error of the at least two 2D laser sensors.
Jiang discloses of calibration of diagonally arranged dual lidar radars, wherein the laser sensor comprises at least two 2D laser sensors deployed on the robot (as per “for relative position calibration of diagonally arranged dual lidar radars” in ¶8, as per “Install two lidars, namely radar A and radar B, on the backpack AGV (Automated Guided Vehicle) to ensure that the AGV scans the 360° area environment at all times during the movement” in ¶9), and the processor is further enabled to:
calibrate the at least two 2D laser sensors offline or online before the obtaining laser point cloud data by using a laser sensor carried by a robot, to obtain a mounting position error of the at least two 2D laser sensors. (as per “Extract the point cloud features of the two walls scanned by radar A and radar B, and obtain two sets of linear feature equations by fitting the least squares method. Then manually rotate the AGV body to ensure that radar A and radar B can scan the wall features of wall 1 and wall 2 at the same time. Perform N rotation actions of the AGV body to obtain N sets of linear feature equations of wall 1 and wall 2 in the coordinate system of radar A and radar B.” in ¶11, as per “Obtain the relative positions of the two lidars based on the average angle deviation and average offset distance” in ¶14, as per “Radar A and Radar B are installed diagonally on the AGV, and the scanning area of each radar is a 270° sector, ensuring that the AGV always scans the 360° area environment during its movement” in ¶58)
In this way, Jiang is concerned with calibrating the relative position of two lidars installed on an AGV body, where two lidars are used so the AGV can scan the surrounding environment in all directions (as per ¶5 and ¶8-¶14). Like Myeong, Liang, and Jeong, Jiang is concerned with mobile robot/AGV localization using lidar data.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the system(s) of Myeong, Liang, and Jeong with the dual-lidar calibration of Jiang to provide calibrated point cloud data from multiple robot-mounted laser sensors. Such modification also improves all-direction environmental sensing and reduces relative mounting error between the lidars before the laser data is used for localization (as per ¶8-¶14).
Claim(s) 19 is rejected under 35 U.S.C. 103 as being unpatentable over Myeong (KR Pub. No. 20180076815) in view of Liang (CN Pub. No. 113686340) in view of Jeong (US Pat. No. 11859994) in view of Jiang (CN Pub. No. 111121625) in further view of Li (CN Pub. No. 112598757).
As per Claim 19, the combination of Myeong, Liang, Jeong, and Jiang teaches or suggests all limitations of Claim 18. Myeong, Liang, Jeong, and Jiang fail to expressly disclose wherein when the processor is enabled to calibrate the at least two 2D laser sensors online to obtain a mounting position error of the at least two 2D laser sensors, the processor is enabled to:
perform feature point matching on image data obtained by a visual device, to obtain a reprojection error corresponding to a feature point;
perform pose correction and registering on two frames of point clouds in laser point cloud data obtained by any one of the at least two 2D laser sensors, and calculate a relative pose between the two frames of point clouds;
calculate a pose deviation between the two frames of point clouds based on the pose data obtained by the motion sensor; and
perform iterative optimization based on the reprojection error, the relative pose, and the pose deviation for a solution within specified duration, and obtain actual mounting positions of the at least two 2D laser sensors, to obtain the mounting position error of the at least two 2D laser sensors.
Li discloses of a multi-sensor temporal-space calibration method, comprising:
perform feature point matching on image data obtained by a visual device, to obtain a reprojection error corresponding to a feature point; (as per “The image data is semantically segmented and feature points are extracted. The feature points are matched according to the results of the semantic segmentation to construct a reprojection error equation. The first time deviation between the IMU sensor and the camera is introduced into the reprojection error equation.” in ¶9, as per “Step S103: Perform semantic segmentation and feature point extraction on the image data, match the feature points according to the semantic segmentation results, construct a reprojection error equation, and introduce the first time deviation between the IMU sensor and the camera into the reprojection error equation;” in ¶21)
perform pose correction and registering on two frames of point clouds in laser point cloud data obtained by any one of the at least two 2D laser sensors, and calculate a relative pose between the two frames of point clouds; (as per “Introducing a second time deviation between the IMU sensor and the lidar, the pose of two frames of point cloud data is corrected, the corrected two frames of point cloud data are registered, and the relative pose between the two frames of point cloud data is calculated” in ¶10, as per “Step S104: Introducing a second time deviation between the IMU sensor and the lidar, pose correction is performed on two frames of point cloud data; the corrected two frames of point cloud data are then registered; and the relative pose between the two frames of point cloud data is calculated… Step S1041: First, calculate the motion velocity of the two frames of point cloud and introduce the second time deviation between the IMU sensor and the lidar. Based on the motion velocity and the second time deviation, calculate the true pose of the two frames of point cloud, correct the true pose of the two frames of point cloud, and obtain the true coordinates of each point in each frame of point cloud based on the true pose of each frame of point cloud” in ¶ 56-¶57)
calculate a pose deviation between the two frames of point clouds based on the pose data obtained by the motion sensor; (as per “calculate the first pose of the two frames of images through pre-integration, acquire IMU data between two frames of point clouds, calculate the second pose of the two frames of point clouds through pre-integration, and calculate the pose deviation between the first pose and the second pose” in ¶11)
perform iterative optimization based on the reprojection error, the relative pose, and the pose deviation for a solution within specified duration, and obtain actual mounting positions of the at least two 2D laser sensors, to obtain the mounting position error of the at least two 2D laser sensors. (as per “A sliding window is set, and the reprojection error equation, the relative pose, and the pose deviation within the sliding window are iteratively optimized to achieve multi-sensor time space calibration” in ¶12, as per “Step S106: Set a sliding window, and perform iterative optimization to solve the reprojection error equation, the relative pose, and the pose deviation within the sliding window to achieve multi-sensor time-space calibration. Specifically, if all data are used for iterative optimization during the iterative optimization process, it will not only increase the computational load of the system, but may even introduce a large error, leading to failure of multi-sensor time-space calibration” in ¶97)
In this way, Li is concerned with multi-sensor time-space calibration using camera image data, LiDAR point cloud data, and IMU data, including reprojection error, point-cloud relative pose, and pose deviation constraints (as per ¶8-¶12). Like Myeong, Liang, Jeong, and Jiang, Li is concerned with localization or detection systems using lidar and motion-sensor data.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the system(s) of Myeong, Liang, Jeong, and Jiang with the online calibration technique of Li to calibrate sensor mounting relationships using image feature matching, lidar point-cloud registration, and IMU-based pose deviation. Such modification also maintains sensor calibration during operation and improves multi-sensor fusion and positioning accuracy (as per ¶20).
Claim(s) 20 is rejected under 35 U.S.C. 103 as being unpatentable over Myeong (KR Pub. No. 20180076815) in view of Liang (CN Pub. No. 113686340) in view of Jeong (US Pat. No. 11859994) in further view of Li (CN Pub. No. 112598757).
As per Claim 20, the combination of Myeong, Liang, and Jeong teaches or suggests all limitations of Claim 17. Myeong, Liang, and Jeong fail to expressly disclose wherein the processor is enabled to:
align, after the processor obtains laser point cloud data by using a laser sensor carried by a robot and obtains pose data of the robot by using a motion sensor carried by the robot, the laser point cloud data with the pose data of the robot temporally.
Li discloses of a multi-sensor temporal-space calibration method, comprising:
align, after the processor obtains laser point cloud data (as per “IMU data is acquired through an IMU sensor, image data is acquired through a camera, and point cloud data is acquired through a LiDAR” in ¶8) by using a laser sensor carried by a robot and obtains pose data of the robot by using a motion sensor carried by the robot, the laser point cloud data with the pose data of the robot temporally. (as per “The image data is semantically segmented and feature points are extracted. The feature points are matched according to the results of the semantic segmentation to construct a reprojection error equation. The first time deviation between the IMU sensor and the camera is introduced into the reprojection error equation… Introducing a second time deviation between the IMU sensor and the lidar, the pose of two frames of point cloud data is corrected, the corrected two frames of point cloud data are registered, and the relative pose between the two frames of point cloud data is calculated” in ¶9-¶10, as per “Because detection systems inevitably suffer from issues such as trigger delay, transmission delay, and clock asynchrony, each sensor... There will inevitably be a time offset between the sensors. In order to improve the effectiveness of multi-sensor data fusion, it is necessary to ensure the time offset between each sensor)” in n0019)
In this way, Li is concerned with multi-sensor time-space calibration using camera image data, LiDAR point cloud data, and IMU data, including reprojection error, point-cloud relative pose, and pose deviation constraints (as per ¶8-¶12). Like Myeong, Liang, and Jeong, Li is concerned with localization or detection systems using lidar and motion-sensor data.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the system(s) of Myeong, Liang, and Jeong with the temporal alignment technique of Li to align LiDAR point cloud data with motion-sensor pose data. Such modification also improves the effectiveness of multi-sensor data fusion by reducing timing mismatch between the sensors (as per n0019 and ¶20).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Li (CN Pub. No. 112254729) discloses a mobile robot positioning method based on multi-sensor fusion.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TYLER R ROBARGE whose telephone number is (703)756-5872. The examiner can normally be reached Monday - Friday, 8:00 am - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramon Mercado can be reached on (571) 270-5744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.R.R./Examiner, Art Unit 3658
/Ramon A. Mercado/Supervisory Patent Examiner, Art Unit 3658
Prosecution Insights