DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 10/03/2024 and 01/14/2025 are being considered by the examiner.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description: 410, 415, 412, 413, 416. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either "Replacement Sheet" or "New Sheet" pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Claims 6, 13, and 19 recite limitations that use words like “means” (or “step”) or similar terms with functional language and do invoke 35 U.S.C. 112(f):
Claim 6; recites the limitation, “using a perspective-n-point module…..” [Line 3-4].
Claim 13; recites the limitation, “using a perspective-n-point module…..” [Line 3-4].
Claim 19; recites the limitation, “using a perspective-n-point module…..” [Line 4].
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
After a careful analysis, as disclosed above, and a careful review of the specification the following limitations in claims 6, 13, and 19;
(i) “perspective-n-point module” (Paragraph [0020, 0036 and 0057]- The present embodiments then leverage the association between near-infrared image and LiDAR point cloud to establish 2D-3D correspondence, whereby the extrinsic calibration can then be estimated using perspective-n-point (PnP) optimization. The reprojection error can be the Euclidean distance between an observed point from the filtered NIR-to-3D points and a reprojected point. A reprojected point can be generated by using 2D points from the filtered NIR-to-3D points and an estimated camera pose computed using the PnP module. The optimized extrinsic calibration can include an extrinsic calibration having the highest number of data points between 3D points and two-dimensional (2D) points which can be determined iteratively until a threshold has been met. An extrinsic calibration can be optimized based on a reprojection error computed from the filtered NIR-to-3D points using a sampling module 347 that can implement RANSAC with PnP to obtain the optimized extrinsic calibration 350. The perspective-n-point module thus does not have sufficient structure or material.).
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 6, 13, and 19 along with their dependent claims are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claims 6, 13, and 19 limitations:
Claim 6; recites the limitation, “using a perspective-n-point module…..” [Line 3-4].
Claim 13; recites the limitation, “using a perspective-n-point module…..” [Line 3-4].
Claim 19; recites the limitation, “using a perspective-n-point module…..” [Line 4].
Claims 6, 13, and 19 respectively invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. The specification is devoid of adequate structure to perform the claimed functions. The specification does not provide sufficient details such that one of the ordinary skill in the art would understand which structure performed(s) the claimed function.
Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 6, 13, and 19 along with their dependent claims are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for pre-AIA the inventor(s), at the time the application was filed, had possession of the claimed invention. As described above, the disclosure does not provide adequate structure to perform the claimed function in the recited limitation.
Claim 6; recites the limitation, “using a perspective-n-point module…..” [Line 3-4].
Claim 13; recites the limitation, “using a perspective-n-point module…..” [Line 3-4].
Claim 19; recites the limitation, “using a perspective-n-point module…..” [Line 4].
The specification does not demonstrate that applicant has made an invention that achieves the claimed function because the invention is not described with sufficient detail such that one of ordinary skill in the art can reasonably conclude that the inventor had possession of the claimed invention.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 7-8, 10, 14-15, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over ZSIROS et al. (US 20250363666 A1), hereinafter referenced as ZSIROS, in view of TANG et al. (US 20210326601 A1), hereinafter referenced as TANG, and further in view of KROEGER (US 10298910 B1), hereinafter referenced as KROEGER.
Regarding claim 1, ZSIROS explicitly teaches a computer-implemented method (Fig. 1. Paragraph a further aspect of the invention refers to a computer-readable storage medium storing program code, the program code comprising instructions that when executed by a processor carry out the method of the second aspect or one of the implementations of the first aspect.) for automatic multi-modality sensor calibration with near-infrared images (NIR) (Fig. 1. Paragraph [0054-0059]-ZSIROS discloses during an automatic extrinsic calibration process, in the following just referred to as calibration, the following data may be used: Regular camera frames (2D) or the depth image created from stereo cameras (3D). Radar point cloud (2D/3D) Lidar point cloud (3D) The sensor to be calibrated may provide additional data, e.g., the velocity, intensity, cross-section size and/or confidence of detections. These may be used for aiding the calibration process.), comprising:
detecting image keypoints from collected images (Fig. 1, illustrates detected image keypoints as small crosses. Paragraph [0103]-ZSIROS discloses analogously for FIG. 1: On the figure, we can see the detected objects on the camera frame marked as small crosses (wherein the detected object in the camera frame is an image keypoint and wherein the camera collects images).) and NIR keypoints from NIR (Fig. 1, illustrates Paragraph [0103]-ZSIROS discloses analogously for FIG. 1: On the figure, we can see the detected objects on the camera frame marked as small crosses, the detected objects on the radar frame marked as circles. Additionally, there can in practice be further points (not shown in FIG. 1), representing the detected objects on the lidar frame (wherein the detected objects in the lidar frame are keypoints from NIR as LiDAR operates in near-infrared wavelengths).);
filtering three dimensional (3D) points from 3D point cloud data (Figs. 3A-3B. Paragraph [0054-0057]-ZSIROS discloses during an automatic extrinsic calibration process, in the following just referred to as calibration, the following data may be used: Regular camera frames (2D) or the depth image created from stereo cameras (3D). Radar point cloud (2D/3D) Lidar point cloud (3D). Further in paragraph [0077]-ZSIROS discloses it is understood that e.g. both the data from a first sensor and the data from a second sensor can be filtered and in that case different filtering rules can be applied to the data from the different sensors (wherein data from one of the sensors is LiDAR point cloud).) based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points (Fig. 1 and 3A. Paragraph [0077]-ZSIROS discloses it is understood that e.g. both the data from a first sensor and the data from a second sensor can be filtered and in that case different filtering rules can be applied to the data from the different sensors. At least one of the applied filtering methods is based on a position of the data, i.e., the decision which datapoint to keep and which datapoint to filter out is based on a position of the datapoint in the data (wherein a position of the data is a 3D point from the NIR keypoints (such as keypoints illustrated in fig. 1.) and wherein the result of filtering the data is filtered points).);
optimizing an extrinsic calibration (Fig. 3B. Paragraph [0101]-ZSIROS discloses for finding the extrinsic calibration of the sensors in question, we compute the transformation, for which the error between the paired points is minimal. A good choice for achieving this is using optimization—it can minimize the error between the paired points iteratively, arriving at a state of transformation that represent the real extrinsic calibration well enough.) based on a reprojection error (Fig. 3B, #342 called reproject paired points and #344 called calculate error.) computed from the filtered NIR-to-3D points (Fig. 3B. Paragraph [0079]-ZSIROS discloses the selection of the sensor pairs can be done by either the user or the software. The pair may be any combination of the camera, radar and lidar devices present, provided they share a common view region. Further in paragraph [0107]-ZSIROS discloses for each sensor calibrated to another, we compute the center points of its detections as projected onto the other sensor's space-using both the a priori and the resulting extrinsic parameters just found. The error between these points and their pairs is then computed, for both the a priori and the resulting cases. This could already been found during the first and last steps of the optimization (wherein the paired points are previously filtered points and wherein the error calculated is a reprojection error).) to obtain an optimized extrinsic calibration for an autonomous entity control system (Fig. 1. Paragraph [0053]-ZSIROS discloses from the perspective of autonomous driving, multiple sensors, for example multiple cameras, are important for improving road safety. However, processing the data of multiple sensors on a vehicle requires both an accurate intrinsic calibration for each camera and an accurate extrinsic calibration.); and
ZSIROS fails to explicitly teach matching the image keypoints and the NIR keypoints using a deep-learning-based neural network that learns relation graphs between the image keypoints and the NIR keypoints.
However, TANG explicitly teaches matching the image keypoints and the NIR keypoints (Fig. 3A. Paragraph [0041]-TANG discloses the keypoint matching system 300 uses a graph convolution model 302 to match 3D keypoints of a 3D map 304 with 2D keypoints of a query image 306 (wherein keypoints of a query image are image keypoints and 3D keypoints are the NIR keypoints).) using a deep-learning-based neural network (Fig. 3A. Paragraph [0040]-TANG discloses a graph convolution model, such as an artificial neural network, may be trained to learn a matching function (wherein a graph convolution model is a deep-learning-based neural network).) that learns relation graphs between the image keypoints and the NIR keypoints (Fig. 3A. Paragraph [0040]-TANG discloses a graph convolution model, such as an artificial neural network, may be trained to learn a matching function. In some such implementations, the matching function may be trained to match 3D keypoints of a 3D map, such as a sparse pre-built map, with 2D keypoints of an image, such as a 2D image captured by a monocular sensor of an agent. As described, in some such implementations, the matching function may match a constellation of 3D keypoints with a constellation of 2D keypoints (wherein learning a matching function to match keypoints is learning relation graphs between the keypoints).);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS of a computer-implemented method for automatic multi-modality sensor calibration with near-infrared images (NIR), comprising: detecting image keypoints from collected images and NIR keypoints from NIR; filtering three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimizing an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of TANG of matching the image keypoints and the NIR keypoints using a deep-learning-based neural network that learns relation graphs between the image keypoints and the NIR keypoints.
Wherein having ZSIROS’s automatic sensor calibration system matching the image keypoints and the NIR keypoints using a deep-learning-based neural network that learns relation graphs between the image keypoints and the NIR keypoints.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and TANG relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while TANG aspects of the present disclosure improve an accuracy of 3D representations of an environment based on one or more images obtained from a camera. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and TANG et al. (US 20210326601 A1), Paragraph [0025].
ZSIROS in view of TANG fail to explicitly teach controlling an entity by employing the optimized extrinsic calibration for the autonomous entity control system.
However, KROEGER explicitly teaches controlling an entity by employing the optimized extrinsic calibration (Fig. 8, illustrates the process of controlling a vehicle based on calibration data. Col. 28, Lines [12-18]-KROEGER discloses at operation 802, the process can include receiving updated calibration data. In some instances, the calibration data can be determined using the calibration techniques discussed herein. At operation 804, the process can include generating a trajectory based at least in part on the updated calibration data (wherein the updated calibration data is the optimized extrinsic calibration and wherein the vehicle is an entity).) for the autonomous entity control system (Fig. 8, #806 called control an autonomous vehicle to follow the trajectory. Col. 28, Lines [24-28]-KROEGER discloses at operation 806, the process can include controlling an autonomous vehicle to follow the trajectory. In some instances, the commands generated in the operation 806 can be relayed to a controller onboard an autonomous vehicle to control the autonomous vehicle to drive the trajectory.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG of a computer-implemented method for automatic multi-modality sensor calibration with near-infrared images (NIR), comprising: detecting image keypoints from collected images and NIR keypoints from NIR; filtering three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimizing an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of KROEGER of controlling an entity by employing the optimized extrinsic calibration for the autonomous entity control system.
Wherein having ZSIROS’s automatic sensor calibration system controlling an entity by employing the optimized extrinsic calibration for the autonomous entity control system.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and KROEGER relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while KROEGER calibration techniques discussed herein can improve the functioning of a computing device by providing a framework to determine optimal calibration for sensors, e.g., an array of cameras, on an autonomous vehicle. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and KROEGER (US 10298910 B1), Col. 3, Lines [37-53].
Regarding claim 3, ZSIROS in view of TANG and further in view of KROEGER explicitly teach the computer-implemented method of claim 1,
ZSIROS in view of TANG fail to explicitly teach wherein controlling the entity further comprises controlling a vehicle based on the optimized extrinsic calibration.
However, KROEGER explicitly teaches wherein controlling the entity further comprises controlling a vehicle based on the optimized extrinsic calibration (Fig. 8, illustrates the process of controlling an entity based on calibration data. Col. 28, Lines [12-18]-KROEGER discloses at operation 802, the process can include receiving updated calibration data. In some instances, the calibration data can be determined using the calibration techniques discussed herein. At operation 804, the process can include generating a trajectory based at least in part on the updated calibration data (wherein the updated calibration data is the optimized extrinsic calibration). Further in Col. 28, Lines [24-28]-KROEGER discloses at operation 806, the process can include controlling an autonomous vehicle to follow the trajectory. In some instances, the commands generated in the operation 806 can be relayed to a controller onboard an autonomous vehicle to control the autonomous vehicle to drive the trajectory.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG and further in view of KROEGER of a computer-implemented method for automatic multi-modality sensor calibration with near-infrared images (NIR), comprising: detecting image keypoints from collected images and NIR keypoints from NIR; filtering three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimizing an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of KROEGER of wherein controlling the entity further comprises controlling a vehicle based on the optimized extrinsic calibration.
Wherein having ZSIROS’s automatic sensor calibration system wherein controlling the entity further comprises controlling a vehicle based on the optimized extrinsic calibration.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and KROEGER relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while KROEGER calibration techniques discussed herein can improve the functioning of a computing device by providing a framework to determine optimal calibration for sensors, e.g., an array of cameras, on an autonomous vehicle. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and KROEGER (US 10298910 B1), Col. 3, Lines [37-53].
Regarding claim 7, ZSIROS in view of TANG and further in view of KROEGER explicitly teach the computer-implemented method of claim 1,
Although ZSIROS explicitly teaches wherein optimizing the extrinsic calibration further comprises iteratively determining the extrinsic calibration.
ZSIROS in view of TANG fail to explicitly teach wherein optimizing the extrinsic calibration further comprises iteratively determining the extrinsic calibration that includes a highest number of data points between 3D points and two-dimensional (2D) points.
However, KROEGER explicitly teach wherein optimizing the extrinsic calibration further comprises iteratively determining the extrinsic calibration (Col. 11, Lines [48-57]-KROEGER discloses in at least some examples, such intrinsic and/or depth optimizations may be performed jointly or iteratively with extrinsic optimization, as discussed generally herein. Thus, for example, the process 100 and the process 200 may be performed simultaneously, and on the same images/point pairs, to provide robust sensor calibration. Moreover, some subset of the extrinsic calibration of the process 100, the depth optimizations and/or the estimated intrinsic parameters may be performed simultaneously and/or iteratively.) that includes a highest number of data points between 3D points and two-dimensional (2D) points (Col. 3, Lines [15-24]-KROEGER discloses the lidar data, which does consider the three-dimensional characteristics of the environment (e.g., feature edges) can be used to further constrain the camera sensors, thereby removing any scale ambiguity and/or translational/rotational offsets. These two calibrations may be performed simultaneously, e.g., in parallel, across large sets of image data and lidar data to determine calibration data useful to calibrate extrinsic characteristics, e.g., physical misalignment, of cameras on an autonomous vehicle (wherein the LiDAR data is 3D points and image data is 2D points). Further in Col. 13, Lines [5-8]-KROEGER discloses the points associated with the highest errors of each region may be removed so as to have the same total number of points per region as the region with the lowest number of points (wherein the points are points from the LiDAR data and image data). Therefore, it would have been obvious to one of ordinary skill of the art at the time the invention was made to have the use of iteratively determining the extrinsic calibration that includes a highest number of data points between 3D points and two-dimensional (2D) points since KROEGER clearly discloses utilizing the lowest number of data points to iteratively determine extrinsic calibration. Thus, in order to have an automatic sensor calibration system that enhances accuracy of calibrated sensors by utilizing more data.).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG and further in view of KROEGER of a computer-implemented method for automatic multi-modality sensor calibration with near-infrared images (NIR), comprising: detecting image keypoints from collected images and NIR keypoints from NIR; filtering three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimizing an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of KROEGER of wherein optimizing the extrinsic calibration further comprises iteratively determining the extrinsic calibration that includes a highest number of data points between 3D points and two-dimensional (2D) points.
Wherein having ZSIROS’s automatic sensor calibration system wherein optimizing the extrinsic calibration further comprises iteratively determining the extrinsic calibration that includes a highest number of data points between 3D points and two-dimensional (2D) points.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and KROEGER relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while KROEGER calibration techniques discussed herein can improve the functioning of a computing device by providing a framework to determine optimal calibration for sensors, e.g., an array of cameras, on an autonomous vehicle. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and KROEGER (US 10298910 B1), Col. 3, Lines [37-53].
Regarding claim 8, ZSIROS explicitly teaches a system for automatic multi-modality sensor calibration with near-infrared images (NIR) (Fig. 1. Paragraph [0054-0059]-ZSIROS discloses during an automatic extrinsic calibration process, in the following just referred to as calibration, the following data may be used: Regular camera frames (2D) or the depth image created from stereo cameras (3D). Radar point cloud (2D/3D) Lidar point cloud (3D) The sensor to be calibrated may provide additional data, e.g., the velocity, intensity, cross-section size and/or confidence of detections. These may be used for aiding the calibration process.), comprising:
a memory device (Fig. 1. Paragraph [0046]-ZSIROS discloses a further aspect of the invention refers to a computer-readable storage medium storing program code (wherein a computer-readable storage medium is a memory device).);
detect image keypoints from collected images (Fig. 1, illustrates detected image keypoints as small crosses. Paragraph [0103]-ZSIROS discloses analogously for FIG. 1: On the figure, we can see the detected objects on the camera frame marked as small crosses (wherein the detected object in the camera frame is an image keypoint and wherein the camera collects images).) and NIR keypoints from NIR (Fig. 1, illustrates Paragraph [0103]-ZSIROS discloses analogously for FIG. 1: On the figure, we can see the detected objects on the camera frame marked as small crosses, the detected objects on the radar frame marked as circles. Additionally, there can in practice be further points (not shown in FIG. 1), representing the detected objects on the lidar frame (wherein the detected objects in the lidar frame are keypoints from NIR as LiDAR operates in near-infrared wavelengths).);
filter three dimensional (3D) points from 3D point cloud data (Figs. 3A-3B. Paragraph [0054-0057]-ZSIROS discloses during an automatic extrinsic calibration process, in the following just referred to as calibration, the following data may be used: Regular camera frames (2D) or the depth image created from stereo cameras (3D). Radar point cloud (2D/3D) Lidar point cloud (3D). Further in paragraph [0077]-ZSIROS discloses it is understood that e.g. both the data from a first sensor and the data from a second sensor can be filtered and in that case different filtering rules can be applied to the data from the different sensors (wherein data from one of the sensors is LiDAR point cloud).) based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points(Fig. 1 and 3A. Paragraph [0077]-ZSIROS discloses it is understood that e.g. both the data from a first sensor and the data from a second sensor can be filtered and in that case different filtering rules can be applied to the data from the different sensors. At least one of the applied filtering methods is based on a position of the data, i.e., the decision which datapoint to keep and which datapoint to filter out is based on a position of the datapoint in the data (wherein a position of the data is a 3D point from the NIR keypoints (such as keypoints illustrated in fig. 1.) and wherein the result of filtering the data is filtered points).);
optimize an extrinsic calibration (Fig. 3B. Paragraph [0101]-ZSIROS discloses for finding the extrinsic calibration of the sensors in question, we compute the transformation, for which the error between the paired points is minimal. A good choice for achieving this is using optimization—it can minimize the error between the paired points iteratively, arriving at a state of transformation that represent the real extrinsic calibration well enough.) based on a reprojection error (Fig. 3B, #342 called reproject paired points and #344 called calculate error.) computed from the filtered NIR-to-3D points (Fig. 3B. Paragraph [0079]-ZSIROS discloses the selection of the sensor pairs can be done by either the user or the software. The pair may be any combination of the camera, radar and lidar devices present, provided they share a common view region. Further in paragraph [0107]-ZSIROS discloses for each sensor calibrated to another, we compute the center points of its detections as projected onto the other sensor's space-using both the a priori and the resulting extrinsic parameters just found. The error between these points and their pairs is then computed, for both the a priori and the resulting cases. This could already been found during the first and last steps of the optimization (wherein the paired points are previously filtered points and wherein the error calculated is a reprojection error).) to obtain an optimized extrinsic calibration for an autonomous entity control system (Fig. 1. Paragraph [0053]-ZSIROS discloses from the perspective of autonomous driving, multiple sensors, for example multiple cameras, are important for improving road safety. However, processing the data of multiple sensors on a vehicle requires both an accurate intrinsic calibration for each camera and an accurate extrinsic calibration.); and
Although ZSIROS explicitly teaches a processor device, ZSIROS fails to explicitly teach one or more processor devices operatively coupled with the memory device to: match the image keypoints and the NIR keypoints using a deep-learning-based neural network that learns relation graphs between the image keypoints and the NIR keypoints.
However, TANG explicitly teaches one or more processor devices (Fig. 4, #420 called processor.) operatively coupled with the memory device to (Fig. 4. Paragraph [0079]-TANG discloses the bus 440 links together various circuits including one or more processors and/or hardware modules, represented by a processor 420, a communication module 422, a location module 418, a sensor module 402, a locomotion module 426, a navigation module 424, and a computer-readable medium 414.):
match the image keypoints and the NIR keypoints (Fig. 3A. Paragraph [0041]-TANG discloses the keypoint matching system 300 uses a graph convolution model 302 to match 3D keypoints of a 3D map 304 with 2D keypoints of a query image 306 (wherein keypoints of a query image are image keypoints and 3D keypoints are the NIR keypoints).) using a deep-learning-based neural network (Fig. 3A. Paragraph [0040]-TANG discloses a graph convolution model, such as an artificial neural network, may be trained to learn a matching function (wherein a graph convolution model is a deep-learning-based neural network).) that learns relation graphs between the image keypoints and the NIR keypoints (Fig. 3A. Paragraph [0040]-TANG discloses a graph convolution model, such as an artificial neural network, may be trained to learn a matching function. In some such implementations, the matching function may be trained to match 3D keypoints of a 3D map, such as a sparse pre-built map, with 2D keypoints of an image, such as a 2D image captured by a monocular sensor of an agent. As described, in some such implementations, the matching function may match a constellation of 3D keypoints with a constellation of 2D keypoints (wherein learning a matching function to match keypoints is learning relation graphs between the keypoints).);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS of a system for automatic multi-modality sensor calibration with near-infrared images (NIR), comprising: a memory device; detect image keypoints from collected images and NIR keypoints from NIR; filter three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimize an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of TANG of one or more processor devices operatively coupled with the memory device to: match the image keypoints and the NIR keypoints using a deep-learning-based neural network that learns relation graphs between the image keypoints and the NIR keypoints.
Wherein having ZSIROS’s automatic sensor calibration system having one or more processor devices operatively coupled with the memory device to: match the image keypoints and the NIR keypoints using a deep-learning-based neural network that learns relation graphs between the image keypoints and the NIR keypoints.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and TANG relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while TANG aspects of the present disclosure improve an accuracy of 3D representations of an environment based on one or more images obtained from a camera. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and TANG et al. (US 20210326601 A1), Paragraph [0025].
ZSIROS in view of TANG fail to explicitly teach control an entity by employing the optimized extrinsic calibration for the autonomous entity control system.
However, KROEGER explicitly teaches control an entity by employing the optimized extrinsic calibration (Fig. 8, illustrates the process of controlling a vehicle based on calibration data. Col. 28, Lines [12-18]-KROEGER discloses at operation 802, the process can include receiving updated calibration data. In some instances, the calibration data can be determined using the calibration techniques discussed herein. At operation 804, the process can include generating a trajectory based at least in part on the updated calibration data (wherein the updated calibration data is the optimized extrinsic calibration and wherein the vehicle is an entity).) for the autonomous entity control system (Fig. 8, #806 called control an autonomous vehicle to follow the trajectory. Col. 28, Lines [24-28]-KROEGER discloses at operation 806, the process can include controlling an autonomous vehicle to follow the trajectory. In some instances, the commands generated in the operation 806 can be relayed to a controller onboard an autonomous vehicle to control the autonomous vehicle to drive the trajectory.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG of a system for automatic multi-modality sensor calibration with near-infrared images (NIR), comprising: a memory device; detect image keypoints from collected images and NIR keypoints from NIR; filter three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimize an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of KROEGER of control an entity by employing the optimized extrinsic calibration for the autonomous entity control system.
Wherein having ZSIROS’s automatic sensor calibration system having control an entity by employing the optimized extrinsic calibration for the autonomous entity control system.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and KROEGER relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while KROEGER calibration techniques discussed herein can improve the functioning of a computing device by providing a framework to determine optimal calibration for sensors, e.g., an array of cameras, on an autonomous vehicle. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and KROEGER (US 10298910 B1), Col. 3, Lines [37-53].
Regarding claim 10, ZSIROS in view of TANG and further in view of KROEGER explicitly teach the system of claim 8,
ZSIROS in view of TANG fail to explicitly teach wherein to control the entity further comprises controlling a vehicle based on the extrinsic calibration.
However, KROEGER explicitly teaches wherein to control the entity further comprises controlling a vehicle based on the extrinsic calibration (Fig. 8, illustrates the process of controlling an entity based on calibration data. Col. 28, Lines [12-18]-KROEGER discloses at operation 802, the process can include receiving updated calibration data. In some instances, the calibration data can be determined using the calibration techniques discussed herein. At operation 804, the process can include generating a trajectory based at least in part on the updated calibration data (wherein the updated calibration data is the optimized extrinsic calibration). Further in Col. 28, Lines [24-28]-KROEGER discloses at operation 806, the process can include controlling an autonomous vehicle to follow the trajectory. In some instances, the commands generated in the operation 806 can be relayed to a controller onboard an autonomous vehicle to control the autonomous vehicle to drive the trajectory.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG and further in view of KROEGER of a system for automatic multi-modality sensor calibration with near-infrared images (NIR), comprising: a memory device; detect image keypoints from collected images and NIR keypoints from NIR; filter three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimize an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of KROEGER of control an entity by employing the optimized extrinsic calibration for the autonomous entity control system.
Wherein having ZSIROS’s automatic sensor calibration system having control an entity by employing the optimized extrinsic calibration for the autonomous entity control system.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and KROEGER relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while KROEGER calibration techniques discussed herein can improve the functioning of a computing device by providing a framework to determine optimal calibration for sensors, e.g., an array of cameras, on an autonomous vehicle. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and KROEGER (US 10298910 B1), Col. 3, Lines [37-53].
Regarding claim 14, ZSIROS in view of TANG and further in view of KROEGER explicitly teach the system of claim 8, Although ZSIROS explicitly teaches wherein to optimize the extrinsic calibration further comprises to iteratively determine the extrinsic calibration.
ZSIROS in view of TANG fail to explicitly teach wherein to optimize the extrinsic calibration further comprises to iteratively determine the extrinsic calibration that includes a highest number of data points between 3D points and two-dimensional (2D) points.
However, KROEGER explicitly teaches wherein to optimize the extrinsic calibration further comprises to iteratively determine the extrinsic calibration (Fig. 1. Col. 11, Lines [48-57]-KROEGER discloses in at least some examples, such intrinsic and/or depth optimizations may be performed jointly or iteratively with extrinsic optimization, as discussed generally herein. Thus, for example, the process 100 and the process 200 may be performed simultaneously, and on the same images/point pairs, to provide robust sensor calibration. Moreover, some subset of the extrinsic calibration of the process 100, the depth optimizations and/or the estimated intrinsic parameters may be performed simultaneously and/or iteratively.) that includes a highest number of data points between 3D points and two-dimensional (2D) points (Fig. 1. Col. 3, Lines [15-24]-KROEGER discloses the lidar data, which does consider the three-dimensional characteristics of the environment (e.g., feature edges) can be used to further constrain the camera sensors, thereby removing any scale ambiguity and/or translational/rotational offsets. These two calibrations may be performed simultaneously, e.g., in parallel, across large sets of image data and lidar data to determine calibration data useful to calibrate extrinsic characteristics, e.g., physical misalignment, of cameras on an autonomous vehicle (wherein the LiDAR data is 3D points and image data is 2D points). Further in Col. 13, Lines [5-8]-KROEGER discloses the points associated with the highest errors of each region may be removed so as to have the same total number of points per region as the region with the lowest number of points (wherein the points are points from the LiDAR data and image data). Therefore, it would have been obvious to one of ordinary skill of the art at the time the invention was made to have the use of iteratively determining the extrinsic calibration that includes a highest number of data points between 3D points and two-dimensional (2D) points since KROEGER clearly discloses utilizing the lowest number of data points to iteratively determine extrinsic calibration. Thus, in order to have an automatic sensor calibration system that enhances accuracy of calibrated sensors by utilizing more data.).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG and further in view of KROEGER of a system for automatic multi-modality sensor calibration with near-infrared images (NIR), comprising: a memory device; detect image keypoints from collected images and NIR keypoints from NIR; filter three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimize an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of KROEGER of wherein to optimize the extrinsic calibration further comprises to iteratively determine the extrinsic calibration that includes a highest number of data points between 3D points and two-dimensional (2D) points.
Wherein having ZSIROS’s automatic sensor calibration system wherein to optimize the extrinsic calibration further comprises to iteratively determine the extrinsic calibration that includes a highest number of data points between 3D points and two-dimensional (2D) points.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and KROEGER relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while KROEGER calibration techniques discussed herein can improve the functioning of a computing device by providing a framework to determine optimal calibration for sensors, e.g., an array of cameras, on an autonomous vehicle. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and KROEGER (US 10298910 B1), Col. 3, Lines [37-53].
Regarding claim 15, ZSIROS explicitly teaches a non-transitory computer program product comprising a computer-readable storage medium including program code for automatic multi-modality sensor calibration with near infrared images (NIR) (Fig. 1. Paragraph [0046]-ZSIROS discloses a further aspect of the invention refers to a computer-readable storage medium storing program code, the program code comprising instructions that when executed by a processor carry out the method of the second aspect or one of the implementations of the first aspect (wherein program code is a computer program product).),
detect image keypoints from collected images (Fig. 1, illustrates detected image keypoints as small crosses. Paragraph [0103]-ZSIROS discloses analogously for FIG. 1: On the figure, we can see the detected objects on the camera frame marked as small crosses (wherein the detected object in the camera frame is an image keypoint and wherein the camera collects images).) and NIR keypoints from NIR (Fig. 1, illustrates Paragraph [0103]-ZSIROS discloses analogously for FIG. 1: On the figure, we can see the detected objects on the camera frame marked as small crosses, the detected objects on the radar frame marked as circles. Additionally, there can in practice be further points (not shown in FIG. 1), representing the detected objects on the lidar frame (wherein the detected objects in the lidar frame are keypoints from NIR as LiDAR operates in near-infrared wavelengths).);
filter three dimensional (3D) points from 3D point cloud data (Figs. 3A-3B. Paragraph [0054-0057]-ZSIROS discloses during an automatic extrinsic calibration process, in the following just referred to as calibration, the following data may be used: Regular camera frames (2D) or the depth image created from stereo cameras (3D). Radar point cloud (2D/3D) Lidar point cloud (3D). Further in paragraph [0077]-ZSIROS discloses it is understood that e.g. both the data from a first sensor and the data from a second sensor can be filtered and in that case different filtering rules can be applied to the data from the different sensors (wherein data from one of the sensors is LiDAR point cloud).) based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points (Fig. 1 and 3A. Paragraph [0077]-ZSIROS discloses it is understood that e.g. both the data from a first sensor and the data from a second sensor can be filtered and in that case different filtering rules can be applied to the data from the different sensors. At least one of the applied filtering methods is based on a position of the data, i.e., the decision which datapoint to keep and which datapoint to filter out is based on a position of the datapoint in the data (wherein a position of the data is a 3D point from the NIR keypoints (such as keypoints illustrated in fig. 1.) and wherein the result of filtering the data is filtered points).);
optimize an extrinsic calibration (Fig. 3B. Paragraph [0101]-ZSIROS discloses for finding the extrinsic calibration of the sensors in question, we compute the transformation, for which the error between the paired points is minimal. A good choice for achieving this is using optimization—it can minimize the error between the paired points iteratively, arriving at a state of transformation that represent the real extrinsic calibration well enough.) based on a reprojection error (Fig. 3B, #342 called reproject paired points and #344 called calculate error.) computed from the filtered NIR-to-3D points (Fig. 3B. Paragraph [0079]-ZSIROS discloses the selection of the sensor pairs can be done by either the user or the software. The pair may be any combination of the camera, radar and lidar devices present, provided they share a common view region. Further in paragraph [0107]-ZSIROS discloses for each sensor calibrated to another, we compute the center points of its detections as projected onto the other sensor's space-using both the a priori and the resulting extrinsic parameters just found. The error between these points and their pairs is then computed, for both the a priori and the resulting cases. This could already been found during the first and last steps of the optimization (wherein the paired points are previously filtered points and wherein the error calculated is a reprojection error).) to obtain an optimized extrinsic calibration for an autonomous entity control system (Fig. 1. Paragraph [0053]-ZSIROS discloses from the perspective of autonomous driving, multiple sensors, for example multiple cameras, are important for improving road safety. However, processing the data of multiple sensors on a vehicle requires both an accurate intrinsic calibration for each camera and an accurate extrinsic calibration.);
Although ZSIROS explicitly teaches wherein the program code when executed (Fig. 1. Paragraph [0046]-ZSIROS discloses a further aspect of the invention refers to a computer-readable storage medium storing program code, the program code comprising instructions that when executed by a processor carry out the method of the second aspect or one of the implementations of the first aspect.).
ZSIROS is silent on executed on a computer causes the computer to, and fails to explicitly teach match the image keypoints and the NIR keypoints using a deep-learning-based neural network that learns relation graphs between the image keypoints and the NIR keypoints.
However, TANG explicitly teaches wherein the program code when executed on a computer causes the computer to (Fig. 4. Paragraph [0112]-TANG discloses it should be appreciated that aspects of the present disclosure result in improvements to the functioning of the processor, computer, machine, or other system implementing such aspects (wherein such aspects is executing of system, wherein executing of the system is executed on the computer). Therefore, it would have been obvious to one of ordinary skill of the art at the time the invention was made to have the use of wherein the program code when executed on a computer causes the computer to since KROEGER clearly discloses executing functions with a computer. Thus, in order to have an automatic sensor calibration system with enhanced functionality to store and manage data.):
and
TANG further explicitly teaches match the image keypoints and the NIR keypoints (Fig. 3A. Paragraph [0041]-TANG discloses the keypoint matching system 300 uses a graph convolution model 302 to match 3D keypoints of a 3D map 304 with 2D keypoints of a query image 306 (wherein keypoints of a query image are image keypoints and 3D keypoints are the NIR keypoints).) using a deep-learning-based neural network (Fig. 3A. Paragraph [0040]-TANG discloses a graph convolution model, such as an artificial neural network, may be trained to learn a matching function (wherein a graph convolution model is a deep-learning-based neural network).) that learns relation graphs between the image keypoints and the NIR keypoints (Fig. 3A. Paragraph [0040]-TANG discloses a graph convolution model, such as an artificial neural network, may be trained to learn a matching function. In some such implementations, the matching function may be trained to match 3D keypoints of a 3D map, such as a sparse pre-built map, with 2D keypoints of an image, such as a 2D image captured by a monocular sensor of an agent. As described, in some such implementations, the matching function may match a constellation of 3D keypoints with a constellation of 2D keypoints (wherein learning a matching function to match keypoints is learning relation graphs between the keypoints).);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS of a non-transitory computer program product comprising a computer-readable storage medium including program code for automatic multi-modality sensor calibration with near infrared images (NIR), wherein the program code when executed on a computer causes the computer to: detect image keypoints from collected images and NIR keypoints from NIR; filter three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimize an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of TANG of wherein the program code when executed on a computer causes the computer to: match the image keypoints and the NIR keypoints using a deep-learning-based neural network that learns relation graphs between the image keypoints and the NIR keypoints.
Wherein having ZSIROS’s automatic sensor calibration system wherein the program code when executed on a computer causes the computer to: match the image keypoints and the NIR keypoints using a deep-learning-based neural network that learns relation graphs between the image keypoints and the NIR keypoints.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and TANG relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while TANG aspects of the present disclosure improve an accuracy of 3D representations of an environment based on one or more images obtained from a camera. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and TANG et al. (US 20210326601 A1), Paragraph [0025].
ZSIROS in view of TANG fail to explicitly teach control an entity by employing the optimized extrinsic calibration for the autonomous entity control system.
However, KROEGER explicitly teaches control an entity by employing the optimized extrinsic calibration for the autonomous entity control system (Fig. 8, illustrates the process of controlling a vehicle based on calibration data. Col. 28, Lines [12-18]-KROEGER discloses at operation 802, the process can include receiving updated calibration data. In some instances, the calibration data can be determined using the calibration techniques discussed herein. At operation 804, the process can include generating a trajectory based at least in part on the updated calibration data (wherein the updated calibration data is the optimized extrinsic calibration and wherein the vehicle is an entity).) for the autonomous entity control system (Fig. 8, #806 called control an autonomous vehicle to follow the trajectory. Col. 28, Lines [24-28]-KROEGER discloses at operation 806, the process can include controlling an autonomous vehicle to follow the trajectory. In some instances, the commands generated in the operation 806 can be relayed to a controller onboard an autonomous vehicle to control the autonomous vehicle to drive the trajectory.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG of a non-transitory computer program product comprising a computer-readable storage medium including program code for automatic multi-modality sensor calibration with near infrared images (NIR), wherein the program code when executed on a computer causes the computer to: detect image keypoints from collected images and NIR keypoints from NIR; filter three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimize an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of KROEGER of control an entity by employing the optimized extrinsic calibration for the autonomous entity control system.
Wherein having ZSIROS’s automatic sensor calibration system having control an entity by employing the optimized extrinsic calibration for the autonomous entity control system.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and KROEGER relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while KROEGER calibration techniques discussed herein can improve the functioning of a computing device by providing a framework to determine optimal calibration for sensors, e.g., an array of cameras, on an autonomous vehicle. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and KROEGER (US 10298910 B1), Col. 3, Lines [37-53].
Regarding claim 20, ZSIROS in view of TANG and further in view of KROEGER explicitly teach the non-transitory computer program product of claim 15, Although ZSIROS explicitly teaches wherein to optimize the extrinsic calibration further comprises iteratively determining the extrinsic calibration.
ZSIROS in view of TANG fail to explicitly teach wherein to optimize the extrinsic calibration further comprises iteratively determining the extrinsic calibration that includes a highest number of data points between 3D points and two-dimensional (2D) points.
However, KROEGER explicitly teach wherein to optimize the extrinsic calibration further comprises iteratively determining the extrinsic calibration (Col. 11, Lines [48-57]-KROEGER discloses in at least some examples, such intrinsic and/or depth optimizations may be performed jointly or iteratively with extrinsic optimization, as discussed generally herein. Thus, for example, the process 100 and the process 200 may be performed simultaneously, and on the same images/point pairs, to provide robust sensor calibration. Moreover, some subset of the extrinsic calibration of the process 100, the depth optimizations and/or the estimated intrinsic parameters may be performed simultaneously and/or iteratively.) that includes a highest number of data points between 3D points and two-dimensional (2D) points (Col. 3, Lines [15-24]-KROEGER discloses the lidar data, which does consider the three-dimensional characteristics of the environment (e.g., feature edges) can be used to further constrain the camera sensors, thereby removing any scale ambiguity and/or translational/rotational offsets. These two calibrations may be performed simultaneously, e.g., in parallel, across large sets of image data and lidar data to determine calibration data useful to calibrate extrinsic characteristics, e.g., physical misalignment, of cameras on an autonomous vehicle (wherein the LiDAR data is 3D points and image data is 2D points). Further in Col. 13, Lines [5-8]-KROEGER discloses the points associated with the highest errors of each region may be removed so as to have the same total number of points per region as the region with the lowest number of points (wherein the points are points from the LiDAR data and image data). Therefore, it would have been obvious to one of ordinary skill of the art at the time the invention was made to have the use of iteratively determining the extrinsic calibration that includes a highest number of data points between 3D points and two-dimensional (2D) points since KROEGER clearly discloses utilizing the lowest number of data points to iteratively determine extrinsic calibration. Thus, in order to have an automatic sensor calibration system that enhances accuracy of calibrated sensors by utilizing more data.).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG ZSIROS in view of TANG and further in view of KROEGER of a non-transitory computer program product comprising a computer-readable storage medium including program code for automatic multi-modality sensor calibration with near infrared images (NIR), wherein the program code when executed on a computer causes the computer to: detect image keypoints from collected images and NIR keypoints from NIR; filter three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimize an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of KROEGER of wherein to optimize the extrinsic calibration further comprises iteratively determining the extrinsic calibration that includes a highest number of data points between 3D points and two-dimensional (2D) points.
Wherein having ZSIROS’s automatic sensor calibration system wherein to optimize the extrinsic calibration further comprises iteratively determining the extrinsic calibration that includes a highest number of data points between 3D points and two-dimensional (2D) points.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and KROEGER relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while KROEGER calibration techniques discussed herein can improve the functioning of a computing device by providing a framework to determine optimal calibration for sensors, e.g., an array of cameras, on an autonomous vehicle. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and KROEGER (US 10298910 B1), Col. 3, Lines [37-53].
Claims 2, 9, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over ZSIROS et al. (US 20250363666 A1), hereinafter referenced as ZSIROS, in view of TANG et al. (US 20210326601 A1), hereinafter referenced as TANG, and further in view of KROEGER (US 10298910 B1), hereinafter referenced as KROEGER (US 10298910 B1), and further in view of ZHANG (US 20210241468 A1), hereinafter referenced as ZHANG.
Regarding claim 2, ZSIROS in view of TANG and further in view of KROEGER explicitly teach the computer-implemented method of claim 1,
ZSIROS in view of TANG and further in view of KROEGER fail to explicitly teach wherein controlling the entity further comprises controlling an autonomous patient monitoring system to monitor patients within a hospital ward.
However, ZHANG explicitly teaches wherein controlling the entity further comprises controlling an autonomous patient monitoring system to monitor patients within a hospital ward (Fig. 1. Paragraph [0143]-ZHANG discloses the smart motion detection system may be implemented in a hospital for patient monitoring and alerting. Taking an epilepsy patient (or people with seizures) as an example; the smart motion detection system may capture videos of the epilepsy patient, analyze the motion of the epilepsy patient in captured videos and immediately transmit an alarm to outside doctors or observers of the epilepsy patient when an abnormal motion is detected (wherein the smart motion detection system implemented in a hospital is an autonomous patient monitoring system).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG and further in view of KROEGER of a computer-implemented method for automatic multi-modality sensor calibration with near-infrared images (NIR), comprising: detecting image keypoints from collected images and NIR keypoints from NIR; filtering three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimizing an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of ZHANG of wherein controlling the entity further comprises controlling an autonomous patient monitoring system to monitor patients within a hospital ward
Wherein having ZSIROS’s automatic sensor calibration system wherein controlling the entity further comprises controlling an autonomous patient monitoring system to monitor patients within a hospital ward
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and TANG relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while ZHANG it is desirable to provide video surveillance and analysis systems for that can efficiently and effectively recognize the objects in the captured videos, understand the what has happened in the videos between the objects (and/or the behaviors of the objects), and generate alarms or notifications based on the observations. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and ZHANG (US 20210241468 A1), Paragraph [0004].
Regarding claim 9, ZSIROS in view of TANG and further in view of KROEGER explicitly teach the system of claim 8,
ZSIROS in view of TANG and further in view of KROEGER fail to explicitly teach wherein to control the entity further comprises controlling an autonomous patient monitoring system based on the extrinsic calibration to monitor patients within a hospital ward.
However, ZHANG explicitly teaches wherein to control the entity further comprises controlling an autonomous patient monitoring system based on the extrinsic calibration to monitor patients within a hospital ward (Fig. 1. Paragraph [0143]-ZHANG discloses the smart motion detection system may be implemented in a hospital for patient monitoring and alerting. Taking an epilepsy patient (or people with seizures) as an example; the smart motion detection system may capture videos of the epilepsy patient, analyze the motion of the epilepsy patient in captured videos and immediately transmit an alarm to outside doctors or observers of the epilepsy patient when an abnormal motion is detected (wherein the smart motion detection system implemented in a hospital is an autonomous patient monitoring system).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG and further in view of KROEGER of a system for automatic multi-modality sensor calibration with near-infrared images (NIR), comprising: a memory device; detect image keypoints from collected images and NIR keypoints from NIR; filter three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimize an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of ZHANG of wherein to control the entity further comprises controlling an autonomous patient monitoring system based on the extrinsic calibration to monitor patients within a hospital ward.
Wherein having ZSIROS’s automatic sensor calibration system wherein to control the entity further comprises controlling an autonomous patient monitoring system based on the extrinsic calibration to monitor patients within a hospital ward.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and TANG relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while ZHANG it is desirable to provide video surveillance and analysis systems for that can efficiently and effectively recognize the objects in the captured videos, understand the what has happened in the videos between the objects (and/or the behaviors of the objects), and generate alarms or notifications based on the observations. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and ZHANG (US 20210241468 A1), Paragraph [0004].
Regarding claim 16, ZSIROS in view of TANG and further in view of KROEGER explicitly teach the non-transitory computer program product of claim 15,
ZSIROS in view of TANG and further in view of KROEGER fail to explicitly teach wherein to control the entity further comprises controlling an autonomous patient monitoring system.
However, ZHANG explicitly teaches wherein to control the entity further comprises controlling an autonomous patient monitoring system (Fig. 1. Paragraph [0143]-ZHANG discloses the smart motion detection system may be implemented in a hospital for patient monitoring and alerting. Taking an epilepsy patient (or people with seizures) as an example; the smart motion detection system may capture videos of the epilepsy patient, analyze the motion of the epilepsy patient in captured videos and immediately transmit an alarm to outside doctors or observers of the epilepsy patient when an abnormal motion is detected (wherein the smart motion detection system implemented in a hospital is an autonomous patient monitoring system).) based on the extrinsic calibration (Fig. 1. Paragraph [0080]-ZHANG discloses the calibration model may be used to transform coordinates of a point in the 2D coordinate system to coordinates of the point in the 3D coordinate system. The calibration model may be defined by parameters (e.g., intrinsic parameters, extrinsic parameters, or distortion parameters) of the visual sensor (wherein the calibration model employs an extrinsic parameter (i.e. extrinsic calibration) and wherein the smart motion detection system uses the calibration model to accurately monitor objects in a sensor’s field of view).) to monitor patients within a hospital ward (Fig. 1. Paragraph [0143]-ZHANG the smart motion detection system may be implemented in a hospital for patient monitoring and alerting.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG and further in view of KROEGER of a non-transitory computer program product comprising a computer-readable storage medium including program code for automatic multi-modality sensor calibration with near infrared images (NIR), wherein the program code when executed on a computer causes the computer to: detect image keypoints from collected images and NIR keypoints from NIR; filter three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimize an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of ZHANG of wherein to control the entity further comprises controlling an autonomous patient monitoring system based on the extrinsic calibration to monitor patients within a hospital ward.
Wherein having ZSIROS’s automatic sensor calibration system wherein to control the entity further comprises controlling an autonomous patient monitoring system based on the extrinsic calibration to monitor patients within a hospital ward.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and ZHANG relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while ZHANG it is desirable to provide video surveillance and analysis systems for that can efficiently and effectively recognize the objects in the captured videos, understand the what has happened in the videos between the objects (and/or the behaviors of the objects), and generate alarms or notifications based on the observations. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and ZHANG (US 20210241468 A1), Paragraph [0004].
Claims 4, 11, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over ZSIROS et al. (US 20250363666 A1), hereinafter referenced as ZSIROS, in view of TANG et al. (US 20210326601 A1), hereinafter referenced as TANG, and further in view of KROEGER (US 10298910 B1), hereinafter referenced as KROEGER (US 10298910 B1), and further in view of BRALEY et al. (US 20210012166 A1), hereinafter referenced as BRALEY.
Regarding claim 4, ZSIROS in view of TANG and further in view of KROEGER explicitly teach the computer-implemented method of claim 1,
ZSIROS in view of TANG and further in view of KROEGER fail to explicitly teach wherein filtering the 3D points further comprises retaining NIR keypoints with corresponding 3D points from the 3D point cloud data.
However, BRALEY explicitly teaches wherein filtering the 3D points further comprises retaining NIR keypoints with corresponding 3D points from the 3D point cloud data (Fig. 2. Paragraph [0107]-BRALEY discloses the system identifies a set of multiple region pairs using the embeddings of the image regions and the embeddings of the point cloud regions (508) Each region pair specifies an image region and a point cloud region that characterize the same area of the environment. To identify the region pairs, the system uses a matching algorithm (e.g., a nearest neighbor matching algorithm) to identify a set of multiple embedding pairs, each of which specifies an embedding of an image region and an embedding of a point cloud region. The system uses each embedding pair to identify a respective region pair that specifies the image region and the point cloud region corresponding to the embedding pair (wherein a region pair represents a retained NIR keypoint with a corresponding 3D point).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG and further in view of KROEGER of a computer-implemented method for automatic multi-modality sensor calibration with near-infrared images (NIR), comprising: detecting image keypoints from collected images and NIR keypoints from NIR; filtering three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimizing an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of BRALEY of wherein filtering the 3D points further comprises retaining NIR keypoints with corresponding 3D points from the 3D point cloud data.
Wherein having ZSIROS’s automatic sensor calibration system wherein filtering the 3D points further comprises retaining NIR keypoints with corresponding 3D points from the 3D point cloud data.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and BRALEY relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while BRALEY by using the alignment system, the on-board system of the vehicle can generate planning decisions that plan the future trajectory of the vehicle and enable the vehicle to operate more safely and efficiently. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and BRALEY et al. (US 20210012166 A1), Paragraph [0024].
Regarding claim 11, ZSIROS in view of TANG and further in view of KROEGER explicitly teach the system of claim 8,
ZSIROS in view of TANG and further in view of KROEGER fail to explicitly teach wherein to filter the 3D points further comprises retaining NIR keypoints with corresponding 3D points from the 3D point cloud data.
However, BRALEY explicitly teaches wherein to filter the 3D points further comprises retaining NIR keypoints with corresponding 3D points from the 3D point cloud data (Fig. 2. Paragraph [0107]-BRALEY discloses the system identifies a set of multiple region pairs using the embeddings of the image regions and the embeddings of the point cloud regions (508) Each region pair specifies an image region and a point cloud region that characterize the same area of the environment. To identify the region pairs, the system uses a matching algorithm (e.g., a nearest neighbor matching algorithm) to identify a set of multiple embedding pairs, each of which specifies an embedding of an image region and an embedding of a point cloud region. The system uses each embedding pair to identify a respective region pair that specifies the image region and the point cloud region corresponding to the embedding pair (wherein a region pair represents a retained NIR keypoint with a corresponding 3D point).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG and further in view of KROEGER of a system for automatic multi-modality sensor calibration with near-infrared images (NIR), comprising: a memory device; detect image keypoints from collected images and NIR keypoints from NIR; filter three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimize an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of BRALEY of wherein to filter the 3D points further comprises retaining NIR keypoints with corresponding 3D points from the 3D point cloud data.
Wherein having ZSIROS’s automatic sensor calibration system wherein to filter the 3D points further comprises retaining NIR keypoints with corresponding 3D points from the 3D point cloud data.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and BRALEY relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while BRALEY by using the alignment system, the on-board system of the vehicle can generate planning decisions that plan the future trajectory of the vehicle and enable the vehicle to operate more safely and efficiently. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and BRALEY et al. (US 20210012166 A1), Paragraph [0024].
Regarding claim 17, ZSIROS in view of TANG and further in view of KROEGER explicitly teach the non-transitory computer program product of claim 15,
ZSIROS in view of TANG and further in view of KROEGER fail to explicitly teach wherein to filter the 3D points further comprises retaining NIR keypoints with corresponding 3D points from the 3D point cloud data.
However, BRALEY explicitly teaches wherein to filter the 3D points further comprises retaining NIR keypoints with corresponding 3D points from the 3D point cloud data (Fig. 2. Paragraph [0107]-BRALEY discloses the system identifies a set of multiple region pairs using the embeddings of the image regions and the embeddings of the point cloud regions (508) Each region pair specifies an image region and a point cloud region that characterize the same area of the environment. To identify the region pairs, the system uses a matching algorithm (e.g., a nearest neighbor matching algorithm) to identify a set of multiple embedding pairs, each of which specifies an embedding of an image region and an embedding of a point cloud region. The system uses each embedding pair to identify a respective region pair that specifies the image region and the point cloud region corresponding to the embedding pair (wherein a region pair represents a retained NIR keypoint with a corresponding 3D point).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG and further in view of KROEGER of a non-transitory computer program product comprising a computer-readable storage medium including program code for automatic multi-modality sensor calibration with near infrared images (NIR), wherein the program code when executed on a computer causes the computer to: detect image keypoints from collected images and NIR keypoints from NIR; filter three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimize an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of BRALEY of wherein to filter the 3D points further comprises retaining NIR keypoints with corresponding 3D points from the 3D point cloud data.
Wherein having ZSIROS’s automatic sensor calibration system wherein to filter the 3D points further comprises retaining NIR keypoints with corresponding 3D points from the 3D point cloud data.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and BRALEY relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while BRALEY by using the alignment system, the on-board system of the vehicle can generate planning decisions that plan the future trajectory of the vehicle and enable the vehicle to operate more safely and efficiently. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and BRALEY et al. (US 20210012166 A1), Paragraph [0024].
Claims 5, 12, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over ZSIROS et al. (US 20250363666 A1), hereinafter referenced as ZSIROS, in view of TANG et al. (US 20210326601 A1), hereinafter referenced as TANG, and further in view of KROEGER (US 10298910 B1), hereinafter referenced as KROEGER (US 10298910 B1), and further in view PARIAN et al. (US 20220128671 A1), hereinafter referenced as PARIAN.
Regarding claim 5, ZSIROS in view of TANG and further in view of KROEGER explicitly teach the computer-implemented method of claim 1,
ZSIROS in view of TANG and further in view of KROEGER fail to explicitly teach wherein filtering the 3D points further comprises employing bilinear interpolation to approximate 3D points from sub-pixel keypoints from the NIR keypoints.
However, PARIAN explicitly teaches wherein filtering the 3D points further comprises employing bilinear interpolation to approximate 3D points from sub-pixel keypoints from the NIR keypoints (Fig. 9. Paragraph [0090]-PARIAN discloses the control image 1000, which is captured by the internal camera 112, is mapped with the 3D point cloud. The 3D coordinate of the matched feature is estimated by identifying the pixel/sub-pixel where the matched feature maps. As noted earlier, if the matched feature maps to a sub-pixel, the surrounding coordinates are used to perform the bilinear interpolation to determine the 3D coordinate of the matched feature (wherein the matched feature is a keypoint).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG and further in view of KROEGER of a computer-implemented method for automatic multi-modality sensor calibration with near-infrared images (NIR), comprising: detecting image keypoints from collected images and NIR keypoints from NIR; filtering three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimizing an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of PARIAN of wherein filtering the 3D points further comprises employing bilinear interpolation to approximate 3D points from sub-pixel keypoints from the NIR keypoints.
Wherein having ZSIROS’s automatic sensor calibration system wherein filtering the 3D points further comprises employing bilinear interpolation to approximate 3D points from sub-pixel keypoints from the NIR keypoints.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and PARIAN relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while PARIAN the measured distance and two angles enable a processor in the device to determine the 3D coordinates of the target. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and PARIAN et al. (US 20220128671 A1), Paragraph [0024].
Regarding claim 12, ZSIROS in view of TANG and further in view of KROEGER explicitly teach the system of claim 8,
ZSIROS in view of TANG and further in view of KROEGER fail to explicitly teach wherein to filter the 3D points further comprises employing bilinear interpolation to approximate 3D points from sub-pixel keypoints from the NIR keypoints.
However, PARIAN explicitly teaches wherein to filter the 3D points further comprises employing bilinear interpolation to approximate 3D points from sub-pixel keypoints from the NIR keypoints (Fig. 9. Paragraph [0090]-PARIAN discloses the control image 1000, which is captured by the internal camera 112, is mapped with the 3D point cloud. The 3D coordinate of the matched feature is estimated by identifying the pixel/sub-pixel where the matched feature maps. As noted earlier, if the matched feature maps to a sub-pixel, the surrounding coordinates are used to perform the bilinear interpolation to determine the 3D coordinate of the matched feature (wherein the matched feature is a keypoint).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG and further in view of KROEGER of a system for automatic multi-modality sensor calibration with near-infrared images (NIR), comprising: a memory device; detect image keypoints from collected images and NIR keypoints from NIR; filter three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimize an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of PARIAN of wherein to filter the 3D points further comprises employing bilinear interpolation to approximate 3D points from sub-pixel keypoints from the NIR keypoints.
Wherein having ZSIROS’s automatic sensor calibration system wherein to filter the 3D points further comprises employing bilinear interpolation to approximate 3D points from sub-pixel keypoints from the NIR keypoints.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and PARIAN relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while PARIAN the measured distance and two angles enable a processor in the device to determine the 3D coordinates of the target. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and PARIAN et al. (US 20220128671 A1), Paragraph [0024].
Regarding claim 18, ZSIROS in view of TANG and further in view of KROEGER explicitly teach the non-transitory computer program product of claim 15,
ZSIROS in view of TANG and further in view of KROEGER fail to explicitly teach wherein to filter the 3D points further comprises employing bilinear interpolation to approximate 3D points from sub-pixel keypoints from the NIR keypoints.
However, PARIAN explicitly teaches wherein to filter the 3D points further comprises employing bilinear interpolation to approximate 3D points from sub-pixel keypoints from the NIR keypoints (Fig. 9. Paragraph [0090]-PARIAN discloses the control image 1000, which is captured by the internal camera 112, is mapped with the 3D point cloud. The 3D coordinate of the matched feature is estimated by identifying the pixel/sub-pixel where the matched feature maps. As noted earlier, if the matched feature maps to a sub-pixel, the surrounding coordinates are used to perform the bilinear interpolation to determine the 3D coordinate of the matched feature (wherein the matched feature is a keypoint).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG and further in view of KROEGER of a non-transitory computer program product comprising a computer-readable storage medium including program code for automatic multi-modality sensor calibration with near infrared images (NIR), wherein the program code when executed on a computer causes the computer to: detect image keypoints from collected images and NIR keypoints from NIR; filter three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimize an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of PARIAN of wherein to filter the 3D points further comprises employing bilinear interpolation to approximate 3D points from sub-pixel keypoints from the NIR keypoints.
Wherein having ZSIROS’s automatic sensor calibration system wherein to filter the 3D points further comprises employing bilinear interpolation to approximate 3D points from sub-pixel keypoints from the NIR keypoints.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and PARIAN relate to calibrating sensors employed by autonomous entities, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while PARIAN the measured distance and two angles enable a processor in the device to determine the 3D coordinates of the target. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and PARIAN et al. (US 20220128671 A1), Paragraph [0024].
Claims 6, 13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over ZSIROS et al. (US 20250363666 A1), hereinafter referenced as ZSIROS, in view of TANG et al. (US 20210326601 A1), hereinafter referenced as TANG, and further in view of KROEGER (US 10298910 B1), hereinafter referenced as KROEGER (US 10298910 B1), and further in view LIAO et al. (US 20230126366 A1), hereinafter referenced as LIAO.
Regarding claim 6, ZSIROS in view of TANG and further in view of KROEGER explicitly teach the computer-implemented method of claim 1, Although ZSIROS teaches filtered NIR-to-3D points.
ZSIROS in view of TANG and further in view of KROEGER fails to explicitly teach wherein optimizing the extrinsic calibration further comprises minimizing the reprojection error between projections of the filtered NIR-to-3D points to an image plane and their corresponding image keypoints using a perspective-n-point module that employs random sample consensus (RANSAC) outlier removal.
However, LIAO explicitly teaches wherein optimizing the extrinsic calibration further comprises minimizing the reprojection error between projections of the filtered NIR-to-3D points to an image plane and their corresponding image keypoints using a perspective-n-point module that employs random sample consensus (RANSAC) outlier removal (Figs. 2A-B. Paragraph [0120]-LIAO discloses in [JDV+13] the 2D-to-3D point correspondences are obtained from the inherent relationship between the real camera's 2D features and their matches on the virtual image (created by projecting the map points in prior map onto a plane using the previously localized pose of the real camera). Then, the well-known perspective-n-point (PnP) problem is solved to find the relative pose between the real and the virtual cameras. The projection error is minimized by using random sample consensus (RANSAC).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG and further in view of KROEGER of a computer-implemented method for automatic multi-modality sensor calibration with near-infrared images (NIR), comprising: detecting image keypoints from collected images and NIR keypoints from NIR; filtering three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimizing an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of LIAO of wherein optimizing the extrinsic calibration further comprises minimizing the reprojection error between projections of the points to an image plane and their corresponding image keypoints using a perspective-n-point module that employs random sample consensus (RANSAC) outlier removal.
Wherein having ZSIROS’s automatic sensor calibration system wherein optimizing the extrinsic calibration further comprises minimizing the reprojection error between projections of the filtered NIR-to-3D points to an image plane and their corresponding image keypoints using a perspective-n-point module that employs random sample consensus (RANSAC) outlier removal.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and LIAO compute extrinsic calibration using 3D data, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while LIAO there is need provide a solution to the second type of pose estimation problems, which is camera relocalization, i.e. estimating a camera pose in real-time in the same or similar environment. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and LIAO et al. (US 20230126366 A1), Paragraph [0004].
Regarding claim 13, ZSIROS in view of TANG and further in view of KROEGER explicitly teach the system of claim 8, Although ZSIROS teaches filtered NIR-to-3D points.
ZSIROS in view of TANG and further in view of KROEGER fails to explicitly teach wherein to optimize the extrinsic calibration further comprises to minimize the reprojection error between projections of the filtered NIR-to-3D points to an image plane and their corresponding image keypoints using a perspective-n-point module that employs random sample consensus (RANSAC) outlier removal.
However, LIAO explicitly teaches wherein to optimize the extrinsic calibration further comprises to minimize the reprojection error between projections of the filtered NIR-to-3D points to an image plane and their corresponding image keypoints using a perspective-n-point module that employs random sample consensus (RANSAC) outlier removal (Figs. 2A-B. Paragraph [0120]-LIAO discloses in [JDV+13] the 2D-to-3D point correspondences are obtained from the inherent relationship between the real camera's 2D features and their matches on the virtual image (created by projecting the map points in prior map onto a plane using the previously localized pose of the real camera). Then, the well-known perspective-n-point (PnP) problem is solved to find the relative pose between the real and the virtual cameras. The projection error is minimized by using random sample consensus (RANSAC).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG and further in view of KROEGER of a system for automatic multi-modality sensor calibration with near-infrared images (NIR), comprising: a memory device; detect image keypoints from collected images and NIR keypoints from NIR; filter three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimize an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of LIAO of wherein to optimize the extrinsic calibration further comprises to minimize the reprojection error between projections of the points to an image plane and their corresponding image keypoints using a perspective-n-point module that employs random sample consensus (RANSAC) outlier removal.
Wherein having ZSIROS’s automatic sensor calibration system wherein to optimize the extrinsic calibration further comprises to minimize the reprojection error between projections of the points to an image plane and their corresponding image keypoints using a perspective-n-point module that employs random sample consensus (RANSAC) outlier removal.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and LIAO compute extrinsic calibration using 3D data, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while LIAO there is need provide a solution to the second type of pose estimation problems, which is camera relocalization, i.e. estimating a camera pose in real-time in the same or similar environment. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and LIAO et al. (US 20230126366 A1), Paragraph [0004].
Regarding claim 19, ZSIROS in view of TANG and further in view of KROEGER explicitly teach the non-transitory computer program product of claim 15, Although ZSIROS teaches filtered NIR-to-3D points.
ZSIROS in view of TANG and further in view of KROEGER fails to explicitly teach wherein to optimize the extrinsic calibration further comprises to minimize the reprojection error between projections of the filtered NIR-to-3D points to an image plane and their corresponding image keypoints using a perspective-n-point module that employs random sample consensus (RANSAC) outlier removal.
However, LIAO explicitly teaches wherein to optimize the extrinsic calibration further comprises to minimize the reprojection error between projections of the filtered NIR-to-3D points to an image plane and their corresponding image keypoints using a perspective-n-point module that employs random sample consensus (RANSAC) outlier removal (Figs. 2A-B. Paragraph [0120]-LIAO discloses in [JDV+13] the 2D-to-3D point correspondences are obtained from the inherent relationship between the real camera's 2D features and their matches on the virtual image (created by projecting the map points in prior map onto a plane using the previously localized pose of the real camera). Then, the well-known perspective-n-point (PnP) problem is solved to find the relative pose between the real and the virtual cameras. The projection error is minimized by using random sample consensus (RANSAC).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of ZSIROS in view of TANG and further in view of KROEGER of a non-transitory computer program product comprising a computer-readable storage medium including program code for automatic multi-modality sensor calibration with near infrared images (NIR), wherein the program code when executed on a computer causes the computer to: detect image keypoints from collected images and NIR keypoints from NIR; filter three dimensional (3D) points from 3D point cloud data based on corresponding 3D points from the NIR keypoints (NIR-to-3D points) to obtain filtered NIR-to-3D points; optimize an extrinsic calibration based on a reprojection error computed from the filtered NIR-to-3D points to obtain an optimized extrinsic calibration for an autonomous entity control system; and with the teachings of PARIAN of wherein to optimize the extrinsic calibration further comprises to minimize the reprojection error between projections of the points to an image plane and their corresponding image keypoints using a perspective-n-point module that employs random sample consensus (RANSAC) outlier removal.
Wherein having ZSIROS’s automatic sensor calibration system wherein to optimize the extrinsic calibration further comprises to minimize the reprojection error between projections of the filtered NIR-to-3D points to an image plane and their corresponding image keypoints using a perspective-n-point module that employs random sample consensus (RANSAC) outlier removal.
The motivation behind the modification would have been to obtain an automatic sensor calibration system that enhances the reliability and accuracy of calibrated sensors. Since both ZSIROS and LIAO compute extrinsic calibration using 3D data, wherein ZSIROS a probability of an erroneous calibration can be reduced and an overall reliability of the calibration is improved, while LIAO there is need provide a solution to the second type of pose estimation problems, which is camera relocalization, i.e. estimating a camera pose in real-time in the same or similar environment. Please see ZSIROS et al. (US 20250363666 A1), Paragraph [0009], and LIAO et al. (US 20230126366 A1), Paragraph [0004].
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant’s disclosure.
SEN et al. (US 20240161341 A1) – In various examples, sensor configuration for autonomous or semi-autonomous systems and applications is described. Systems and methods are disclosed that may use image feature correspondences between camera images along with an assumption that image features are locally planar to determine parameters for calibrating an image sensor with a LiDAR sensor and/or another image sensor. In some examples, an optimization problem is constructed that attempts to minimize a geometric loss function, where the geometric loss function encodes the notion that corresponding image features are views of a same point on a locally planar surface (e.g., a surfel or mesh) that is constructed from LiDAR data generated using a LiDAR sensor. …Abstract, Figs. 8-10
BURLINA et al. (US 12525025 B1) - Techniques for determining a presence of an object, especially an object such as animal or debris, in a path of a vehicle, are discussed herein. For example, sensors of various modalities, which may include multispectral sensors, may capture data representing an environment the vehicle is traversing. In examples, one or more trained machine learned (ML) models, operating on a vehicle computing system, may detect and/or classify objects in the environment, based on input data of one or more modalities or spectral bands. The ML models may be pre-trained using training data including real sensor data, synthetic data, and/or augmented data, along with auto-generated annotations. …Abstract, Fig. 2.
XU et al. (US 20220005154 A1) – Provided is a method for processing a point cloud, including: acquiring a first point cloud; acquiring an image of a same environment corresponding to the first point cloud; resampling the first point cloud to obtain a second point cloud with a density lower than that of the first point cloud; registering the second point cloud and the image; and processing the second point cloud according to the image to generate a target point cloud containing a color. Since a color of the image shot by a camera may reflect a real driving environment, the above-mentioned solution may provide a more intuitive visual point cloud for a driver to facilitate the driver performing certain complex operations including backing, lane changing, overtaking, and the like, based on the target point cloud…Abstract, Fig. 2.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ETHAN N WOLFSON whose telephone number is (571)272-1898. The examiner can normally be reached Monday - Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ETHAN N WOLFSON/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights