DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 9/4/2025 was filed after the mailing date of the claims on 6/10/2025. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
1. Claim(s) 1, 4, 7, 8, 9, 11, 12, and 13 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent 12,315,197 Miao et al. (hereinafter Miao) in view of U.S. Patent 11,360,197 Sutavani et al. (hereinafter Sutavani).
2. Regarding Claim 1, Miao discloses A method for calibrating the relative orientation of a spatial sensor and an area sensor, wherein the measuring zones of the spatial sensor and the area sensor are overlapping (Col. 1 lines 17-18, “On an AV, camera calibration is used to align the position among cameras, other sensors, and the vehicle body.” Col. 4 lines 45-47, “Each of the sensors is configured to generate data pertaining to objects 110, 111 that are within a range of detection of the sensors.”-i.e., LiDAR (spatial sensor) and camera (area sensor) with overlapping fields of detection. Abstract, “using image frames captured by a camera of the AV and LIDAR point clouds captured by a LIDAR system of the AV”), the method comprising
receiving, by a computing system, a plurality of sensor frames of the spatial sensor
and the area sensor (Col. 5 lines 1-5, “the calibration module calculates a vehicle pose calibration metric that can be used to validate camera calibration. In this process, two camera frames and LIDAR point cloud data, along with a transformation of the vehicle pose, are analyzed.” Claim 1, “using a plurality of image frames captured by a camera of an AV and LIDAR point clouds captured by a LIDAR system of the AV to calculate an AV pose calibration metric.”);
grouping, by the computing system, sensor frames of the spatial sensor with sensor
frames of the area sensor based on temporal correlation (Col. 5 lines 3-13, “two camera frames…will be consecutive frames, which in this disclosure means that in a sequence of captured image frames, the frames are either immediately consecutive to each other (i.e., adjacent) or nearly-consecutive to each other (in which “nearly” means that the frames are captured no more than a very small threshold of time from each other, such as no more than 0.5 seconds, or 0.1 seconds, away from each other).” Col. 6 lines 8-11, “The LIDAR point cloud is generated at a time that overlaps with or is substantially close to the first time. The calibration module will receive the LIDAR point cloud data.” This temporal co-registration of LIDAR frames with camera frames based on proximity in time constitutes grouping based on temporal correlation);
selecting, by the computing system, at least one group of frames comprising a spatial sensor frame and an area sensor frame (Claim 2, “receiving a first image frame that the camera captured in a sequence at a first time point; receiving a second image frame that the camera captured at a second time point that is consecutive to the first time point in the sequence as the AV moves in its environment… receiving LIDAR point cloud data that the LIDAR system captured at the first time point or the second time point.” Col. 6 lines 14-18, “the calibration module may include a feature selection module 405 configured to analyze points in the LIDAR point cloud to identify a feature 407 (i.e., an area) within the LIDAR point cloud that corresponds to the location of a particular object”);
receiving, by the computing system, annotations for the sensor frames in the at least
one selected group, wherein the annotations comprise bounding boxes for detected objects (Col. 5 lines 29-37, “The calibration module further includes an object detection module configured to generate a distance calibration metric that can be used to validate camera calibration using object detection information. According to various embodiments, and as shown for example in FIG. 3, the object detection module will look for a particular object in the image, such as a traffic light, and it will project a three-dimensional (3D) bounding box 305 and a two-dimensional (2D) bounding box 310, onto a traffic light or other object.” Col. 7 lines 6-14, “at 450 a 3D object bounding box is generated. The 3D bounding box labels an object in the image. According to various embodiments, generating the 3D object bounding box includes determining a position of the object based on a calculated position and pose of the AV. This may be done automatically, by human labeling, or another process. At 460, the system analyzes the image to automatically generate a 2D object bounding box surrounding the object within the image”);
projecting, by the computing system, the corners of a three-dimensional bounding box
in the spatial sensor frame to an image plane of the area sensor, to produce a projected rectangle (Col. 7 lines 14-16, “At 465, the 3D object bounding box is projected over the 2D object bounding box.” Fig. 4B: shows the flow: Capture Data-> 3D Bounding Box (450)-> Project to Camera Frame (465)-> Calculate Distance Metric (470). Col. 5 lines 53-57, “The system will project the 3D bounding box onto the camera frame 300, as shown in FIG. 3, and the system will generate a distance calibration metric 315 that is a measure of distance from the projected 3D bounding box 305 to the corresponding 2D bounding box 310.”)
calculating, by the computing system, an energy term based on the coordinates of the
corners of the projected rectangle and the corresponding bounding box in the area sensor
frame (Col. 7 lines 16-23, “at 470, a distance between the 3D object bounding box and the 2D object bounding box is calculated. This distance equates to a distance calibration metric. The system may calculate the distance metric using any suitable process. For example, the system may calculate the distance as the difference between the location of the center of the projected 3D bounding box and the center of the 2D bounding box.” Abstract, “measuring a distance metric between a three-dimensional bounding box around an object and a two-dimensional bounding box in an image captured by the camera.”); and
optimizing, by the computing system, the relative sensor orientation by minimizing
the energy term (Col. 7 lines 24-29, “Referring to FIG. 4C, after computing the two metrics described above (the pose calibration metric and the distance calibration metric), at 475 the system may use the two metrics to generate a confidence score, which is an assessment of confidence in the accuracy of the calibration of the camera.” Col. 7 lines 40-49, “At 480, the system may determine whether the confidence score is above or below a threshold….If the confidence score is below the threshold, then, at 490, the system may consider the sensor (e.g., camera) be not calibrated.” Col. 7 lines 50-66, “When the system determines that the camera is not calibrated, then in response the system will generate a signal that will result in an action….The action may include recalibrating the sensor, altering a trajectory of the AV, or altering a velocity of the AV.”).
However, Miao uses the distance metric only to detect miscalibration and generate a signal- it does not iteratively optimize the relative sensor orientation by minimizing the energy term.
Sutavani teaches optimizing, by the computing system, the relative sensor orientation by minimizing the energy term (Col. 3 lines 53-59, “The computer vision system can generate candidate transformation parameters corresponding to translation (x, y, z) and rotation (yaw, pitch, roll) in a 3D space. The computer vision system can apply the transformation parameters to one of the sets of sensor data in an attempt to bring the set of sensor data, or a subset of the sensor data, into the frame of reference of the other set of sensor data.” Col. 3 line 66-Col. 4 line7, “The computer vision system can apply these steps as a part of an iterative optimization technique so as to identify a set of transformation parameters that yield a metric of MI above a certain threshold for example, or until another exit condition is satisfied (e.g., meeting or exceeding a computational/time cost threshold of further iterations, reaching a maximum number of optimization iterations, meeting or exceeding cost difference between a current iteration and a previous iteration).” Iterative optimization specifically for LIDAR-camera-sensor pairs at Fig. 13, steps 862-878: iteratively transforming LIDAR point cloud data according to transformation parameters (step 862), computing mutual information (step 874), checking exit conditions (step 876), and updating transformation parameters (step 878) until convergence- directly teaching optimizing the relative sensor orientation by minimizing a cost metric between the spatial and area sensor data.)
It would have been obvious to one of ordinary skill in the art to incorporate the iterative optimization framework of Sutavani into the method of Miao. Miao detects miscalibration using distance metric between projected 3D and 2D bounding boxes but provides no mechanism to correct it. Sutavani provides precisely that missing correction mechanism, an iterative technique for optimizing sensor transformation parameters- including translation and rotation- by minimizing a cost metric between LIDAR and camera data, which is the identical sensor pairing and calibration problems addressed by Miao. One of ordinary skill would have been motivated to combine the two references to correct, rather than merely detect, sensor misalignment, with a reasonable expectation of success.
3. Regarding Claim 4, Miao in view of Sutavani discloses The method according to claim 1,
Miao teaches wherein the annotations for at least one group of frames are produced by a human annotator. (Col. 7 lines 6-11, The 3D bounding box labels an object in the image…This may be done automatically, by human labeling, or another process). Claimed in the alternative (neural network trained on spatial sensor frames and/or a neural network trained on area sensor frames).
4. Regarding Claim 7, Miao in view of Sutavani discloses The method according to claim 1,
Claim 7 is a well-known least squares formulation algorithm:
The exact formula for this energy term, which computes the sum of squared differences, can be formalized as:
\(E(v) = v(x_0 - x_0')^2 + v(y_0 - y_0')^2 + V(x_1 - x_1')^2 + V(y_1 - y_1')^2\)
Where:
\((x_0, y_0, x_1, y_1)\) are the corner coordinates of the extracted 2D bounding box.
\((x_0', y_0', x_1', y_1')\) are the corresponding corner coordinates of the rectangle projected from your 3D or object model.
\(v\) and \(V\) are weight or variance terms that dictate the importance or confidence associated with specific corners.
wherein for corner coordinates x0, y0, x1, y1 of the bounding box and for corner coordinates x0', y0', x1', y1' of the corresponding projected rectangle, the energy term is
PNG
media_image1.png
68
837
media_image1.png
Greyscale
5. Regarding Claim 8, Miao in view of Sutavani discloses The method according to claim 1,
Sutavani discloses wherein optimizing the relative sensor orientation is performed iteratively until a convergence criterion is met (Col. 3 line 66-col. 4 line 7, “The computer vision system can apply these steps as a part of an iterative optimization technique so as to identify a set of transformation parameters that yield a metric of MI above a certain threshold for example, or until another exit condition is satisfied (e.g., meeting or exceeding a computational/time cost threshold of further iterations, reaching a maximum number of optimization iterations, meeting or exceeding cost difference between a current iteration and a previous iteration).” Fig. 13, steps 876-878: Exit Condition(s) Met?-> if NO-< Update Transformation Parameters-> loop back. The exit condition at step 626/876 is precisely the convergence criterion).
6. Regarding Claim 9, Miao in view of Sutavani discloses The method according to claim 8, wherein the relative sensor orientation is described using a matrix describing rotation and translation of a sensor in three-dimensional space, and wherein during or subsequent to each optimization iteration, a factorization algorithm is applied to the rotation matrix for ensuring orthonormality.
Examiner takes Official Notice that representing relative orientation as a 3x3 rotation matrix, and applying a factorization technique (e.g., SVD, QR, Gram-Schmidt) during or after each iteration of an optimization to restore orthonormality, was well known prior to the effective filing date. Iterative numerical updates to a rotation matrix routinely violate its orthonormality constraints (orthogonal unit-norm columns, determinant +1), and periodic re-orthonormalization via factorization is the standard remedy, allowing optimization to continue while maintaining a physically valid rotation estimate.
7. Claim 12 is a non-transitory CRM claim, rejected with respect to the same limitations rejected in method claim 1.
8. Claim 13 is a computer system claim, rejected with respect to the same limitations rejected in method claim 1.
Miao: Processor: Central Processing Unit (CPU) 506”
Miao: Working Memory: “memory 512”
Miao: Display: “display 554”
Miao: Input Device: “keyboard (physical and/or touch) 550)”
Miao: Non-volatile memory: “Disk drive unit 516 comprising a computer-readable storage medium 518 on which is stored one or more sets of instructions 520” Col. 8 lines 53-56.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
9. Claim(s) 2 are rejected under 35 U.S.C. 103 as being unpatentable over Miao in view of Sutavani as applied to claim 1 above, and further in view of U.S. Patent 11676307, Gong et al. (hereinafter Gong).
10. Regarding Claim 2, Miao in view of Sutavani discloses The method according to claim 1,
Miao discloses wherein the plurality of sensor frames are consecutive (Col. 5 lines 3-9, “two camera frames…will be consecutive frames, which in this disclosure means that in a sequence of captured image frames, the frames are either immediately consecutive to each other (i.e., adjacent) or nearly-consecutive to each other”), and
Sutavani discloses (Col. 4 lines 28-31, 28-31, “the computer vision system in some implementations obtains sensor data for the calibration process outlined above, only during periods of time when the vehicle is stationary.”- teaching that only a subset of available frames need be used for calibration optimization rather than all frames).
However, neither reference, alone or in combination teaches or suggest wherein a limited number of groups of frames is selected for optimizing the relative sensor orientation.
Further, Gong teaches a limited number of groups of frames is selected for optimizing the relative sensor orientation (Claim 1, “selecting, at the vehicle as the vehicle travels, at least a subset of the LIDAR scans and at least a subset of the camera images for calibration; computing… LIDAR-to-camera transformations for the subset of the LIDAR scans and the subset of camera images using an optimization algorithm.” Further discloses the basis for narrowing the full stream of captured frames down to the selected subset, e.g., Col. 42 lines 55-560, “the subset of the LIDAR scans and the camera images may be determined to be useful for calibration based on detecting straight line features in the LIDAR scans by separating ground points from non-ground points, profiling intensity of the ground points, and detecting intensity edges in the ground points.” And alternatively, Col. 42 lines 63-66, “based on dividing a field of view of the LIDAR scans into multiple regions, and determining that each of the regions has sufficient straight line features to be useful for calibration.”).
One of ordinary skill would have been motivated to apply this frame-selection step to the Miao/Sutavani combination to improve calibration accuracy and computational efficiency, by ensuring the optimization in Sutavani operates only on a limited number of well-conditioned frame groups rather than indiscriminately on every grouped/consecutive frame pair, with a reasonable expectation of success since both deal with the same LIDAR-to-camera transformation problem.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
11. Claim(s) 6 are rejected under 35 U.S.C. 103 as being unpatentable over Miao in view of Sutavani as applied to claim 1 above, and further in view of U.S. Patent 10298910, Kroeger.
12. Regarding Claim 6, Miao in view of Sutavani discloses The method according to claim 1,
Further, Kroeger claims wherein the intrinsic parameters of the area sensor are kept constant when optimizing the relative orientation (Claim 1, “determining first errors associated with the plurality of point pairs, the first errors comprising, at least in part, a first distance between the first point and an epipolar line corresponding to the second point; determining, based at least in part on the first errors, a first subset of the plurality of point pairs; determining, from the first subset of the plurality of point pairs, a first correction function representative of a misalignment of an estimated relative pose of the cameras… calibrating the plurality of cameras, based at least”).
One of ordinary skill would have been motivated to hold the camera’s intrinsic parameters constant during the Miao/Sutavani relative-orientation optimization because intrinsic and extrinsic parameters describe different physical quantities and are conventionally calibrated separately, as Kroeger confirms; fixing already-known intrinsics reduces the optimization to the extrinsic degrees of freedom alone, improving convergence stability and computational efficiency, with a reasonable expectation of success since Kroeger addresses the same general class of sensor relative pose calibration problem.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
13. Claim(s) 3 are rejected under 35 U.S.C. 103 as being unpatentable over Miao in view of Sutavani further in view of Gong as applied to claim 2 above, and further in view of U.S. Patent 12238256 James.
Regarding Claim 3, Miao in view of Sutavani further in view of Gong discloses The method according to claim 2,
Miao in view of Sutavani further in view of Gong does not explicitly disclose wherein the selected groups of frames comprise one group at the beginning of the sequence of consecutive frames and one group at the end of the sequence of consecutive frames.
Further, James teaches wherein the selected groups of frames comprise one group at the beginning of the sequence of consecutive frames and one group at the end of the sequence of consecutive frames (Col. 24 lines 21-41, “the calibration factor that is calculated by comparing color values measured from the calibration slate 10 appearing in the still image 250 to known color values associated with the color chart 16 of the slate 10 may then be applied to captured colors of all of the frames 248 in a sequence 246 to color calibrate the video 244. In another embodiment, the present method may include a dynamic calibration method in which a calibration slate 10 is used to color calibrate one or more frames 248 in which the slate 10 appears and then the one or more color calibrated frames 248 are used instead of the calibration slate 10 to color calibrate additional frames 248 of a sequence 246. In this embodiment, a calibration slate 10 appears in a still image 250 of a first frame 248a of the sequence 246 of frames of a captured video 242. The first frame 248a of the sequence that is calibrated may be at any position within the sequence, such as at the beginning of the sequence, as shown in FIG. 24, or at the end of the sequence, as shown in FIG. 24A. The first frame 248a is calibrated using the calibration slate 10 in accordance with the present calibration method as previously described.”).
One of ordinary skill would have been motivated to select reference frame groups from both positions taught by James rather than only one to bound the calibration with data from across the full span of the sequence, improving robustness against drift or pose changes that accumulate over a driving segment, with a reasonable expectation of success since James, Miao, and Sutavani all address calibration using reference data selected from a sequence of captured frames.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
14. Claim(s) 5 are rejected under 35 U.S.C. 103 as being unpatentable over Miao in view of Sutavani as applied to claim 1 above, and further in view of U.S. Patent 12205329 El Dokor et al. (hereinafter El Dokor).
15. Regarding Claim 5, Miao in view of Sutavani discloses The method according to claim 1,
Miao in view of Sutavani does not explicitly disclose wherein for a group of frames comprising multiple spatial sensor frames and area sensor frames and/or for a group of spatial sensor frame and area sensor frame comprising multiple annotated objects, optimizing the relative sensor orientation is based on the sum of the energy terms for all annotated objects in the selected group.
Further, El Dokor teaches wherein for a group of frames comprising multiple spatial sensor frames and area sensor frames and/or for a group of spatial sensor frame and area sensor frame comprising multiple annotated objects (Claim 1, “detecting one or more calibration features in each image set by a processor.” Col. 3 lines 59-62, “Including multiple patterns on the same target is a method of obtaining additional high accuracy calibration features from a single image.”), optimizing the relative sensor orientation (Col. 1 lines 56-60, “The extrinsic parameters are ones that define the distance and orientation relationships (degrees of freedom) between the various camera nodes in a stereo rig. These parameters include rotation parameters, like pitch, yaw, and roll.” Col. 1 lines 63-64, “Calibration aims at estimating such parameters based on a series of observations.” Claim 1, “adjusting the calibration parameters if it is determined that that the combination of the aggregated errors has not been minimized.” is based on (Col. 5 lines 23-24, “The camera parameters are adjusted using an optimization algorithm until the cost has been minimized.”) the sum of the energy terms (Col. 8 lines 16-20, “The reprojection error cost term recovers pitch and roll by minimizing: ε=.Math.j=1J.Math.i=1I.Math.pL,i,j-mL,i,j.Math.2+.Math.pR,i,j-mR,i,j. Equation 10, an explicit summation of a per-feature squared-error/energy term. Claim 1, “aggregating the dimensioning errors and the reprojection errors.” Claim 2, “computing an individual cost term for each of the identified relevant calibration parameters.”) for all annotated objects in the selected group (Col. 6 lines 33, “Given J tracked features…” Col. 5 lines 39-45, “the cost of each dataset is determined by a weighted (α) combination of K terms which target individual calibration parameters:
εn2=1N.Math.Kk=1αkεk2 Equation 2. The sum spans all features/objects identified within the selected image set/group, not a single feature in isolation).
One of ordinary skill would have been motivated to apply El Dokor’s known technique of summing a per-feature energy term across multiple detected features to Miao’s per-object distance metric when a selected frame group contains multiple annotated objects, producing a more robust, averaged calibration signal, with a reasonable expectation of success since El Dokor, Miao and Sutavani all address optimizing sensor orientation by minimizing an error term derived from detected feature correspondence.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
16. Claim(s) 10 are rejected under 35 U.S.C. 103 as being unpatentable over Miao in view of Sutavani as applied to claim 1 above, and further in view of U.S. Patent 2024/0161514, Choi et al. (hereinafter Choi).
17. Regarding Claim 10, Miao in view of Sutavani discloses The method according to claim 1,
Miao in view of Sutavani does not explicitly disclose wherein for data sets comprising additional sensor data, additional groups of spatial sensor frames and area sensor frames are selected based on the acceleration in at least one space axis.
Further, Choi teaches wherein for data sets comprising additional sensor data, additional groups of spatial sensor frames and area sensor frames are selected based on the acceleration in at least one space axis ([0167], “the processor 110 may determine whether the vehicle moves in the up and down direction when it passes a speed bump… calculate the vertical movement range between the feature points of the first image frame and the feature points of the second image frame and obtain an average movement in the vertical direction of the feature points based on the calculated movement range.” [0189], “the vehicular electronic device may detect a bounding box 400 for identifying a speed bump from a captured image of the road in front of a vehicle… The vehicular electronic device may assess the proper operation of a speed bump detection function based on image frames included in the images captured while the vehicle approaches the location of the speed bump.” [0252]).
One of ordinary skill would have been motivated to combine the frame-selection technique of Choi with the Miao/Sutavani combination because it is known to select frames before/after a speed bump, which changes acceleration and includes vertical movement, with a reasonable expectation of success since both approaches address selecting additional frame groups based on a vertical-axis motion event of the vehicle.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
18. Claim(s) 11 are rejected under 35 U.S.C. 103 as being unpatentable over Miao in view of Sutavani as applied to claim 1 above, and further in view of U.S. Patent 10531004, Wheeler et al. (hereinafter Wheeler) further in view of U.S. Patent 12525013 Alokhina et al. (hereinafter Alokhina).
19. Regarding Claim 11, Miao in view of Sutavani discloses The method according to claim 1,
Miao teaches wherein the plurality of sensor frames are consecutive sensor frames (Col. 5 lines 3-9, “two camera frames…will be consecutive frames, which in this disclosure means that in a sequence of captured image frames, the frames are either immediately consecutive to each other (i.e., adjacent) or nearly-consecutive to each other”);
wherein the method further comprises:
projecting the corners of a three-dimensional bounding box in the space sensor frame to the image plane of the area sensor, to produce a projected rectangle (Col. 7 lines 14-16, “At 465, the 3D object bounding box is projected over the 2D object bounding box.” Fig. 4B: shows the flow: Capture Data-> 3D Bounding Box (450)-> Project to Camera Frame (465)-> Calculate Distance Metric (470). Col. 5 lines 53-57, “The system will project the 3D bounding box onto the camera frame 300, as shown in FIG. 3, and the system will generate a distance calibration metric 315 that is a measure of distance from the projected 3D bounding box 305 to the corresponding 2D bounding box 310.”),
Miao in view of Sutavani does not explicitly disclose using the projected rectangle as two-dimensional bounding box in the area sensor frame, and determining at least one attribute of the object
wherein the selected groups of frames represent a fraction of the received sequence
of consecutive frames; and
wherein all of the consecutive spatial sensor frames are labeled;
Further, Wheeler teaches wherein the selected groups of frames represent a fraction of the received sequence (Col. 17 lines 52-56, “The sensor calibration module 290 uses 1250 the selected frame if number of inliers is greater than a threshold value, otherwise the sensor calibration module 290 skips the frame and repeats the above steps by selecting 1210 another frame.” This same skip/select logic repeats at step 1370 for the second calibration pass.) of consecutive frames; and
wherein all of the consecutive spatial sensor frames are labeled (Wheeler discloses per-frame corner/edge labeling-e.g., Col. 18 lines 25-26, “detects 1320 2D points representing checkerboard corners.” And for LIDAR frames specifically: Col. 16 line8, “Detecting corners from lidar points” and later, the edgel based approach discloses determining an edge score for each point in a LIDAR scan (spatial sensor frame): Col. 31 56-65, “the edgel based calibration module 950 determines an edge score representing a degree of confidence with which the point corresponds to an edge…for each point above ground…for each point on the ground” This is a labeling operation applied across the points of each processed LIDAR frame;
One of ordinary skill would have been motivated to apply Wheeler’s frame-filtering technique- discarding frames that fail a quality/inlier threshold and retaining only a qualifying subset to the Miao/Sutavani combination in order to exclude low-quality frames from the calibration optimization, improving calibration accuracy and reducing unnecessary computation, with a reasonable expectation of success since Wheeler, Miao, and Sutavani all address calibration of LIDAR and camera sensor pairs using data selected from a sequence of captured frames.
Further, Alokhina teaches using the projected rectangle as two-dimensional bounding box in the area sensor frame (Claim 1, “determining one or more region proposals …comprises projecting three-dimensional (3D) points in a cuboid bounding an object onto two-dimensional (2D) points in the image based on parameters related to capture of the image by a camera” and Claim 5, “generating a region proposal for the object as a minimum bounding box for the 2D points in the image”)
determining at least one attribute of the object 9Col. 8 lines 50-53, “confidence scores 416 representing probabilities that each refined proposal belongs (or does not belong) to a set of object classes”-i.e., determining a class/attribute of the object from the (projection-derived) bounding box-via the refinement stage’s classification branch. Object class as an attribute: Col. 8 lines 14-15, “prelabels 430 include estimates of bounding boxes 418 for certain types of objects” and Col. 7 line 49, “assign labels to objects bounded by cuboids”
One of ordinary skill would have been motivated to use Miao’s projected rectangle directly as the 2D bounding box, as taught by Alokhina, to reduce the computational cost of running a separate 2D object-detection stage, with a reasonable expectation of success since Alokhina, Miao, and Sutavani all address generating object bounding boxes from projections of 3D sensor data onto 2D camera frames.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OMER KHALID whose telephone number is (571)270-5997. The examiner can normally be reached Monday- Friday 9am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Miller can be reached at (571) 272-7353. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OMER KHALID/Examiner, Art Unit 2422
/BRIAN P YENKE/Primary Examiner, Art Unit 2422
Prosecution Insights