DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Currently, claims 1-20 are pending in the application. Claims 1, 13, 14 & 20 are amended.
Response to Arguments / Amendments
Applicant’s arguments have been fully considered but are rendered moot in view of the new ground of rejection necessitated by amendments initiated by the applicant.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7, 9, 14-18 & 20 are rejected under 35 U.S.C. 103 as being unpatentable over HUANG et al. (US 20240010225, hereinafter HUANG) in view of RAJAN et al. (US 20230334673, hereinafter RAJAN) and Elogeel et al. ( US 20230194715 , hereinafter Elogeel)
Regarding Claim 1, HUANG discloses a method comprising:
forecasting a set of [[new]] subsequent positions of a set of objects in a geographic region based on a first set of object tracks of past positions to obtain a set of forecasted subsequent object positions ([0029], LIDAR sensor/camera performs representation learning for object detection from unlabeled point cloud sequences captured and detect moving object traces from temporally-ordered; [0031], planning a trajectory of ego vehicle according to objects within a scene surrounding the ego vehicle detected from the feature vectors; [0074] detect object traces by following a standard proposal-and-rejection framework; [0032] run-time engine 20 performs tracking of moving objects in subsequent point cloud sequences of a LIDAR camera stream; [0085] compute motion estimates for providing object cluster proposals with a first method accommodates point cloud sequences by optimizing for a spatiotemporal velocity field mapping any point location and a second method estimates scene flow between pairs of adjacent frames - a model is trained via self-supervised learning and the pre-trained model may be applied to each pair of adjacent frames in the sequence).
obtaining a new LiDAR point cloud ([0024], learning from unlabeled LIDAR point cloud sequences and recognize that moving objects are reliably detected from point cloud sequences such as a set of moving objects from a single LIDAR frame extracted from a point cloud sequence provides sufficient supervision for single-frame object detection; [0032] run-time engine 20 performs tracking of moving objects in subsequent point cloud sequences of a LIDAR camera stream);
processing, by a detector model, the new LiDAR point cloud to obtain a new set of bounding boxes around the set of objects detected in the new LiDAR point cloud ([0029], classify the set of moving objects extracted from on the moving object traces detected from the sequence of temporally-ordered and estimate 3D bounding boxes for the set of moving objects based on the classifying of the set of moving objects; [0062], using the information obtained from the LIDAR camera 458 to estimate bounding boxes using self-supervised learning based on detected object traces within LIDAR point clouds);
updating the first set of object tracks with the new set of bounding boxes ([0059], FIG. 3 & FIG. 4, representation learning and object detection system 300 may be configured for planning and control of an ego vehicle based on detected objects according to 3D bounding boxes estimated from pseudo labels of point cloud sequences from LIDAR camera sensors during operation of an ego vehicle);
filtering, after updating, the updated set of object tracks to remove object tracks that have a total track length less than a track length threshold, to generate a training set of object tracks ([0074] Then, a Kalman filter is applied to acquire object traces according to their motion. Next, a subset of object traces with smooth trajectories is collected; [0101], FIG. 4 & FIG. 8C, each 3D bounding box is represented by a 7-dimensional vector representing location, size, and orientation regarding the 3D bounding box estimation subtask in LIDAR object detection and a fraction of objects is densely captured by a LIDAR sensor while the remaining points represent portions of the objects For each object class);
selecting at least a subset of the new set of bounding boxes that are in the training set of object tracks ([0101], FIG. 4 & FIG. 8C, each 3D bounding box is represented by a 7-dimensional vector representing location, size, and orientation regarding the 3D bounding box estimation subtask in LIDAR object detection and a fraction of objects is densely captured by a LIDAR sensor while the remaining points represent portions of the objects for each object class); and
retraining the detector model using the at least the subset of the new set of bounding boxes ([0067], FIGS. 6A-6E, moving objects detected from input point cloud sequences are used to train self-supervised tasks for feature extraction learning for object detection from unlabeled point cloud sequences performed by a self-supervised learning framework).
HUANG does not explicitly disclose matching the new set of bounding boxes to the set of forecasted subsequent object positions to generate a set of matches & the updating according to the set of matches to obtain an updated set of object tracks.
RAJAN teaches matching the new set of bounding boxes to the set of forecasted object positions to generate a set of matches & the updating according to the set of matches to obtain an updated set of object tracks ([0091]-[0093], FIG. 8, A K Nearest Neighbour (KNN) stage to address drifting of Bounding Box (BB) due to Handling abrupt size variation due to diverging/converging motion of objects and a plurality of geometric and feature matching stages comprising a padding, an optical flow, a Scale-Invariant Feature Transform (SIFT) for handling track ID association between object detection and the 2D camera tracker in complicated scenarios based feature matching approach to control ghost tracking; Claim 3).
Therefore, it would have been obvious to one ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of matching the new set of bounding boxes to the set of forecasted object positions as taught by RAJAN ([0091]) into the system of HUANG in order to enable achieving improved accuracy of object detection, robustly and seamlessly tracking detected objects, providing better regularization and generalization in training enabling enhanced detection in noisy scenarios for received streaming data (RAJAN, [0004]).
HUANG & RAJAN do not explicitly disclose the set of subsequent positions for a subsequent time that is after a plurality of past times of the past positions
Elogeel teaches the set of subsequent positions for a subsequent time that is after a plurality of past times of the past positions ([0021], predict a future path for the objects and outputs several likely paths that an object is predicted to take along with a probability associated with each path and outputs a range of points along the path corresponding to a predicted location of the object along the path at future time intervals along with an expected error value for each of the points that indicates a probabilistic deviation from that point; [0060], calculate drift between a corner of a box from a corner of the keypoint and the drift is then passed into a Long Short-Term Memory (LSTM) neural network to determine precise kinematics estimation or stacking keypoints temporally and passing the keypoints to a Multilayered Perceptron and/or a PointNet to extract features and then passing the features to a LSTM neural network; [0070], weighting the kinematics of the object more heavily towards to observable features of the object, the objects kinematics are more likely to match the observed kinematics for the visible features of the object.).
Therefore, it would have been obvious to one ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of set of subsequent positions for a subsequent time that is after a plurality of past times of the past positions as taught by Elogeel ([0021]) into the system of HUANG in order to enable using self-supervised machine learning algorithm to optimize keypoint identification such that keypoints are located on similar objects at similar locations to reduce a variance value for locations of detected keypoints on similar objects (Elogeel, [0076]).
Regarding Claim 2, HUANG in view of RAJAN and Elogeel discloses the method of claim 1, HUANG discloses further comprising: obtaining the first set of object tracks of the set of objects in the geographic region at least by processing, by the detector model, a plurality of LiDAR point clouds of the geographic region over a span of time to detect a set of bounding boxes around the set of objects in the plurality of LiDAR point clouds ([0023] Among the modalities used for object detection during autonomous driving, LIDAR point clouds capture an accurate 3D scene structure, yielding state-of-the-art performance; [0059], the bounding box estimation module 318 is configured to estimate 3D bounding boxes for the set of moving objects based on the pseudo labels on the set of moving objects).
Regarding Claim 3, HUANG in view of RAJAN and Elogeel discloses the method of claim 1, HUANG discloses wherein the detector model is an unsupervised model ([0024], object detection, which achieves strong performance on self-supervised representation learning and unsupervised object detection tasks).
Regarding Claim 4, HUANG in view of RAJAN and Elogeel discloses the method of claim 1, HUANG discloses further comprising: generating, using the new LiDAR point cloud, a second set of object tracks of the set of objects, wherein the second set of object tracks are in a temporally reversed direction from the first set of object tracks; and filtering the second set of object tracks to remove object tracks failing to satisfy the track length threshold, and to further generate the training set of object tracks, wherein the training set of object tracks are in a temporally forward direction and the temporally reverse direction ([0029], detect moving object traces from temporally-ordered, unlabeled point cloud sequences captured by the sensors 114. The instructions loaded into the NPU 108 may also include code to extract a set of moving objects based on the moving object traces detected from the sequence of temporally-ordered, unlabeled point cloud sequences. The instructions loaded into the NPU 108 may also include code to classify the set of moving objects extracted from on the moving object traces detected from the sequence of temporally-ordered, unlabeled point cloud sequences; [0112] The method 900 may extract the set of moving objects by training a single-frame semantic instance segmentation model to differentiate between feature vectors representing the set of moving objects and the feature vectors representing a background of the temporally-ordered, unlabeled point cloud sequences).
Regarding Claim 5, HUANG in view of RAJAN and Elogeel discloses the method of claim 1, HUANG discloses further comprising: for at least one bounding box in the at least the subset of the new set of bounding boxes, revising the bounding box according to remaining bounding boxes in a corresponding object track of the first set of object tracks ([0059], the bounding box estimation module 318 is configured to estimate 3D bounding boxes for the set of moving objects based on the pseudo labels on the set of moving objects).
Regarding Claim 7, HUANG in view of RAJAN and Elogeel discloses the method of claim 1, HUANG discloses further comprising:
obtaining an initial set of LiDAR points in LiDAR data; removing a set of ground LiDAR points from the initial set of LiDAR points corresponding to ground to obtain a revised set of LiDAR points; clustering, by a clustering algorithm, the revised set of LiDAR points to obtain at least one cluster ([0065] FIG. 5 is a diagram 500 of an object trace 510 detected from temporally-ordered, unlabeled point cloud sequences, according to aspects of the present disclosure. As described, an object trace is a sequence of point clusters that move along smooth trajectories); labeling the at least one cluster with at least one label to generate at least one labeled cluster; and initially training the detector model using the initial set of LiDAR points and the at least one labeled cluster ([0059] In some aspects of the present disclosure, the object classification and labeling module 316 is configured as a pseudo-label generator to provide pseudo labels to the set of moving objects based on the classification (e.g., vehicle, pedestrian, or cyclist). Based on the pseudo labels, the bounding box estimation module 318 is configured to estimate 3D bounding boxes for the set of moving objects based on the pseudo labels on the set of moving objects; [0092] FIGS. 8A-8C are diagrams illustrating an overview of a bounding box inference step based on detected object traces and pseudo labels, according to aspects of the present disclosure).
Regarding Claim 9, Analogous rejection as the rejection of Claim 7 applies.
Regarding Claims 14-18, System claims 14-18 of using the corresponding method claimed in claims 1-9, and the rejections of which are incorporated herein for the same reasons as used above.
Regarding Claim 20, Computer readable medium claim 20 of using the corresponding method claimed in claims 1, and the rejections of which are incorporated herein for the same reasons as used above.
Claims 6, 8, 10-13 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over HUANG et al. (US 20240010225, hereinafter HUANG) in view of RAJAN et al. (US 20230334673, hereinafter RAJAN) , Elogeel and Manglani et al (US 20210405168, hereinafter, Manglani)
Regarding Claim 6, HUANG in view of RAJAN and Elogeel discloses the method of claim 1, but does not explicitly disclose discloses further comprising: performing a sparsity simulation filter on a new set of LiDAR points in the new LiDAR point cloud to obtain a revised new LiDAR point cloud, wherein retraining the detector model further uses the revised new LiDAR point cloud.
Manglani teaches performing a sparsity simulation filter on a new set of LiDAR points in the new LiDAR point cloud to obtain a revised new LiDAR point cloud, wherein retraining the detector model further uses the revised new LiDAR point cloud ([0062], performs a Simultaneous Localization and Mapping (SLAM) procedure to generate a map of the structure 14 to generate the map of the structure 14 based on data received by the lidar sensor 404 and/or RGBD camera 422. The calibration device 400 can use any suitable SLAM procedures to generate a map of the structure 14. For example, the SLAM procedures may comprise an Extended Kalman Filter (EKF) SLAM, a Sparse Extended Information Filter (SEIF) SLAM, or an Unscented Kalman filter (UKF) SLAM).
Therefore, it would have been obvious to one ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of performing a sparsity simulation filter on a new set of LiDAR points as taught by Manglani ([0062]) into the system of HUANG in view of RAJAN and Elogeel in order to provide system that determines the a multi-degree-of-freedom (MDF) position of the camera using a deep neural network that receives an input image that includes image data representative of the calibration board. The edge devices compute a multi-degree-of-freedom (MDF) position of the vehicle and use the computed MDF position to determine whether certain objects is within a path of the vehicle (Manglani, [0030]) resulting in the predictable result of improving quality of the camera pose in reliable manner.
Regarding Claim 8, 10 Analogous rejection as the rejection of Claim 6 applies.
Regarding Claim 11, HUANG in view of RAJAN and Elogeel and Manglani discloses the method of claim 10,
HUANG discloses clustering, by a clustering algorithm, the initial set of LiDAR points to obtain at least one cluster with an object trace 510 detected from temporally-ordered, unlabeled point cloud sequences, according to aspects of the present disclosure ([0065] FIG. 5 is a diagram 500(
In addition, Manglani teaches disclose wherein performing the sparsity simulation filtering comprises: selecting a set of LiDAR beams for dropping to obtain a set of dropped beams; and filtering the initial set of LiDAR points corresponding to the set of dropped beams ([0062], performs a Simultaneous Localization and Mapping (SLAM) procedure to generate a map of the structure 14 to generate the map of the structure 14 based on data received by the lidar sensor 404 and/or RGBD camera 422. The calibration device 400 can use any suitable SLAM procedures to generate a map of the structure 14. For example, the SLAM procedures may comprise an Extended Kalman Filter (EKF) SLAM, a Sparse Extended Information Filter (SEIF) SLAM, or an Unscented Kalman filter (UKF) SLAM). The same reason or rational of obviousness motivation applied as used above in claim 6.
Regarding Claim 12, HUANG in view of RAJAN, Elogeel and Manglani discloses the method of claim 10,
HUANG discloses using the velocity computed in Equation (3) to bring all point clusters into the same coordinate system and compute a 3D bounding box that covers all points with minimal volume. The resulting bounding box size vectors are used to train a model that regresses bounding box size. These aspects of the present disclosure use the model to predict bounding box sizes from all other object traces ([0102], FIGS. 8B and 8C)
In addition, Manglani discloses wherein performing the sparsity simulation filtering comprises: projecting the initial set of LiDAR points in spherical coordinates; and filtering the initial set of LiDAR points using at least one of evenly spaced rows and evenly spaced columns with the initial set of LiDAR points in spherical coordinates([0062], performs a Simultaneous Localization and Mapping (SLAM) procedure to generate a map of the structure 14 to generate the map of the structure 14 based on data received by the lidar sensor 404 and/or RGBD camera 422. The calibration device 400 can use any suitable SLAM procedures to generate a map of the structure 14. For example, the SLAM procedures may comprise an Extended Kalman Filter (EKF) SLAM, a Sparse Extended Information Filter (SEIF) SLAM, or an Unscented Kalman filter (UKF) SLAM). The same reason or rational of obviousness motivation applied as used above in claim 6.
Regarding Claim 13, HUANG in view of RAJAN and Elogeel discloses the method of claim 1, HUANG discloses further comprising: obtaining an initial set of LiDAR points in LiDAR data; clustering, by a clustering algorithm, the initial set of LiDAR points to obtain at least one cluster; labeling the at least one cluster with at least one label to generate at least one labeled cluster ([0065] FIG. 5 is a diagram 500 of an object trace 510 detected from temporally-ordered, unlabeled point cloud sequences, according to aspects of the present disclosure. As described, an object trace is a sequence of point clusters that move along smooth trajectories); and initially training the detector model using the filtered set of LiDAR points and the at least one labeled cluster ([0059] In some aspects of the present disclosure, the object classification and labeling module 316 is configured as a pseudo-label generator to provide pseudo labels to the set of moving objects based on the classification (e.g., vehicle, pedestrian, or cyclist). Based on the pseudo labels, the bounding box estimation module 318 is configured to estimate 3D bounding boxes for the set of moving objects based on the pseudo labels on the set of moving objects; [0092] FIGS. 8A-8C are diagrams illustrating an overview of a bounding box inference step based on detected object traces and pseudo labels, according to aspects of the present disclosure).
HUANG in view of RAJAN and Elogeel does not explicitly disclose performing a sparsity simulation filtering of the initial set of LiDAR points within a threshold distance to generate a filtered set of LiDAR points.
Manglani performing a sparsity simulation filtering of the initial set of LiDAR points within a threshold distance to generate a filtered set of LiDAR points ([0062], performs a Simultaneous Localization and Mapping (SLAM) procedure to generate a map of the structure 14 to generate the map of the structure 14 based on data received by the lidar sensor 404 and/or RGBD camera 422. The calibration device 400 can use any suitable SLAM procedures to generate a map of the structure 14. For example, the SLAM procedures may comprise an Extended Kalman Filter (EKF) SLAM, a Sparse Extended Information Filter (SEIF) SLAM, or an Unscented Kalman filter (UKF) SLAM).
Therefore, it would have been obvious to one ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of performing a sparsity simulation filter on a new set of LiDAR points as taught by Manglani ([0062]) into the system of HUANG in view of RAJAN and Elogeel in order to provide system that determines the a multi-degree-of-freedom (MDF) position of the camera using a deep neural network that receives an input image that includes image data representative of the calibration board. The edge devices compute a multi-degree-of-freedom (MDF) position of the vehicle and use the computed MDF position to determine whether certain objects is within a path of the vehicle (Manglani, [0030]) resulting in the predictable result of improving quality of the camera pose in reliable manner.
Regarding Claim 19, System claim 19 of using the corresponding method claimed in claim 10, and the rejections of which are incorporated herein for the same reasons as used above.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Samuel D Fereja whose telephone number is (469)295-9243. The examiner can normally be reached 8AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, DAVID CZEKAJ can be reached at (571) 272-7327. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SAMUEL D FEREJA/Primary Examiner, Art Unit 2487
